Deploy and isolate a GitLab Runner with Docker executor and cache
Operate a protected GitLab Runner 19.x on a dedicated host with nonprivileged Docker jobs, digest-pinned image admission, trust-isolated cache, explicit tags, adversarial tests, safe cleanup, and rebuild-first recovery.
Accelerate protected pipelines without giving job containers host-root capability, letting mutable images enter, mixing protected/unprotected cache, exposing runner tokens, or making a shared Docker host the hidden trust boundary.
- Ubuntu Server 24.04 LTS
- GitLab Runner 19.x; match GitLab instance
- GitLab 19.x
- Docker Engine current supported stable
- GitLab governance Project/group owner, protected refs, explicit runner tag, runner creation workflow, audit access, and separate untrusted runner path.
- Dedicated host Patched fixed-purpose Ubuntu VM with Docker, narrow network, console access, monitoring, capacity, and no unrelated workloads.
- Promoted images Private CI registry, digest locks, scan/review/promotion process, nonroot images, and approved service dependencies.
- Cache isolation Dedicated bucket/prefix, workload identity, encryption, lifecycle, protected/project keys, and prohibition on secrets/executable trust.
- Operations Drain/upgrade, cleanup, negative tests, kill switch, token rotation, sanitized config baseline, cache quarantine, and rebuild recovery.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A dedicated Ubuntu 24.04 LTS host running a current GitLab Runner 19.x package and Docker executor, with a project/group-scoped protected runner, required tags, and no untagged work.
- A nonprivileged Docker job profile with no privileged mode, Docker socket, host namespaces/devices, broad volumes, or unapproved images/services; CI images are promoted and pinned by digest.
- A project- and protection-isolated cache using a dedicated object-store prefix and workload identity, plus capacity cleanup, monitoring, negative tests, token rotation, and rebuild-first recovery.
- Only protected refs in approved projects with every required tag can schedule jobs, while unprotected, untagged, disallowed-image, privileged, and cross-cache tests fail closed.
- Each job receives an isolated container/network and disposable build directory; persistent cache contains only nonsecret reproducible dependencies and cannot cross trust boundaries.
- Operators can upgrade, drain, recover, rotate, clean, and disable the runner without restoring live credentials or depending on a mutable local host.
Architecture
How the parts fit together
A dedicated Linux host runs the GitLab Runner service and Docker Engine for one protected project or tightly controlled group. The daemon remains a host-level privilege held by the runner service, so the entire host is one trust boundary and is never shared with lower-trust projects. Job containers are unprivileged, digest-pinned, capability-reduced, and isolated on per-job networks. Distributed cache uses workload identity, a dedicated bucket/prefix, Shared=false, and project/protection-aware keys. Configuration is rebuilt from a sanitized baseline; tokens are rotated rather than restored.
- A protected pipeline with all required tags requests the project/group runner.
- Runner pulls only an allowed digest-pinned image and creates a nonprivileged container plus isolated service network.
- Pre-job restores a project/protection-scoped cache; the job builds/tests without host daemon/socket access; post-job uploads a bounded cache.
- Artifacts return through GitLab with retention and access policy, while cache remains optional/untrusted.
- Monitoring records runner/job/container/cache lifecycle; cleanup handles only unused managed objects after drain.
Assumptions
- GitLab Runner 19.x is compatible with the organization's GitLab 19.x instance and exact upgrade notes are reviewed for every minor/patch transition.
- The runner is dedicated to one protected project or same-trust group. Untrusted merge requests, forks, public projects, and unprotected refs use another runner class.
- Docker Engine access makes the runner service/host a high-trust boundary even though individual job containers are unprivileged.
- The object store supports workload identity or an instance profile so static cache credentials are not embedded in config.toml.
- Container builds that require privileged Docker-in-Docker are excluded from this runner and use the separately isolated supply-chain pipeline in the next guide.
Key concepts
- Protected runner
- A runner configured in GitLab to serve protected branches/tags, optionally controlled merge-request cases, reducing exposure of privileged environments.
- Runner authentication token
- A glrt-/glrtr- credential that identifies runner managers; it can register execution capacity and must not be logged or restored broadly.
- Docker executor
- GitLab executor that runs prepare, pre-job, job, and post-job phases in containers using a host Docker Engine.
- Privileged mode
- A container mode that disables key isolation controls and can lead to host breakout; it is disabled in this design.
- Protected cache
- Cache namespace separated by project and protected/unprotected trust, containing only disposable reproducible inputs.
- Artifact
- A pipeline output with GitLab lifecycle/access semantics. It is distinct from a cache and should carry provenance/integrity appropriate to deployment.
- Allowed image
- A runner-side registry/name pattern limiting which job and service images may be requested; promotion policy and digest pinning complete the control.
Before you copy
Values used in this guide
{{gitlabUrl}}Approved GitLab instance URL.
Example: https://gitlab.example.com{{runnerName}}Stable descriptive runner/manager name.
Example: payments-protected-docker-01{{runnerTag}}Required trust/capability job tag.
Example: protected-docker{{ciRegistry}}Approved registry namespace for CI job images.
Example: registry.example.com/ci{{ciImageDigest}}Reviewed sha256 digest for the default CI image.
Example: sha256:reviewed-64-hex-digest{{cacheBucket}}Dedicated object-store bucket for this trust boundary.
Example: gitlab-runner-cache-prod{{cachePath}}Dedicated cache prefix for the protected pool.
Example: protected/payments{{gitlabProjectId}}Numeric GitLab project ID used in cache/test evidence.
Example: 4242{{gitlabRunnerPackageVersion}}Exact reviewed apt version shared by gitlab-runner and gitlab-runner-helper-images.
Example: 19.0.0-1{{dockerEnginePackageVersion}}Exact reviewed apt version shared by docker-ce and docker-ce-cli.
Example: 5:28.3.2-1~ubuntu.24.04~noble{{containerdPackageVersion}}Exact reviewed containerd.io apt package version.
Example: 1.7.27-1{{dockerBuildxPackageVersion}}Exact reviewed docker-buildx-plugin apt package version.
Example: 0.25.0-1~ubuntu.24.04~noble{{dockerComposePackageVersion}}Exact reviewed docker-compose-plugin apt package version.
Example: 2.38.2-1~ubuntu.24.04~nobleSecurity and production boundaries
- Never mount /var/run/docker.sock, host root, devices, or host PID/IPC/network into job containers. Docker daemon access is host-root capability.
- Set privileged=false and do not weaken SELinux/AppArmor/capabilities to make an arbitrary image work. Fix the image or move the workload.
- Protected status and tags must align with protected refs and project scope; tags alone do not prove trust.
- Caches can be poisoned and can leak source/dependencies. Keep secrets out, isolate buckets/prefixes and keys by trust, and validate restored content.
- Runner authentication token and config.toml are secrets. Prefer rebuild plus new registration/rotation over restoring a live token-bearing file.
Stop before continuing if
- Stop if unprotected/fork projects or refs can target the runner, untagged jobs are accepted, or project/group scope is broader than the cache/network trust boundary.
- Stop if privileged mode, Docker socket, host namespaces/devices, broad host volumes, or root build images are required without separate risk architecture.
- Stop if images/services are mutable or outside the allowed promoted registry and digest policy.
- Stop if cache uses shared static credentials, Shared=true, common prefixes across trust boundaries, secrets, or unvalidated executable outputs.
- Stop if the token appears in argv/logs/backups, Runner/GitLab versions are incompatible, or negative isolation tests fail.
decision
Define one runner trust boundary and its protected scope
Map projects, refs, merge-request rules, tags, secrets, network destinations, images, artifacts, cache, and operators. Create a project or narrow group runner, mark it Protected, disable untagged jobs, assign one explicit tag, and prohibit lower-trust projects from sharing the host or cache.
Why this step matters
A Docker executor is not a universal multitenant sandbox. Restricting project/ref/tag scope keeps job code, caches, networks, and daemon exposure within one understood trust boundary.
What to understand
Configure Protected and tag/untagged policy in GitLab when creating/editing the runner; registration does not replace governance.
Decide explicitly whether protected merge-request pipelines are allowed and what code/ref they execute.
Use separate runner managers/hosts for materially different trust, even when a group hierarchy makes sharing convenient.
System changes
- The command prints the reviewed nonsecret policy. GitLab owner/maintainer actions create the runner scope, protected setting, and tags.
Syntax explained
protected=true- Restricts runner use to protected refs under GitLab policy.
run_untagged=false- Requires jobs to request the approved tag explicitly.
project_id- Binds evidence and cache design to the exact approved project.
Values stay on this page and are never sent or saved.
printf '%s\n' 'runner={{runnerName}}' 'tag={{runnerTag}}' 'protected=true' 'run_untagged=false' 'project_id={{gitlabProjectId}}'runner=payments-protected-docker-01 tag=protected-docker protected=true run_untagged=false project_id=4242 assigned projects: payments-api only unprotected merge requests: separate low-trust runner cache trust boundary: project + protected refs
Checkpoint: Verify this layer before continuing
Continue whenProtected scope, project assignment, tags, merge-request behavior, secrets, network, image and cache boundaries are approved.
Stop whenStop if a public/unprotected project/ref shares the runner, if tags are optional, or if cache/network authority spans different trust.
If this step fails
An unprotected pipeline is eligible in the UI.
Likely causeRunner protected/ref settings or project assignment are too broad.
Pause the runner and inspect effective protected status and ref rules.Review all assigned projects and merge-request pipeline options.
ResolutionNarrow the runner/ref assignment and repeat a negative pipeline before installing a host.
Security notes
- Audit runner setting changes and restrict who can attach projects.
Alternatives
- Use separate GitLab-hosted or low-trust self-managed runners for unprotected work.
Stop conditions
- Stop if a public/unprotected project/ref shares the runner, if tags are optional, or if cache/network authority spans different trust.
command
Install current GitLab Runner and Docker on a dedicated host
Build a clean Ubuntu 24.04 LTS VM, add the official GitLab Runner package repository only after downloading and reviewing its setup script, install a GitLab Runner 19.x version compatible with the GitLab instance plus matching helper images, install supported Docker Engine, and record package/image versions.
Why this step matters
Runner and helper compatibility can change across GitLab releases, while the Docker daemon defines the execution boundary. A dedicated, versioned host makes upgrades and compromise recovery reproducible.
What to understand
Download and inspect the repository script instead of piping it directly to a shell.
When pinning a nonlatest Runner version, GitLab documents installing the matching helper-images package.
Keep GitLab Runner close to the GitLab instance minor version and review every 19.x upgrade note before rollout.
System changes
- Adds official package repository metadata, GitLab Runner service/package, Docker Engine, and their host dependencies.
Syntax explained
curl --fail --location- Downloads the official repository configuration script and fails on HTTP errors.
less before bash- Creates a required human/automated review gate before executing repository setup.
gitlab-runner --version- Records version/revision used for compatibility and rollback.
package=exact-version- Installs the reviewed Runner/helper pair and Docker components instead of accepting whichever versions apt currently prefers.
Values stay on this page and are never sent or saved.
curl --fail --location --proto '=https' --tlsv1.2 --output /tmp/gitlab-runner-repo.sh https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh; less /tmp/gitlab-runner-repo.sh; sudo bash /tmp/gitlab-runner-repo.sh; sudo apt-get update; sudo apt-get install --yes ca-certificates curl; sudo install -m 0755 -d /etc/apt/keyrings; sudo curl --fail --location https://download.docker.com/linux/ubuntu/gpg --output /etc/apt/keyrings/docker.asc; sudo chmod a+r /etc/apt/keyrings/docker.asc; . /etc/os-release; printf 'Types: deb\nURIs: https://download.docker.com/linux/ubuntu\nSuites: %s\nComponents: stable\nArchitectures: %s\nSigned-By: /etc/apt/keyrings/docker.asc\n' "${UBUNTU_CODENAME:-$VERSION_CODENAME}" "$(dpkg --print-architecture)" | sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null; sudo apt-get update; sudo apt-get install --yes gitlab-runner={{gitlabRunnerPackageVersion}} gitlab-runner-helper-images={{gitlabRunnerPackageVersion}} docker-ce={{dockerEnginePackageVersion}} docker-ce-cli={{dockerEnginePackageVersion}} containerd.io={{containerdPackageVersion}} docker-buildx-plugin={{dockerBuildxPackageVersion}} docker-compose-plugin={{dockerComposePackageVersion}}; gitlab-runner --version; docker version; docker buildx version; docker compose versionGitLab Runner repository script: reviewed GitLab Runner: 19.x.y Revision: reviewed release Operating system: linux/amd64 GitLab instance compatibility: approved Docker Engine: current supported stable release gitlab-runner service: active host roles: GitLab Runner and Docker only
Checkpoint: Verify this layer before continuing
Continue whenPackages come from approved repositories, versions are compatible/current, service is active, and the host has no unrelated workloads.
Stop whenStop on unreviewed script/repository, incompatible helper/instance version, unsupported OS, or shared host roles.
If this step fails
Runner package installation selects mismatched helper images.
Likely causeA specific Runner version was pinned without its exact helper package.
Inspect apt policy and official repository versions.Compare the required package pair in GitLab's installation documentation.
ResolutionInstall the exact Runner/helper package pair, canary it, and record both versions.
Security notes
- Protect apt repository keys/configuration and runner package rollout as CI supply-chain controls.
Alternatives
- Use an official runner container on a dedicated host only if its config volume and Docker boundary are equally protected.
Stop conditions
- Stop on unreviewed script/repository, incompatible helper/instance version, unsupported OS, or shared host roles.
command
Register interactively with the new authentication-token workflow
Create the protected runner in GitLab first, obtain its short-lived display of the glrt- authentication token, and run sudo gitlab-runner register interactively. Enter the token only at the protected prompt, choose Docker, the exact URL/name/tag/default image, and prevent token capture in shell history or diagnostics.
Why this step matters
The current GitLab creation workflow separates owner-controlled scope/settings from manager registration and uses runner authentication tokens. Interactive input avoids a token literal in argv/history.
What to understand
The token can register/operate runner managers that execute jobs with repository/job-token access; treat exposure as CI control-plane compromise.
Registration appends config stanzas, so verify only one intended runner exists afterward.
Runner Protected and tag policy are set in GitLab, not trusted merely because local config uses a descriptive name.
System changes
- Creates a runner manager in GitLab and writes secret-bearing authentication/configuration to root-owned config.toml.
Syntax explained
gitlab-runner register- Starts the supported interactive registration workflow.
runner authentication token prompt- Accepts glrt- material without embedding it in the documented command.
docker executor- Selects isolated job containers rather than direct shell execution.
sudo gitlab-runner registerEnter the GitLab instance URL: https://gitlab.example.com Enter the runner authentication token: [redacted; not echoed] Verifying runner... is valid Enter a name for the runner: payments-protected-docker-01 Enter an executor: docker Enter the default Docker image: registry.example.com/ci/base@sha256:[reviewed digest] Runner registered successfully. Authentication token in command argv/output: false
Checkpoint: Verify this layer before continuing
Continue whenOne manager is registered to the intended protected runner, token is absent from argv/log/output, and config.toml is root-owned mode 0600.
Stop whenStop if a legacy registration token, noninteractive argv token, duplicate stanza, broad runner scope, or token exposure occurs.
If this step fails
Registration returns 410 Gone.
Likely causeLegacy runner registration tokens are disabled, as expected on modern GitLab.
Confirm the runner was first created in GitLab and the token has glrt-/glrtr- form.Check the instance URL and runner creation permissions.
ResolutionUse the new runner creation/authentication-token workflow; do not re-enable deprecated registration globally.
Security notes
- Never back up, paste, or commit the generated token line.
Alternatives
- Provision via a protected secret manager/stdin automation if interactive registration cannot scale, while ensuring the token never enters argv/logs.
Stop conditions
- Stop if a legacy registration token, noninteractive argv token, duplicate stanza, broad runner scope, or token exposure occurs.
config
Harden the Docker executor and pin allowed images
Merge the security stanza into the generated runner entry without replacing its secret token. Disable privileged execution, host sockets/devices/namespaces and untagged work; allow only promoted registry images/services, use digest-pinned default image, drop capabilities, create per-job networks, and keep only a managed /cache volume.
Why this step matters
Docker executor isolation depends on runner-side config, not on a cooperative .gitlab-ci.yml. A hostile job must be unable to request privileged mode, host sockets, mutable public images, or shared host paths.
What to understand
The runner service can control Docker and is therefore host-trusted; job containers remain nonprivileged and the host is dedicated to the same trust.
Digest references identify content; allowed_images/services limit registry namespace, and registry promotion policy decides what enters it.
Per-job networking and capability drop reduce cross-job/kernel surface, but application images may need narrowly reviewed capabilities restored.
System changes
- Changes runner scheduling, container privilege, image/service admission, pull behavior, capabilities, networks, concurrency, and persistent mount policy.
Syntax explained
privileged/services_privileged=false- Prevents privileged job and service containers.
allowed_images/services- Restricts jobs to digest-addressed promoted registry content.
cap_drop=['ALL']- Removes default Linux capabilities; add back only proven minimum.
volumes=['/cache']- Avoids host Docker socket and broad host bind mounts.
/etc/gitlab-runner/config.toml (merge into generated stanza)Values stay on this page and are never sent or saved.
concurrent = 2
check_interval = 3
[[runners]]
name = "{{runnerName}}"
url = "{{gitlabUrl}}"
executor = "docker"
environment = ["FF_NETWORK_PER_BUILD=1"]
[runners.docker]
image = "{{ciRegistry}}/base@{{ciImageDigest}}"
privileged = false
services_privileged = false
pull_policy = "always"
allowed_pull_policies = ["always"]
allowed_images = ["{{ciRegistry}}/*@sha256:*"]
allowed_services = ["{{ciRegistry}}/*@sha256:*"]
cap_drop = ["ALL"]
volumes = ["/cache"]
shm_size = 134217728
config validation: passed executor: docker privileged: false host Docker socket in job: absent allowed image namespace: registry.example.com/ci default image digest-pinned: true capabilities: ALL dropped per-job network: enabled persistent paths: /cache only
Checkpoint: Verify this layer before continuing
Continue whengitlab-runner config validation and canary pass, while privileged, socket, device, host namespace, public/mutable image, and unexpected volume requests fail.
Stop whenStop if any job needs privileged/socket/root host access, allowed patterns accept mutable/untrusted images, or capabilities are re-added broadly.
If this step fails
The approved CI image fails after ALL capabilities are dropped.
Likely causeIts entrypoint/tool incorrectly assumes root or a Linux capability.
Run the exact container locally in an isolated test with cap-drop ALL.Identify syscall/capability need from audit evidence, not trial-and-error broadening.
ResolutionFix the image to run nonroot or add one narrowly justified capability in a dedicated runner class.
Security notes
- Protect config.toml and Docker daemon; jobs must not read either.
Alternatives
- Use rootless container runtime/executor where supported and tested for an additional boundary.
Stop conditions
- Stop if any job needs privileged/socket/root host access, allowed patterns accept mutable/untrusted images, or capabilities are re-added broadly.
config
Isolate distributed cache by project and protected trust
Use a dedicated object-store bucket/prefix and workload identity, Shared=false, encryption/TLS, bounded lifecycle, and protected project cache separation. Cache only reproducible dependency directories with a lockfile-derived key; never cache secrets, credentials, deployment bundles, or unverifiable executables.
Why this step matters
Cache trades integrity for speed: a job can write data consumed later. Separating trust and treating restored cache as untrusted prevents poisoning and cross-project disclosure.
What to understand
Use the runner's workload identity/instance profile so static keys never appear in config.toml.
GitLab cache paths include project context and modern protection suffix behavior, but separate runner/bucket/prefix remains the clear security boundary.
Lockfile-derived keys and clean fallback policy reduce stale dependency reuse; package signatures/checksums still require validation.
System changes
- Configures remote cache storage, path, sharing, maximum archive size, transport, encryption/lifecycle through object-store policy.
Syntax explained
Shared=false- Prevents one cache namespace being deliberately shared by multiple runners.
Path- Creates a trust-specific object prefix in the dedicated bucket.
MaxUploadedArchiveSize- Bounds cache upload size to reduce resource abuse.
Insecure=false- Requires secure object-store transport.
/etc/gitlab-runner/config.toml (cache subsection)Values stay on this page and are never sent or saved.
[runners.cache]
Type = "s3"
Path = "{{cachePath}}"
Shared = false
MaxUploadedArchiveSize = 1073741824
[runners.cache.s3]
BucketName = "{{cacheBucket}}"
BucketLocation = "eu-central-1"
Insecure = false
cache type: S3-compatible object store static credentials in config.toml: none workload identity: runner-cache-prod bucket: gitlab-runner-cache-prod path: protected/payments Shared: false cache key: project-4242-protected-lockfile-[digest] unprotected marker visible to protected job: false encryption/TLS/lifecycle: enabled
Checkpoint: Verify this layer before continuing
Continue whenProtected jobs see only their project/protection cache, synthetic unprotected markers never cross, no static secret exists, and lifecycle/size/encryption alerts pass.
Stop whenStop if cache keys/prefix/bucket or IAM span trust boundaries, executable output is trusted without validation, or secrets enter cache paths.
If this step fails
Protected job restores a marker from an unprotected branch.
Likely causeCache separation or fallback keys collapse protection context.
Inspect object path metadata and CI cache key/fallback behavior.Disable cache restore for the protected job while preserving the suspect object.
ResolutionDelete/quarantine poisoned entries, separate the runner and bucket/prefix, fix keys/protected cache settings, and repeat synthetic negatives.
Security notes
- Object-store access logs and cache contents can expose project/dependency metadata; restrict them.
Alternatives
- Disable cache and accept slower clean dependency fetches for the highest-risk job.
Stop conditions
- Stop if cache keys/prefix/bucket or IAM span trust boundaries, executable output is trusted without validation, or secrets enter cache paths.
config
Create a protected digest-pinned canary pipeline
Require the runner tag, reference the approved CI image by digest, use a lockfile-derived cache key and nonsecret dependency directory, keep artifacts separate, and run as the image's nonroot USER. Do not request privileged services, host mounts, or Docker-in-Docker.
Why this step matters
The pipeline must preserve runner controls: explicit tag and digest, nonroot execution, cache scoped by project/protection, and separately retained artifacts.
What to understand
CI_COMMIT_REF_PROTECTED becomes part of the key but should not be the sole trust control; runner/bucket separation still applies.
Package managers must verify lockfile/integrity even when using cached downloads.
Artifact retention/access is intentional and not conflated with cache speed.
System changes
- Adds a protected test job, cache key/path/policy, artifact retention, and nonroot/socket assertions to repository CI configuration.
Syntax explained
tags- Requires the protected Docker runner capability.
image@digest- Fixes exact job image content.
CI_COMMIT_REF_PROTECTED- Separates cache keys by ref protection status.
test ! -S /var/run/docker.sock- Negative assertion that host daemon control is unavailable.
.gitlab-ci.ymlValues stay on this page and are never sent or saved.
stages: [test]
secure-test:
stage: test
tags: [{{runnerTag}}]
image: {{ciRegistry}}/base@{{ciImageDigest}}
cache:
key:
files: [package-lock.json]
prefix: "$CI_PROJECT_ID-$CI_COMMIT_REF_PROTECTED"
paths: [.npm/]
policy: pull-push
script:
- test "$(id -u)" -ne 0
- test ! -S /var/run/docker.sock
- npm ci --cache .npm --prefer-offline
- npm test
artifacts:
paths: [test-results/]
expire_in: 7 days
pipeline lint: valid runner tag: protected-docker image resolved to reviewed sha256 digest container user: ci (uid 10001) privileged: false Docker socket: absent cache key includes project/protection and lockfile digest artifact checksum: recorded
Checkpoint: Verify this layer before continuing
Continue whenProtected canary runs nonroot on exact image digest, cannot see Docker socket, verifies dependencies, produces bounded artifacts, and keeps cache separated.
Stop whenStop if image/tag mutates, job is root/privileged, socket appears, unprotected cache restores, or artifact contains secrets.
If this step fails
npm ci fails only with restored cache.
Likely causeCache is stale/poisoned or inconsistent with the lockfile.
Retry once with a new/disabled cache key.Compare package integrity errors and cache object origin.
ResolutionQuarantine/delete the bad cache, preserve lockfile verification, and never bypass integrity to reuse it.
Security notes
- Repository CI changes require protected-branch review and CODEOWNERS appropriate to runner trust.
Alternatives
- Use cache policy pull for protected release jobs so they do not publish new cache entries.
Stop conditions
- Stop if image/tag mutates, job is root/privileged, socket appears, unprotected cache restores, or artifact contains secrets.
verification
Run adversarial scheduling, privilege, image, network, and cache tests
Before secrets/internal write access exist, submit synthetic pipelines from protected and unprotected refs: no tag, wrong tag, public image, mutable tag, privileged service, socket probe, cross-project cache marker, and concurrent network canaries. Every negative must fail before job code gains the prohibited capability.
Why this step matters
Configuration review misses interactions between GitLab scheduling, Runner config, Docker, registry, and cache. Negative jobs prove enforcement at the actual boundaries.
What to understand
Use synthetic projects/markers and no secrets; a failure is expected and must not become a real incident.
Test concurrent jobs because network/cache isolation bugs often appear only under overlap.
Retain GitLab, Runner, Docker, object-store, and network evidence for each denial.
System changes
- Creates test pipelines/containers/cache markers and audit events; prohibited actions must not succeed.
Syntax explained
unprotected/untagged- Tests GitLab protected scheduling and tag requirements.
public/mutable image- Tests registry/image admission and digest policy.
privileged/socket- Tests host-root capability denial.
cross-cache/network- Tests state and service isolation between trust/jobs.
printf '%s\n' 'protected+tag=allow' 'unprotected=deny' 'untagged=deny' 'public-or-mutable-image=deny' 'privileged-or-socket=deny' 'cross-cache=deny' 'cross-job-network=deny'protected ref + required tag: passed unprotected ref on protected runner: denied untagged job: pending/no match public image: rejected by allowed_images mutable tag without digest: rejected by policy privileged service: unavailable /var/run/docker.sock: absent unprotected cache marker in protected job: absent cross-job service marker: unreachable secrets/internal writes during test: none
Checkpoint: Verify this layer before continuing
Continue whenOnly the approved protected tagged canary runs; every prohibited capability is denied with observable evidence and no secret/internal effect.
Stop whenPause and isolate the runner on any unexpected success, ambiguous result, or missing evidence.
If this step fails
The public image job starts despite allowed_images.
Likely causeThe configured wildcard is too broad, the job used a service/helper path not covered, or config reload failed.
Inspect effective config and runner logs for resolved image.Verify service/helper admission and systemd reload/restart state.
ResolutionNarrow all image/service paths, restart/drain safely, rebuild if untrusted code ran, and repeat the full matrix.
Security notes
- Assume host compromise if privileged/socket tests unexpectedly succeeded.
Alternatives
- Run the matrix continuously after Runner/Docker/GitLab upgrades.
Stop conditions
- Pause and isolate the runner on any unexpected success, ambiguous result, or missing evidence.
verification
Validate capacity, cleanup, and observability without deleting active data
Monitor queue/concurrency, job failures, Docker daemon events, disk/inodes, cache upload/download errors, protected scheduling denials, and container/network cleanup. Run clear-docker-cache space as dry evidence, drain the runner, then prune only unused runner-managed objects with reviewed label filters.
Why this step matters
Docker and cache state can exhaust a runner or retain job data. Capacity management must distinguish active objects, runner-managed volumes, images, and remote cache rather than running destructive global prune blindly.
What to understand
clear-docker-cache does not remove all images; review Docker image/build-cache policy separately.
Drain and verify zero active jobs before deletion, and use labels/age filters.
Alert before capacity emergency and monitor cleanup failures as security state-retention failures.
System changes
- Commands inspect capacity and runner registration. A separately approved drained cleanup removes only verified unused managed objects.
Syntax explained
clear-docker-cache space- Shows reclaimable managed Docker data without deleting it.
docker system df- Breaks down image, container, volume, and build-cache usage.
df -h/-i- Checks byte and inode exhaustion independently.
clear-docker-cache space; docker system df; df -h /var/lib/docker; df -i /var/lib/docker; gitlab-runner verifyDocker disk usage: 18.2 GB total, 6.1 GB reclaimable active containers: 0 (runner drained) runner-managed volume filter: com.gitlab.gitlab-runner.managed=true filesystem use: 31% inode use: 4% runner verify: alive orphan job networks: 0 cache error rate: 0.0% alert delivery: passed
Checkpoint: Verify this layer before continuing
Continue whenCapacity remains within threshold, no orphan networks/containers exist, cleanup is drained/scoped/audited, and all health alerts deliver.
Stop whenStop cleanup if any active job/object ownership is uncertain, filters are broad, or backup/evidence must be preserved.
If this step fails
Disk remains full after managed-volume cleanup.
Likely causeImages, build cache, logs, or nonrunner data consume space.
Use docker system df -v and journal/log size reports.Map each large object to owner and active job/image policy.
ResolutionDrain and apply a separately reviewed image/build-cache/log retention action; do not use unscoped docker system prune.
Security notes
- Docker metadata can contain registry/image names and job identifiers; restrict monitoring.
Alternatives
- Replace the entire disposable runner host when cleanup complexity exceeds recovery value.
Stop conditions
- Stop cleanup if any active job/object ownership is uncertain, filters are broad, or backup/evidence must be preserved.
command
Upgrade by drain and canary, preserving a known-good rollback package
Pause the runner, wait for active jobs to finish, back up only sanitized configuration, install the reviewed GitLab Runner/helper and Docker versions, validate config, run the negative matrix, and unpause gradually. Accept documented cold-cache behavior after storage naming changes rather than weakening isolation.
Why this step matters
Runner upgrades can change helper images, cache paths, config semantics, and scheduling. Draining and canarying prevent job interruption and security regression.
What to understand
The secret-bearing temporary backup is root-only and must be encrypted/deleted after rollback window; the durable baseline omits tokens.
Install matching helper packages when pinning a Runner release.
Never restore cache sharing or if-not-present behavior just to mask expected cold cache after an upgrade.
System changes
- Stops/starts job polling, changes Runner/Docker packages and helper images, may change cache layout, and creates a temporary protected config backup.
Syntax explained
runner pause/drain- Prevents new jobs and allows active work to finish before service/package change.
config validation/verify- Checks syntax and GitLab connectivity before reopening.
apt exact versions- Installs the approved Runner/helper and Docker package set, including an intentional tested downgrade during rollback.
negative matrix- Makes security regressions a blocking release test.
Values stay on this page and are never sent or saved.
sudo gitlab-runner stop; sudo install --mode 0600 /etc/gitlab-runner/config.toml /root/config.toml.secret-backup; sudo apt-get update; sudo apt-get install --yes --allow-downgrades gitlab-runner={{gitlabRunnerPackageVersion}} gitlab-runner-helper-images={{gitlabRunnerPackageVersion}} docker-ce={{dockerEnginePackageVersion}} docker-ce-cli={{dockerEnginePackageVersion}} containerd.io={{containerdPackageVersion}} docker-buildx-plugin={{dockerBuildxPackageVersion}} docker-compose-plugin={{dockerComposePackageVersion}}; sudo gitlab-runner verify; gitlab-runner --version; docker version; sudo gitlab-runner startrunner paused/drained in GitLab: true active jobs before stop: 0 secret-bearing backup path: root-only temporary recovery copy previous package versions retained: true new GitLab Runner: 19.x.y config validation: passed positive canary: passed negative matrix: passed cache migration: expected cold cache, no trust relaxation
Checkpoint: Verify this layer before continuing
Continue whenNo jobs are interrupted, versions/config are compatible, all positives/negatives pass, alerts are healthy, and the previous verified package remains available.
Stop whenStop rollback/rollout if jobs remain active, token backup is uncontrolled, config changes silently, or any negative test regresses.
If this step fails
New runner ignores old local cache volumes.
Likely causeGitLab Runner 18.4+ storage naming changed or runner identity differs.
Compare official release/upgrade notes and volume labels.Confirm jobs rebuild dependencies with integrity checks.
ResolutionAccept and monitor cold cache or migrate only through documented safe tooling; do not mount broad legacy volumes into new jobs.
Security notes
- Package/repository compromise affects every job; verify sources and image rollout.
Alternatives
- Blue/green runner hosts provide cleaner rollback than in-place upgrades.
Stop conditions
- Stop rollback/rollout if jobs remain active, token backup is uncontrolled, config changes silently, or any negative test regresses.
warning
Recover by rebuild, token rotation, and isolated cache treatment
For host compromise, token exposure, failed upgrade, or disaster, pause/delete the runner manager, isolate the host, preserve external evidence, rotate the authentication token and reachable credentials, rebuild from the verified image/config baseline, and register a new manager. Treat cache as disposable/untrusted; do not restore a live config.toml identity.
Why this step matters
A runner host executes arbitrary project code and stores authentication plus transient source/cache. Recovery must replace identity and host state rather than trusting copied configuration.
What to understand
Deleting a manager and rotating token prevents a restored clone from reconnecting.
Durable backup contains sanitized config and package/image manifests; secret material comes from new registration.
Cache is reconstructed from verified lockfiles/registries after quarantine, not restored as trusted executable state.
System changes
- Disables scheduling, invalidates runner identity, replaces the host, quarantines cache, rotates credentials, and may delay pipelines.
Syntax explained
delete-or-rotate- Prevents duplicate or stolen managers from polling jobs.
rebuild-host- Removes persistent job/daemon compromise from the trust boundary.
quarantine-cache- Stops poisoned data from entering the rebuilt runner.
negative-matrix- Proves isolation before reopening.
printf '%s\n' 'pause-runner' 'isolate-host' 'delete-or-rotate-runner-manager-token' 'rebuild-host' 'register-new-manager' 'quarantine-cache' 'repeat-negative-matrix'recovery runbook: CI-GLR-072 approved kill switch owner: CI Operations external evidence independent of host: yes sanitized config baseline: versioned live config.toml restoration: prohibited runner authentication token rotation: tested cache quarantine/rebuild: tested reopen gate: positive and all negative tests
Checkpoint: Verify this layer before continuing
Continue whenOwners can disable and rebuild from independent evidence, old manager cannot reconnect, tokens rotate, cache starts clean, and all tests pass.
Stop whenDo not reconnect a restored disk/config, reuse a potentially poisoned cache, or reopen before every trust test passes.
If this step fails
The old manager comes online after recovery.
Likely causeIts authentication token remained valid or a cloned config.toml exists.
Pause the runner globally and list runner managers/system IDs.Identify all backup/image copies containing config.toml.
ResolutionDelete managers, rotate the runner authentication token, sanitize backups, and register only the new verified manager.
Security notes
- Assume the runner token and all reachable job credentials are exposed after host compromise.
Alternatives
- Use blue/green disposable hosts routinely so disaster recovery matches normal replacement.
Stop conditions
- Do not reconnect a restored disk/config, reuse a potentially poisoned cache, or reopen before every trust test passes.
Finish line
Verification checklist
sudo gitlab-runner list; sudo gitlab-runner verifyExactly one intended manager is alive for the protected project/group; token is never printed.sudo gitlab-runner verify; docker ps --all --filter label=com.gitlab.gitlab-runner.managed=trueConfig validates; jobs are nonprivileged, digest-pinned, capability-reduced, with no socket/host namespaces or orphan containers.printf '%s\n' 'unprotected denied' 'untagged no match' 'unauthorized project denied'Only protected refs in the approved scope with every tag can schedule.printf '%s\n' 'project scoped' 'protected scoped' 'untrusted marker absent' 'no secrets'Synthetic lower-trust cache never enters protected jobs and object access is workload-identity/prefix limited.printf '%s\n' 'manager token rotated' 'host rebuilt' 'cache quarantined' 'matrix passed'A new manager replaces the old identity from sanitized configuration and all positive/negative tests pass.Recovery guidance
Common problems and safe checks
The runner is online but protected jobs remain pending.
Likely causeThe job lacks every required runner tag, the runner is paused or not assigned to the project/group, protected status does not match the ref, or run_untagged is disabled as designed.
Compare job tags and protected ref status with the runner UI, scope, tag list, and pause/online state.Run gitlab-runner verify and inspect service logs without printing the runner authentication token.
ResolutionCorrect the narrow project/group assignment, protected ref, or explicit tags. Do not enable untagged jobs or make the runner unprotected to drain a queue.
A job obtains host-root access from the Docker executor.
Likely causeprivileged mode, host Docker socket, dangerous device/capability/PID mounts, broad host volumes, or runner-user Docker access exposed the daemon.
Isolate the runner host and inspect config.toml, container inspect data, runner groups, daemon events, and host changes from a trusted system.Treat any /var/run/docker.sock mount or docker-group execution as host-root compromise.
ResolutionDestroy/rebuild the dedicated host, rotate reachable tokens, set privileged false, remove sockets/devices/host namespaces, and move privileged builds to a separate disposable trust boundary.
A job pulls an unapproved or changed image.
Likely causeallowed_images is broad, a mutable tag was used without digest, pull policy reused stale local content, or registry policy was bypassed.
Inspect the job's resolved image digest, runner allowed_images/allowed_services, registry audit, and pull policy.Compare the digest with the reviewed dependency lock and promotion record.
ResolutionDeny the job, pin the approved digest, narrow registry namespace patterns, use always pull for shared trust where appropriate, and review the substituted image.
Protected and unprotected jobs read the same cache content.
Likely causeRunner/project settings allow shared cache keys, a common bucket/prefix crosses trust boundaries, Shared is enabled, or cache key design omits project/protection context.
Inspect cache object path/metadata, project protected-cache setting, runner token/project ID, key, and bucket policy without downloading untrusted artifacts into a privileged job.Run synthetic distinct markers from protected and unprotected branches.
ResolutionDisable shared cache, separate runner/bucket/prefix by trust boundary, rotate/delete poisoned entries, use project/protection-scoped keys, and treat cache as untrusted input.
Registration succeeds but config.toml contains an unexpected runner or token.
Likely causeRepeated registration appended another stanza, the legacy registration workflow was used, or provisioning logged/reused a runner authentication token.
Run gitlab-runner list, inspect sanitized stanza names/system IDs, and compare the GitLab runner-manager inventory.Do not copy the token line into tickets or source control.
ResolutionPause the runners, remove/unregister unintended managers through the supported UI/API, rotate the authentication token, and rebuild a single reviewed configuration.
Jobs fail with permission errors in /builds or /cache.
Likely causeHost-bound directory ownership, SELinux labels, runner upgrade storage naming, or a nonroot build image conflicts with the configured mount.
Inspect mount type, UID/GID, SELinux denial events, runner version, and a synthetic file operation in the intended image.Do not disable SELinux or run privileged as the first fix.
ResolutionCorrect ownership or use supported MCS/volume configuration, preserve nonroot containers, and retest; accept cold cache after documented 18.4+ storage naming changes.
Cache or Docker storage exhausts disk/inodes.
Likely causeCaches are unbounded, clear-docker-cache is not scheduled, images/build cache are not covered by volume cleanup, or concurrency exceeds capacity.
Run clear-docker-cache space, docker system df, df -h/-i, and inspect active jobs before deletion.Review retention and object-store lifecycle separately from host Docker volumes.
ResolutionDrain the runner, delete only verified unused managed objects with label filters, add capacity alerts and bounded lifecycle, then reopen. Never prune during active jobs without scope evidence.
The runner accepts an unprotected branch or merge request unexpectedly.
Likely causeProtected was not enabled, merge-request pipeline settings allow it, project assignment is broad, or tags are reused across trust levels.
Cancel the job, preserve pipeline/ref/runner metadata, and inspect runner protected status plus ref protection.Assume exposure if job steps began and review all reachable credentials/cache/network.
ResolutionPause/isolate the runner, restrict it to protected refs and dedicated tags/projects, rotate exposure, rebuild host if code ran, and repeat negative tests.
Docker service containers can reach unintended jobs or networks.
Likely causePer-job networking is disabled/misconfigured, host networks are shared, service aliases collide, or custom networks persist.
Inspect container network IDs, membership, lifecycle labels, and firewall/egress logs for two concurrent synthetic jobs.Verify no host/bridge service remains after post-job cleanup.
ResolutionEnable supported per-job network isolation, remove persistent shared networks, narrow services, and rebuild the dedicated host after possible cross-job exposure.
Runner and GitLab versions behave incompatibly.
Likely causeRunner minor/version drift exceeds the GitLab instance's supported behavior or an upgrade changed cache/helper/config semantics.
Compare gitlab-runner --version, GitLab instance version, official upgrade notes, helper image, and config validator output.Canary the exact image and cache workflow before changing production.
ResolutionUpgrade in a reviewed sequence with matching helper packages, drain old runners, accept planned cold caches where documented, and retain the previous verified package/image for rollback.
A runner authentication token appears in logs or backup.
Likely causeNoninteractive registration passed it in argv, shell tracing captured it, config.toml was backed up unencrypted, or diagnostics printed configuration.
Restrict evidence, identify every copy/access, pause the runner, and review GitLab runner manager registrations.Do not validate by replaying the exposed token.
ResolutionRotate/delete the runner authentication token and managers, re-register interactively or through a protected secret channel, sanitize backups, and treat affected jobs/host as exposed.
Restoring an old runner backup reconnects duplicate managers.
Likely causeconfig.toml with a live token/system identity was restored instead of rebuilding and re-registering, or old and new hosts share credentials.
Keep the restored host network-isolated and compare runner manager system IDs and token rotation history.Inspect the sanitized configuration baseline, not the secret-bearing file, to plan recovery.
ResolutionDelete the duplicate manager, rotate authentication, rebuild from the verified host baseline, and register a new manager. Cache is disposable and should not drive recovery.
Reference
Frequently asked questions
Is the nonprivileged Docker executor a perfect sandbox?
No. It is materially safer than privileged mode, but all containers share a host kernel and the runner service controls Docker. Keep one trust boundary per dedicated host and patch/rebuild it.
Can protected and unprotected jobs share cache?
They should not. Separate runners, bucket/prefix, keys, and GitLab protected-cache behavior, and still treat cache as untrusted reproducible input.
Why use digest-pinned images if pull_policy is always?
Always controls freshness; the digest controls identity. A mutable tag can point to different content even when pulled every time.
Should config.toml be restored from backup?
Restore only a sanitized baseline. Rebuild the host and register a new runner manager with a rotated token rather than reviving a copied live identity.
Recovery
Rollback
Pause the runner before rollback and keep untrusted jobs on a separate pool. Reinstall the previous verified Runner/helper/Docker package set or rebuild the prior host image, but create/rotate runner authentication rather than restoring a copied live config.toml. Quarantine caches created by suspect versions or jobs.
- Pause the runner in GitLab, drain active jobs, and preserve external Runner/Docker/object-store/audit evidence.
- Delete or rotate any exposed/suspect runner manager and authentication token; isolate the old host.
- Rebuild the previous signed host/package baseline and merge sanitized config with a newly registered manager.
- Use the prior protected runner/image/cache policy; do not restore privileged mode, broad image patterns, shared cache, or if-not-present private image reuse.
- Quarantine cache written during the failed change and rebuild dependencies from locked verified sources.
- Run protected positive plus unprotected, untagged, image, privilege/socket, cross-cache, and cross-network negatives before unpausing.
Evidence