• Configuration validation
• The global section
  • log_format examples (truncated)
    • runner
    • text
    • json
  • How check_interval works
• The [session_server] section
• The [[runners]] section
  • How clone_url works
    • Modify Git LFS endpoints
  • How unhealthy_requests_limit and unhealthy_interval works
• The executors
• The shells
  • Use a POSIX-compliant shell
• The [runners.docker] section
  • The [[runners.docker.services]] section
  • Volumes in the [runners.docker] section
    • Example 1: Add a data volume
    • Example 2: Mount a host directory as a data volume
  • Use a private container registry
    • Support for GitLab integrated registry
    • Precedence of Docker authorization resolving
• The [runners.parallels] section
• The [runners.virtualbox] section
• Overriding the base VM image
• The [runners.ssh] section
• The [runners.machine] section
  • The [[runners.machine.autoscaling]] sections
  • Periods syntax
• The [runners.instance] section
• The [runners.autoscaler] section
• The [runners.autoscaler.plugin_config] section
• The [runners.autoscaler.scale_throttle] section
  • Relationship between limit and burst
• The [runners.autoscaler.connector_config] section
• The [runners.autoscaler.state_storage] section
• The [[runners.autoscaler.policy]] sections
  • Periods syntax
• The [runners.autoscaler.vm_isolation] section
• The [runners.autoscaler.vm_isolation.connector_config] section
• The [runners.custom] section
• The [runners.cache] section
  • The [runners.cache.s3] section
    • Use KMS key encryption in S3 bucket for runner cache
    • Enable multipart uploads with UploadRoleARN
      • How uploads work with UploadRoleARN
    • Enable IAM roles for Kubernetes ServiceAccount resources
  • The [runners.cache.gcs] section
  • The [runners.cache.azure] section
    • Azure workload and managed identities
• The [runners.kubernetes] section
• Helper image
  • Helper image configuration for Kubernetes on Arm
  • Runner images that use an old version of Alpine Linux
  • Alpine pwsh images
  • Helper image registry
  • Override the helper image
    • When using PowerShell Core
• The [runners.custom_build_dir] section
  • Default Build Directory
• The [runners.referees] section
  • Use the Metrics Runner referee
  • Configure the Metrics Runner Referee for GitLab Runner

Advanced configuration

To change the behavior of GitLab Runner and individual registered runners, modify the config.toml file.

You can find the config.toml file in:

• /etc/gitlab-runner/ on *nix systems when GitLab Runner is executed as root. This directory is also the path for service configuration.
• ~/.gitlab-runner/ on *nix systems when GitLab Runner is executed as non-root.
• ./ on other systems.

GitLab Runner does not require a restart when you change most options. This includes parameters in the [[runners]] section and most parameters in the global section, except for listen_address. If a runner was already registered, you don’t need to register it again.

GitLab Runner checks for configuration modifications every 3 seconds and reloads if necessary. GitLab Runner also reloads the configuration in response to the SIGHUP signal.

Configuration validation

Configuration validation is a process that checks the structure of the config.toml file. The output from the configuration validator provides only info level messages.

The configuration validation process is for informational purposes only. You can use the output to to identify potential issues with your runner configuration. The configuration validation might not catch all possible problems, and the absence of messages does not guarantee that the config.toml file is flawless.

The global section

These settings are global. They apply to all runners.

Here’s a configuration example:

# Example `config.toml` file 

concurrent = 100 # A global setting for job concurrency that applies to all runner sections defined in this `config.toml` file
log_level = "warning"
log_format = "info"
check_interval = 3 # Value in seconds

[[runners]]
  name = "first"
  url = "Your Gitlab instance URL (for example, `https://gitlab.com`)"
  executor = "shell"
  (...)

[[runners]]
  name = "second"
  url = "Your Gitlab instance URL (for example, `https://gitlab.com`)"
  executor = "docker"
  (...)

  [[runners]]
  name = "third"
  url = "Your Gitlab instance URL (for example, `https://gitlab.com`)"
  executor = "docker-autoscaler"
  (...)

log_format examples (truncated)

runner

Runtime platform                                    arch=amd64 os=darwin pid=37300 revision=HEAD version=development version
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
WARNING: Running in user-mode.
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0

text

INFO[0000] Runtime platform                              arch=amd64 os=darwin pid=37773 revision=HEAD version="development version"
INFO[0000] Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
WARN[0000] Running in user-mode.
WARN[0000] Use sudo for system-mode:
WARN[0000] $ sudo gitlab-runner...
INFO[0000]
INFO[0000] Configuration loaded                          builds=0
INFO[0000] listen_address not defined, metrics & debug endpoints disabled  builds=0
INFO[0000] [session_server].listen_address not defined, session endpoints disabled  builds=0

json

{"arch":"amd64","level":"info","msg":"Runtime platform","os":"darwin","pid":38229,"revision":"HEAD","time":"2020-06-05T15:57:35+02:00","version":"development version"}
{"builds":0,"level":"info","msg":"Starting multi-runner from /etc/gitlab-runner/config.toml...","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"Running in user-mode.","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"Use sudo for system-mode:","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"$ sudo gitlab-runner...","time":"2020-06-05T15:57:35+02:00"}
{"level":"info","msg":"","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"Configuration loaded","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"listen_address not defined, metrics \u0026 debug endpoints disabled","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"[session_server].listen_address not defined, session endpoints disabled","time":"2020-06-05T15:57:35+02:00"}

How check_interval works

If config.toml has more than one [[runners]] section, GitLab Runner contains a loop that constantly schedules job requests to the GitLab instance where GitLab Runner is configured.

The following example has check_interval of 10 seconds and two [[runners]] sections (runner-1 and runner-2). GitLab Runner sends a request every 10 seconds and sleeps for five seconds:

1. Get check_interval value (10s).
2. Get list of runners (runner-1, runner-2).
3. Calculate the sleep interval (10s / 2 = 5s).
4. Start an infinite loop:
   1. Request a job for runner-1.
   2. Sleep for 5s.
   3. Request a job for runner-2.
   4. Sleep for 5s.
   5. Repeat.

Here’s a check_interval configuration example:

# Example `config.toml` file 

concurrent = 100 # A global setting for job concurrency that applies to all runner sections defined in this `config.toml` file.
log_level = "warning"
log_format = "info"
check_interval = 10 # Value in seconds

[[runners]]
  name = "runner-1"
  url = "Your Gitlab instance URL (for example, `https://gitlab.com`)"
  executor = "shell"
  (...)

[[runners]]
  name = "runner-2"
  url = "Your Gitlab instance URL (for example, `https://gitlab.com`)"
  executor = "docker"
  (...)

In this example, a job request from the runner’s process is made every five seconds. If runner-1 and runner-2 are connected to the same GitLab instance, this GitLab instance also receives a new request from this runner every five seconds.

Two sleep periods occur between the first and second requests for runner-1. Each period takes five seconds, so it’s approximately 10 seconds between subsequent requests for runner-1. The same applies for runner-2.

If you define more runners, the sleep interval is smaller. However, a request for a runner is repeated after all requests for the other runners and their sleep periods are called.

The [session_server] section

The [session_server] section lets users interact with jobs, for example, in the interactive web terminal.

The [session_server] section should be specified at the root level, not per runner. It should be defined outside the [[runners]] section.

# Example `config.toml` file with session server configured

concurrent = 100 # A global setting for job concurrency that applies to all runner sections defined in this `config.toml` file
log_level = "warning"
log_format = "info"
check_interval = 3 # Value in seconds

[session_server]
  listen_address = "[::]:8093" # Listen on all available interfaces on port `8093`
  advertise_address = "runner-host-name.tld:8093"
  session_timeout = 1800

When you configure the [session_server] section:

• For listen_address and advertise_address, use the format host:port, where host is the IP address (127.0.0.1:8093) or domain (my-runner.example.com:8093). The runner uses this information to create a TLS certificate for a secure connection.
• Ensure your web browser can connect to the advertise_address. Live sessions are initiated by the web browser.
• Ensure that advertise_address is a public IP address, unless you have enabled the application setting, allow_local_requests_from_web_hooks_and_services.

To disable the session server and terminal support, delete the [session_server] section.

If you are using the GitLab Runner Docker image, you must expose port 8093 by adding -p 8093:8093 to your docker run command.

The [[runners]] section

Each [[runners]] section defines one runner.

Example:

[[runners]]
  name = "ruby-3.3-docker"
  url = "https://CI/"
  token = "TOKEN"
  limit = 0
  executor = "docker"
  builds_dir = ""
  shell = ""
  environment = ["ENV=value", "LC_ALL=en_US.UTF-8"]
  clone_url = "http://gitlab.example.local"

How clone_url works

When the GitLab instance is available at a URL that the runner can’t use, you can configure a clone_url.

For example, a firewall might prevent the runner from reaching the URL. If the runner can reach the node on 192.168.1.23, set the clone_url to http://192.168.1.23.

If the clone_url is set, the runner constructs a clone URL in the form of http://gitlab-ci-token:s3cr3tt0k3n@192.168.1.23/namespace/project.git.

Modify Git LFS endpoints

To modify Git LFS endpoints, set pre_get_sources_script in one of the following files:

• config.toml:

  pre_get_sources_script = "mkdir -p $RUNNER_TEMP_PROJECT_DIR/git-template; git config -f $RUNNER_TEMP_PROJECT_DIR/git-template/config lfs.url https://<alternative-endpoint>"
  

• .gitlab-ci.yml:

  default:
    hooks:
      pre_get_sources_script:
        - mkdir -p $RUNNER_TEMP_PROJECT_DIR/git-template
        - git config -f $RUNNER_TEMP_PROJECT_DIR/git-template/config lfs.url https://localhost
  

How unhealthy_requests_limit and unhealthy_interval works

When a GitLab instance is unavailable for an extended period of time (for example, during a version upgrade), the runners configured for that instance become idle and do not resume job processing for 30-60 minutes after the GitLab instance is available again.

To increase or decrease the duration that runners are idle, change the unhealthy_interval setting.

To change the number of times the runner attempts to connect to the GitLab server and receive an unhealthy sleep before becoming idle, change the unhealthy_requests_limit setting. For more information, see How check_interval works.

The executors

The following executors are available.

The shells

CI/CD jobs run locally on the host machine when configured to use the shell executor. The supported operating system shells are:

When the shell option is set to bash or sh, Bash’s ANSI-C quoting is used to shell escape job scripts.

Use a POSIX-compliant shell

In GitLab Runner 14.9 and later, enable the feature flag named FF_POSIXLY_CORRECT_ESCAPES to use a POSIX-compliant shell (like dash). When enabled, “Double Quotes”, which is POSIX-compliant shell escaping mechanism, is used.

The [runners.docker] section

The following settings define the Docker container parameters. This is applicable when the runner is configured to use the Docker executor.

Docker-in-Docker as a service, or any container runtime configured inside a job, does not inherit these parameters.

The [[runners.docker.services]] section

Specify additional services to run with the job. For a list of available images, see the Docker Registry. Each service runs in a separate container and is linked to the job.

Example:

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:3.3"
  memory = "128m"
  memory_swap = "256m"
  memory_reservation = "64m"
  oom_kill_disable = false
  cpuset_cpus = "0,1"
  cpuset_mems = "0,1"
  cpus = "2"
  dns = ["8.8.8.8"]
  dns_search = [""]
  service_memory = "128m"
  service_memory_swap = "256m"
  service_memory_reservation = "64m"
  service_cpuset_cpus = "0,1"
  service_cpus = "2"
  services_limit = 5
  privileged = false
  group_add = ["docker"]
  userns_mode = "host"
  cap_add = ["NET_ADMIN"]
  cap_drop = ["DAC_OVERRIDE"]
  devices = ["/dev/net/tun"]
  disable_cache = false
  wait_for_services_timeout = 30
  cache_dir = ""
  volumes = ["/data", "/home/project/cache"]
  extra_hosts = ["other-host:127.0.0.1"]
  shm_size = 300000
  volumes_from = ["storage_container:ro"]
  links = ["mysql_container:mysql"]
  allowed_images = ["ruby:*", "python:*", "php:*"]
  allowed_services = ["postgres:9", "redis:*", "mysql:*"]
  [runners.docker.ulimit]
    "rtprio" = "99"
  [[runners.docker.services]]
    name = "registry.example.com/svc1"
    alias = "svc1"
    entrypoint = ["entrypoint.sh"]
    command = ["executable","param1","param2"]
    environment = ["ENV1=value1", "ENV2=value2"]
  [[runners.docker.services]]
    name = "redis:2.8"
    alias = "cache"
  [[runners.docker.services]]
    name = "postgres:9"
    alias = "postgres-db"
  [runners.docker.sysctls]
    "net.ipv4.ip_forward" = "1"

Volumes in the [runners.docker] section

For more information about volumes, see the Docker documentation.

The following examples show how to specify volumes in the [runners.docker] section.

Example 1: Add a data volume

A data volume is a specially-designated directory in one or more containers that bypasses the Union File System. Data volumes are designed to persist data, independent of the container’s life cycle.

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:3.3"
  privileged = false
  disable_cache = true
  volumes = ["/path/to/volume/in/container"]

This example creates a new volume in the container at /path/to/volume/in/container.

Example 2: Mount a host directory as a data volume

When you want to store directories outside the container, you can mount a directory from your Docker daemon’s host into a container:

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:3.3"
  privileged = false
  disable_cache = true
  volumes = ["/path/to/bind/from/host:/path/to/bind/in/container:rw"]

This example uses /path/to/bind/from/host of the CI/CD host in the container at /path/to/bind/in/container.

GitLab Runner 11.11 and later mount the host directory for the defined services as well.

Use a private container registry

To use private registries as a source of images for your jobs, configure authorization with the CI/CD variable DOCKER_AUTH_CONFIG. You can set the variable in one of the following:

• The CI/CD settings of the project as the file type
• The config.toml file

Using private registries with the if-not-present pull policy may introduce security implications. For more information about how pull policies work, see Configure how runners pull images.

For more information about using private container registries, see:

• Access an image from a private container registry
• .gitlab-ci.yml keyword reference

The steps performed by the runner can be summed up as:

1. The registry name is found from the image name.
2. If the value is not empty, the executor searches for the authentication configuration for this registry.
3. Finally, if an authentication corresponding to the specified registry is found, subsequent pulls makes use of it.

Support for GitLab integrated registry

GitLab sends credentials for its integrated registry along with the job’s data. These credentials are automatically added to the registry’s authorization parameters list.

After this step, authorization against the registry proceeds similarly to configuration added with the DOCKER_AUTH_CONFIG variable.

In your jobs, you can use any image from your GitLab integrated registry, even if the image is private or protected. For information on the images jobs have access to, read the CI/CD job token documentation documentation.

Precedence of Docker authorization resolving

As described earlier, GitLab Runner can authorize Docker against a registry by using credentials sent in different way. To find a proper registry, the following precedence is taken into account:

1. Credentials configured with DOCKER_AUTH_CONFIG.
2. Credentials configured locally on the GitLab Runner host with ~/.docker/config.json or ~/.dockercfg files (for example, by running docker login on the host).
3. Credentials sent by default with a job’s payload (for example, credentials for the integrated registry described earlier).

The first credentials found for the registry are used. So for example, if you add credentials for the integrated registry with the DOCKER_AUTH_CONFIG variable, then the default credentials are overridden.

The [runners.parallels] section

The following parameters are for Parallels.

Example:

[runners.parallels]
  base_name = "my-parallels-image"
  template_name = ""
  disable_snapshots = false

The [runners.virtualbox] section

The following parameters are for VirtualBox. This executor relies on the vboxmanage executable to control VirtualBox machines, so you have to adjust your PATH environment variable on Windows hosts: PATH=%PATH%;C:\Program Files\Oracle\VirtualBox.

Example:

[runners.virtualbox]
  base_name = "my-virtualbox-image"
  base_snapshot = "my-image-snapshot"
  disable_snapshots = false
  start_type = "headless"

The start_type parameter determines the graphical front end used when starting the virtual image. Valid values are headless (default), gui or separate as supported by the host and guest combination.

Overriding the base VM image

For both the Parallels and VirtualBox executors, you can override the base VM name specified by base_name. To do this, use the image parameter in the .gitlab-ci.yml file.

For backward compatibility, you cannot override this value by default. Only the image specified by base_name is allowed.

To allow users to select a VM image by using the .gitlab-ci.yml image parameter:

[runners.virtualbox]
  ...
  allowed_images = [".*"]

In the example, any existing VM image can be used.

The allowed_images parameter is a list of regular expressions. Configuration can be as precise as required. For instance, if you want to allow only certain VM images, you can use regex like:

[runners.virtualbox]
  ...
  allowed_images = ["^allowed_vm[1-2]$"]

In this example, only allowed_vm1 and allowed_vm2 are allowed. Any other attempts will result in an error.

The [runners.ssh] section

The following parameters define the SSH connection.

Example:

[runners.ssh]
  host = "my-production-server"
  port = "22"
  user = "root"
  password = "production-server-password"
  identity_file = ""

The [runners.machine] section

The following parameters define the Docker Machine-based autoscaling feature. For more information, see Docker Machine Executor autoscale configuration.

The [[runners.machine.autoscaling]] sections

The following parameters define the configuration available when using the Instance or Docker Autoscaler executor.

Example:

[runners.machine]
  IdleCount = 5
  IdleTime = 600
  MaxBuilds = 100
  MachineName = "auto-scale-%s"
  MachineDriver = "google" # Refer to Docker Machine docs on how to authenticate: https://docs.docker.com/machine/drivers/gce/#credentials
  MachineOptions = [
      # Additional machine options can be added using the Google Compute Engine driver.
      # If you experience problems with an unreachable host (ex. "Waiting for SSH"),
      # you should remove optional parameters to help with debugging.
      # https://docs.docker.com/machine/drivers/gce/
      "google-project=GOOGLE-PROJECT-ID",
      "google-zone=GOOGLE-ZONE", # e.g. 'us-central1-a', full list in https://cloud.google.com/compute/docs/regions-zones/
  ]
  [[runners.machine.autoscaling]]
    Periods = ["* * 9-17 * * mon-fri *"]
    IdleCount = 50
    IdleCountMin = 5
    IdleScaleFactor = 1.5 # Means that current number of Idle machines will be 1.5*in-use machines,
                          # no more than 50 (the value of IdleCount) and no less than 5 (the value of IdleCountMin)
    IdleTime = 3600
    Timezone = "UTC"
  [[runners.machine.autoscaling]]
    Periods = ["* * * * * sat,sun *"]
    IdleCount = 5
    IdleTime = 60
    Timezone = "UTC"

Periods syntax

The Periods setting contains an array of string patterns of time periods represented in a cron-style format. The line contains following fields:

[second] [minute] [hour] [day of month] [month] [day of week] [year]

Like in the standard cron configuration file, the fields can contain single values, ranges, lists, and asterisks. View a detailed description of the syntax.

The [runners.instance] section

The [runners.autoscaler] section

The following parameters configure the autoscaler feature. You can only use these parameters with the Instance and Docker Autoscaler executors.

The [runners.autoscaler.plugin_config] section

This hash table is re-encoded to JSON and passed directly to the configured plugin.

fleeting plugins typically have accompanying documentation on the supported configuration.

The [runners.autoscaler.scale_throttle] section

Relationship between limit and burst

The scale throttle uses a token quota system to create instances. This system is defined by two values:

• burst: The maximum size of the quota.
• limit: The rate at which the quota refreshes per second.

The number of instances you can create at once depends on your remaining quota. If you have sufficient quota, you can create instances up to that amount. If the quota is depleted, you can create limit instances per second. When instance creation stops, the quota increases by limit per second until it reaches the burst value.

For example, if limit is 1 and burst is 60:

• You can create 60 instances instantly, but you’re throttled.
• If you wait 60 seconds, you can instantly create another 60 instances.
• If you do not wait, you can create 1 instance every second.

The [runners.autoscaler.connector_config] section

fleeting plugins typically have accompanying documentation on the supported connection options.

Plugins automatically update the connector configuration. You can use the [runners.autoscaler.connector_config] to override automatic update of the connector configuration, or to fill in the empty values that the plugin cannot determine.

The [runners.autoscaler.state_storage] section

If GitLab Runner starts when state storage is disabled (default), the existing fleeting instances are removed immediately for safety reasons. With configurations such as max_use_count: 1, we might inadvertently assign a job to an instance that’s already been used if we don’t know its usage status.

Enabling the state storage feature allows an instance’s state to persist on the local disk. In this case, if an instance exists when GitLab Runner starts, it is not deleted. Its cached connection details, use count, and other configurations are restored.

Consider the following information when enabling the state storage feature:

• The authentication details for an instance (username, password, keys) remain in the disk.

• If an instance is restored when it is actively running a job, GitLab Runner removes it by default. This behavior ensures safety, as GitLab Runner cannot resume jobs. To keep the instance, set keep_instance_with_acquisitions to true.

  Setting keep_instance_with_acquisitions to true is useful when you’re not concerned about ongoing jobs on the instance. It is also useful to clean the environment. For example, you can use the instance_ready_command configuration option to clean the environment to keep the instance. This might involve stopping all executing commands or forcefully removing Docker containers.

The [[runners.autoscaler.policy]] sections

Note - idle_count in this context refers to the number of jobs, not the number of autoscaled machines as in the legacy autoscaling method.

When scale_factor is set, idle_count becomes the minimum idle capacity and the scaler_factor_limit the maximum idle capacity.

You can define multiple policies. The last matching policy is the one used.

In the following example, the idle count 1 is used between 08:00 and 15:59, Monday through Friday. Otherwise, the idle count is 0.

[[runners.autoscaler.policy]]
  idle_count        = 0
  idle_time         = "0s"
  periods           = ["* * * * *"]

[[runners.autoscaler.policy]]
  idle_count        = 1
  idle_time         = "30m0s"
  periods           = ["* 8-15 * * mon-fri"]

Periods syntax

The periods setting contains an array of unix-cron formatted strings to denote the period a policy is enabled for. The cron format consists of 5 fields:

 ┌────────── minute (0 - 59)
 │ ┌──────── hour (0 - 23)
 │ │ ┌────── day of month (1 - 31)
 │ │ │ ┌──── month (1 - 12)
 │ │ │ │ ┌── day of week (1 - 7 or MON-SUN, 0 is an alias for Sunday)
 * * * * *

• - can be used between two numbers to specify a range.
• * can be used to represent the whole range of valid values for that field.
• / followed by a number or can be used after a range to skip that number through the range. For example, 0-12/2 for the hour field would activate the period every 2 hours between the hours of 00:00 and 00:12.
• , can be used to separate a list of valid numbers or ranges for the field. For example, 1,2,6-9.

It’s worth keeping in mind that this cron job represents a range in time. For example:

The [runners.autoscaler.vm_isolation] section

VM Isolation uses nesting, which is currently only supported on MacOS.

The [runners.autoscaler.vm_isolation.connector_config] section

The parameters for the [runners.autoscaler.vm_isolation.connector_config] section are identical to the [runners.autoscaler.connector_config] section, but are used to connect to the nesting provisioned virtual machine, rather than the autoscaled instance.

The [runners.custom] section

The following parameters define configuration for the custom executor.

The [runners.cache] section

The following parameters define the distributed cache feature. View details in the runner autoscale documentation.

The cache mechanism uses pre-signed URLs to upload and download cache. URLs are signed by GitLab Runner on its own instance. It does not matter if the job’s script (including the cache upload/download script) are executed on local or external machines. For example, shell or docker executors run their scripts on the same machine where the GitLab Runner process is running. At the same time, virtualbox or docker+machine connects to a separate VM to execute the script. This process is for security reasons: minimizing the possibility of leaking the cache adapter’s credentials.

If the S3 cache adapter is configured to use an IAM instance profile, the adapter uses the profile attached to the GitLab Runner machine. Similarly for GCS cache adapter, if configured to use the CredentialsFile. The file needs to be present on the GitLab Runner machine.

This table lists config.toml, CLI options, and ENV variables for register.

The [runners.cache.s3] section

The following parameters define S3 storage for cache.

Example:

[runners.cache]
  Type = "s3"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.s3]
    ServerAddress = "s3.amazonaws.com"
    AccessKey = "AWS_S3_ACCESS_KEY"
    SecretKey = "AWS_S3_SECRET_KEY"
    BucketName = "runners-cache"
    BucketLocation = "eu-west-1"
    Insecure = false
    ServerSideEncryption = "KMS"
    ServerSideEncryptionKeyID = "alias/my-key"

If any of ServerAddress, AccessKey or SecretKey aren’t specified and AuthenticationType is not provided, the S3 client uses the IAM instance profile available to the gitlab-runner instance. In an autoscale configuration, this is not the on-demand machine that jobs are executed on. If ServerAddress, AccessKey and SecretKey are all specified but AuthenticationType is not provided, access-key will be used as the authentication type.

When you use Helm charts to install GitLab Runner, and rbac.create is set to true in the values.yaml file, a ServiceAccount is created. This ServiceAccount’s annotations are retrieved from the rbac.serviceAccountAnnotations section.

For runners on Amazon EKS, you can specify an IAM role to assign to the service account. The specific annotation needed is: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<IAM_ROLE_NAME>.

The IAM policy for this role must have permissions to do the following actions for the specified bucket:

• s3:PutObject
• s3:GetObjectVersion
• s3:GetObject
• s3:DeleteObject

If you use ServerSideEncryption of type KMS, this role must also have permission to do the following actions for the specified AWS KMS Key:

• kms:Encrypt
• kms:Decrypt
• kms:ReEncrypt*
• kms:GenerateDataKey*
• kms:DescribeKey

ServerSideEncryption of type SSE-C is not supported. SSE-C requires that the headers, which contain the user-supplied key, are provided for the download request, in addition to the presigned URL. This would mean passing the key material to the job, where the key can’t be kept safe. This does have the potential to leak the decryption key. A discussion about this issue is in this merge request.

Use KMS key encryption in S3 bucket for runner cache

The GenerateDataKey API uses the KMS symmetric key to create a data key for client-side encryption (https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey.html). KMS key configuration must be as follows:

The IAM policy for the role assigned to the ServiceAccount defined in rbac.serviceAccountName must have permissions to do the following actions for the KMS Key:

• kms:GetPublicKey
• kms:Decrypt
• kms:Encrypt
• kms:DescribeKey
• kms:GenerateDataKey

Enable multipart uploads with UploadRoleARN

To limit access to the cache, the runner manager generates timed-limited, presigned URLs for jobs to download from and upload to the cache. However, AWS S3 limits a single PUT request to 5 GB. For files larger than 5 GB, you must use the multipart upload API.

Multipart uploads are only supported with AWS S3 and not for other S3 providers. Because the runner manager handles jobs for different projects, the runner manager cannot pass around S3 credentials that have bucket-wide permissions. Instead, the runner manger uses time-limited presigned URLs and narrowly-scoped credentials to restrict access to one specific object.

To use S3 multipart uploads with AWS, specify an IAM role in UploadRoleARN in the arn:aws:iam:::<ACCOUNT ID>:<YOUR ROLE NAME> format. This role generates time-limited AWS credentials that are narrowly scoped to write to a specific blob in the bucket. Ensure that your original S3 credentials can access AssumeRole for the specified UploadRoleARN.

The IAM role specified in UploadRoleARN must have the following permissions:

• s3:PutObject access to the bucket specified in BucketName.
• kms:Decrypt and kms:GenerateDataKey if server side encryption with KMS or DSSE-KMS is enabled.

For example, suppose you have an IAM role called my-instance-role attached to an EC2 instance with the ARN arn:aws:iam::1234567890123:role/my-instance-role.

You can create a new role arn:aws:iam::1234567890123:role/my-upload-role that only has s3:PutObject permissions for BucketName. In the AWS settings for my-instance-role, the Trust relationships might look similar to this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::1234567890123:role/my-upload-role"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

You can also reuse my-instance-role as the UploadRoleARN and avoid creating a new role. Make sure that my-instance-role has the AssumeRole permission. For example, an IAM profile associated with an EC2 instance might have the following Trust relationships:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com",
                "AWS": "arn:aws:iam::1234567890123:role/my-instance-role"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

You can use the AWS command-line interface to verify that your instance has the AssumeRole permission. For example:

aws sts assume-role --role-arn arn:aws:iam::1234567890123:role/my-upload-role --role-session-name gitlab-runner-test1

How uploads work with UploadRoleARN

If UploadRoleARN is present, every time the runner uploads to the cache:

1. The runner manager retrieves the original S3 credentials (specified through AuthenticationType, AccessKey, and SecretKey).

2. With the S3 credentials, the runner manager sends a request to the Amazon Security Token Service (STS) for AssumeRole with UploadRoleARN. The policy request looks similar to this:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": ["s3:PutObject"],
                "Resource": "arn:aws:s3:::<YOUR-BUCKET-NAME>/<CACHE-FILENAME>"
            }
        ]
    }
   

3. If the request is successful, the runner manager obtains temporary AWS credentials with a restricted session.
4. The runner manager passes these credentials and URL in the s3://<bucket name>/<filename> format to the cache archiver, which then uploads the file.

Enable IAM roles for Kubernetes ServiceAccount resources

To use IAM roles for service accounts, an IAM OIDC provider must exist for your cluster. After an IAM OIDC provider is associated with your cluster, you can create an IAM role to associate to the service account of the runner.

1. On the Create Role window, under Select type of trusted entity, select Web Identity.

2. On the Trusted Relationships tab of the role:

   • The Trusted entities section must have the format: arn:aws:iam::<ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>. The OIDC ID can be found on EKS cluster’s Configuration tab.

   • The Condition section must have the GitLab Runner service account defined in rbac.serviceAccountName or the default service account created if rbac.create is set to true:

     Condition	Key	Value
     StringEquals	oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub	system:serviceaccount:<GITLAB_RUNNER_NAMESPACE>:<GITLAB_RUNNER_SERVICE_ACCOUNT>

The [runners.cache.gcs] section

The following parameters define native support for Google Cloud Storage. For more information about these values, see the Google Cloud Storage (GCS) authentication documentation.

Examples:

Credentials configured directly in config.toml file:

[runners.cache]
  Type = "gcs"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.gcs]
    AccessID = "cache-access-account@test-project-123456.iam.gserviceaccount.com"
    PrivateKey = "-----BEGIN PRIVATE KEY-----\nXXXXXX\n-----END PRIVATE KEY-----\n"
    BucketName = "runners-cache"

Credentials in JSON file downloaded from GCP:

[runners.cache]
  Type = "gcs"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.gcs]
    CredentialsFile = "/etc/gitlab-runner/service-account.json"
    BucketName = "runners-cache"

Application Default Credentials (ADC) from the metadata server in GCP:

When you use GitLab Runner with Google Cloud ADC, you typically use the default service account. Then you don’t need to supply credentials for the instance:

[runners.cache]
  Type = "gcs"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.gcs]
    BucketName = "runners-cache"

If you use ADC, be sure that the service account that you use has the iam.serviceAccounts.signBlob permission. Typically this is done by granting the Service Account Token Creator role to the service account.

The [runners.cache.azure] section

The following parameters define native support for Azure Blob Storage. To learn more, view the Azure Blob Storage documentation. While S3 and GCS use the word bucket for a collection of objects, Azure uses the word container to denote a collection of blobs.

Example:

[runners.cache]
  Type = "azure"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.azure]
    AccountName = "<AZURE STORAGE ACCOUNT NAME>"
    AccountKey = "<AZURE STORAGE ACCOUNT KEY>"
    ContainerName = "runners-cache"
    StorageDomain = "blob.core.windows.net"

Azure workload and managed identities

To use Azure workload or managed identities, omit AccountKey from the configuration. When AccountKey is blank, the runner attempts to:

1. Obtain temporary credentials via DefaultAzureCredential.
2. Get a User Delegation Key.
3. Generate a SAS token with that key to access a Storage Account blob.

Ensure that the instance has the Storage Blob Data Contributor role assigned to it. If the instance does not have access to perform the actions above, GitLab Runner reports an AuthorizationPermissionMismatch error.

The [runners.kubernetes] section

The following table lists configuration parameters available for the Kubernetes executor. For more parameters, see the documentation for the Kubernetes executor.

Example:

[runners.kubernetes]
  host = "https://45.67.34.123:4892"
  cert_file = "/etc/ssl/kubernetes/api.crt"
  key_file = "/etc/ssl/kubernetes/api.key"
  ca_file = "/etc/ssl/kubernetes/ca.crt"
  image = "golang:1.8"
  privileged = true
  allow_privilege_escalation = true
  image_pull_secrets = ["docker-registry-credentials", "optional-additional-credentials"]
  allowed_images = ["ruby:*", "python:*", "php:*"]
  allowed_services = ["postgres:9.4", "postgres:latest"]
  logs_base_dir = "/tmp"
  scripts_base_dir = "/tmp"
  [runners.kubernetes.node_selector]
    gitlab = "true"

Helper image

When you use docker, docker+machine, or kubernetes executors, GitLab Runner uses a specific container to handle Git, artifacts, and cache operations. This container is created from an image named helper image.

The helper image is available for amd64, arm, arm64, s390x, and ppc64le architectures. It contains a gitlab-runner-helper binary, which is a special compilation of GitLab Runner binary. It contains only a subset of available commands, as well as Git, Git LFS and SSL certificates store.

The helper image has a few flavors: alpine, alpine3.17, alpine3.18, alpine3.19, alpine-latest, ubi-fips and ubuntu. The alpine image is currently the default due to its small footprint but can have DNS issues in some environments. Using helper_image_flavor = "ubuntu" will select the ubuntu flavor of the helper image.

In GitLab Runner 16.1 to 17.1, the alpine flavor is an alias for alpine3.18. In GitLab Runner 17.2 and later, it’s an alias for alpine3.19.

The alpine-latest flavor uses alpine:latest as its base image, which could potentially mean it will be more unstable.

When GitLab Runner is installed from the DEB/RPM packages, images for the supported architectures are installed on the host. When the runner prepares to execute the job, if the image in the specified version (based on the runner’s Git revision) is not found on Docker Engine, it is automatically loaded. Both the docker and docker+machine executors work this way.

For the alpine flavors, only the default alpine flavor image is included in the package. All other flavors will be downloaded from the registry.

The kubernetes executor and manual installations of GitLab Runner work differently.

• For manual installations, the gitlab-runner-helper binary is not included.
• For the kubernetes executor, the Kubernetes API doesn’t allow the gitlab-runner-helper image to be loaded from a local archive.

In both cases, GitLab Runner downloads the helper image. The GitLab Runner revision and architecture define which tag to download.

Helper image configuration for Kubernetes on Arm

To use the arm64 helper image on arm64 Kubernetes clusters, set the following values in your configuration file.

[runners.kubernetes]
        helper_image = "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:arm64-latest"

Runner images that use an old version of Alpine Linux

Images are built with multiple versions of Alpine Linux, so you can use a newer version of Alpine, but at the same time use older versions as well.

For the helper image, change the helper_image_flavor or read the Helper image section.

For the GitLab Runner image, follow the same logic, where alpine, alpine3.16, alpine3.17, alpine3.18, alpine3.19 or alpine-latest is used as a prefix in the image, before the version:

docker pull gitlab/gitlab-runner:alpine3.19-v16.1.0

Alpine pwsh images

As of GitLab Runner 16.1 and later all alpine helper images have a pwsh variant. The only exception being alpine-latest as the pwsh Docker images which the GitLab Runner helper images are based on do not support alpine:latest.

Example:

docker pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:alpine3.18-x86_64-v16.1.0-pwsh

Helper image registry

In GitLab 15.0 and earlier, you configure helper images to use images from Docker Hub.

In GitLab 15.1 and later, the helper image is pulled from the GitLab Container Registry.

To retrieve the base gitlab-runner-helper image from the GitLab registry, use a helper-image value: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}.

Self-managed instances also pull the helper image from the GitLab Container Registry on GitLab.com. To check the status of the GitLab Container Registry, see the GitLab System Status.

Override the helper image

In some cases, you may need to override the helper image. There are many reasons for doing this:

1. To speed up jobs execution: In environments with slower internet connection, downloading the same image multiple times can increase the time it takes to execute a job. Downloading the helper image from a local registry, where the exact copy of registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:XYZ is stored, can speed things up.

2. Security concerns: You may not want to download external dependencies that were not checked before. There might be a business rule to use only dependencies that were reviewed and stored in local repositories.

3. Build environments without internet access: If you have Kubernetes clusters installed in an offline environment, you can use a local image registry or package repository to pull images used in CI/CD jobs.

4. Additional software: You may want to install some additional software to the helper image, like openssh to support submodules accessible with git+ssh instead of git+http.

In these cases, you can configure a custom image by using the helper_image configuration field, which is available for the docker, docker+machine, and kubernetes executors:

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:tag"

The version of the helper image should be considered to be strictly coupled with the version of GitLab Runner. One of the main reasons for providing these images is that GitLab Runner is using the gitlab-runner-helper binary. This binary is compiled from part of the GitLab Runner source. This binary uses an internal API that is expected to be the same in both binaries.

By default, GitLab Runner references a registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:XYZ image, where XYZ is based on the GitLab Runner architecture and Git revision. In GitLab Runner 11.3 and later, you can define the image version by using one of the version variables:

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}"

With this configuration, GitLab Runner instructs the executor to use the image in version x86_64-v${CI_RUNNER_VERSION}, which is based on its compilation data. After updating GitLab Runner to a new version, GitLab Runner tries to download the proper image. The image should be uploaded to the registry before upgrading GitLab Runner, otherwise the jobs start failing with a “No such image” error.

In GitLab Runner 13.2 and later, the helper image is tagged by $CI_RUNNER_VERSION in addition to $CI_RUNNER_REVISION. Both tags are valid and point to the same image.

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}"

When using PowerShell Core

An additional version of the helper image for Linux, which contains PowerShell Core, is published with the registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:XYZ-pwsh tag.

The [runners.custom_build_dir] section

This section defines custom build directories parameters.

This feature, if not configured explicitly, is enabled by default for kubernetes, docker, docker+machine, docker autoscaler, and instance executors. For all other executors, it is disabled by default.

This feature requires that GIT_CLONE_PATH is in a path defined in runners.builds_dir. To use the builds_dir, use the $CI_BUILDS_DIR variable.

By default, this feature is enabled only for docker and kubernetes executors, because they provide a good way to separate resources. This feature can be explicitly enabled for any executor, but use caution when you use it with executors that share builds_dir and have concurrent > 1.

Example:

[runners.custom_build_dir]
  enabled = true

Default Build Directory

GitLab Runner clones the repository to a path that exists under a base path better known as the Builds Directory. The default location of this base directory depends on the executor. For:

• Kubernetes, Docker and Docker Machine executors, it is /builds inside of the container.
• Instance, it is ~/builds in the home directory of the user configured to handle the SSH or WinRM connection to the target machine.
• Docker Autoscaler, it is /builds inside of the container.
• Shell executor, it is $PWD/builds.
• SSH, VirtualBox and Parallels executors, it is ~/builds in the home directory of the user configured to handle the SSH connection to the target machine.
• Custom executors, no default is provided and it must be explicitly configured, otherwise, the job fails.

The used Builds Directory may be defined explicitly by the user with the builds_dir setting.

GitLab Runner uses the Builds Directory for all the jobs that it runs, but nests them using a specific pattern {builds_dir}/$RUNNER_TOKEN_KEY/$CONCURRENT_PROJECT_ID/$NAMESPACE/$PROJECT_NAME. For example: /builds/2mn-ncv-/0/user/playground.

GitLab Runner does not stop you from storing things inside of the Builds Directory. For example, you can store tools inside of /builds/tools that can be used during CI execution. We HIGHLY discourage this, you should never store anything inside of the Builds Directory. GitLab Runner should have total control over it and does not provide stability in such cases. If you have dependencies that are required for your CI, we recommend installing them in some other place.

The [runners.referees] section

Use GitLab Runner referees to pass extra job monitoring data to GitLab. Referees are workers in the runner manager that query and collect additional data related to a job. The results are uploaded to GitLab as job artifacts.

Use the Metrics Runner referee

If the machine or container running the job exposes Prometheus metrics, GitLab Runner can query the Prometheus server for the entirety of the job duration. After the metrics are received, they are uploaded as a job artifact that can be used for analysis later.

Only the docker-machine executor supports the referee.

Configure the Metrics Runner Referee for GitLab Runner

Define [runner.referees] and [runner.referees.metrics] in your config.toml file within a [[runner]] section and add the following fields:

Here is a complete configuration example for node_exporter metrics:

[[runners]]
  [runners.referees]
    [runners.referees.metrics]
      prometheus_address = "http://localhost:9090"
      query_interval = 10
      metric_queries = [
        "arp_entries:rate(node_arp_entries{{selector}}[{interval}])",
        "context_switches:rate(node_context_switches_total{{selector}}[{interval}])",
        "cpu_seconds:rate(node_cpu_seconds_total{{selector}}[{interval}])",
        "disk_read_bytes:rate(node_disk_read_bytes_total{{selector}}[{interval}])",
        "disk_written_bytes:rate(node_disk_written_bytes_total{{selector}}[{interval}])",
        "memory_bytes:rate(node_memory_MemTotal_bytes{{selector}}[{interval}])",
        "memory_swap_bytes:rate(node_memory_SwapTotal_bytes{{selector}}[{interval}])",
        "network_tcp_active_opens:rate(node_netstat_Tcp_ActiveOpens{{selector}}[{interval}])",
        "network_tcp_passive_opens:rate(node_netstat_Tcp_PassiveOpens{{selector}}[{interval}])",
        "network_receive_bytes:rate(node_network_receive_bytes_total{{selector}}[{interval}])",
        "network_receive_drops:rate(node_network_receive_drop_total{{selector}}[{interval}])",
        "network_receive_errors:rate(node_network_receive_errs_total{{selector}}[{interval}])",
        "network_receive_packets:rate(node_network_receive_packets_total{{selector}}[{interval}])",
        "network_transmit_bytes:rate(node_network_transmit_bytes_total{{selector}}[{interval}])",
        "network_transmit_drops:rate(node_network_transmit_drop_total{{selector}}[{interval}])",
        "network_transmit_errors:rate(node_network_transmit_errs_total{{selector}}[{interval}])",
        "network_transmit_packets:rate(node_network_transmit_packets_total{{selector}}[{interval}])"
      ]

Metrics queries are in canonical_name:query_string format. The query string supports two variables that are replaced during execution:

For example, a shared GitLab Runner environment that uses the docker-machine executor would have a {selector} similar to node=shared-runner-123.