• Prerequisites
  • Supported images and packages
• Enable the Dependency Proxy
• View the Dependency Proxy
• Use the Dependency Proxy for Docker images
  • Authenticate with the Dependency Proxy
    • Authenticate within CI/CD
  • Store a Docker image in Dependency Proxy cache
• Clear the Dependency Proxy cache
• Docker Hub rate limits and the Dependency Proxy
  • How does GitLab know if a cached tagged image is stale?
  • Check your Docker Hub rate limit
    • Check the rate limit in a CI/CD job

Dependency Proxy

The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed upstream images.

In the case of CI/CD, the Dependency Proxy receives a request and returns the upstream image from a registry, acting as a pull-through cache.

Prerequisites

The Dependency Proxy must be enabled by an administrator.

Supported images and packages

The following images and packages are supported.

For a list of planned additions, view the direction page.

Enable the Dependency Proxy

The Dependency Proxy is disabled by default. Learn how an administrator can enable it.

View the Dependency Proxy

To view the Dependency Proxy:

• Go to your group’s Packages & Registries > Dependency Proxy.

The Dependency Proxy is not available for projects.

Use the Dependency Proxy for Docker images

You can use GitLab as a source for your Docker images.

Prerequisites:

• Your images must be stored on Docker Hub.

Authenticate with the Dependency Proxy

Because the Dependency Proxy is storing Docker images in a space associated with your group, you must authenticate against the Dependency Proxy.

Follow the instructions for using images from a private registry, but instead of using registry.example.com:5000, use your GitLab domain with no port gitlab.example.com.

For example, to manually log in:

docker login gitlab.example.com --username my_username --password my_password

You can authenticate using:

• Your GitLab username and password.
• A personal access token with the scope set to read_registry and write_registry.

Authenticate within CI/CD

Runners log in to the Dependency Proxy automatically. To pull through the Dependency Proxy, use the CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX predefined CI/CD variable:

Example pulling the latest alpine image:

# .gitlab-ci.yml
image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/alpine:latest

There are other additional predefined CI/CD variables you can also use:

• CI_DEPENDENCY_PROXY_USER: A CI/CD user for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_PASSWORD: A CI/CD password for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_SERVER: The server for logging in to the Dependency Proxy.
• CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: The image prefix for pulling images through the Dependency Proxy.

CI_DEPENDENCY_PROXY_SERVER and CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX include the server port. If you explicitly include the Dependency Proxy path, the port must be included, unless you have logged into the Dependency Proxy manually without including the port:

docker pull gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest

You can also use custom CI/CD variables to store and access your personal access token or other valid credentials.

Store a Docker image in Dependency Proxy cache

To store a Docker image in Dependency Proxy storage:

1. Go to your group’s Packages & Registries > Dependency Proxy.
2. Copy the Dependency Proxy URL.
3. Use one of these commands. In these examples, the image is alpine:latest.

4. You can also pull images by digest to specify exactly which version of an image to pull.

   • Pull an image by tag by adding the image to your .gitlab-ci.yml file:

     image: gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
     

   • Pull an image by digest by adding the image to your .gitlab-ci.yml file:

     image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/alpine@sha256:c9375e662992791e3f39e919b26f510e5254b42792519c180aad254e6b38f4dc
     

   • Manually pull the Docker image:

     docker pull gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
     

   • Add the URL to a Dockerfile:

     FROM gitlab.example.com/groupname/dependency_proxy/containers/alpine:latest
     

GitLab pulls the Docker image from Docker Hub and caches the blobs on the GitLab server. The next time you pull the same image, GitLab gets the latest information about the image from Docker Hub, but serves the existing blobs from the GitLab server.

Clear the Dependency Proxy cache

Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be stored.

To reclaim disk space used by image blobs that are no longer needed, use the Dependency Proxy API.

Docker Hub rate limits and the Dependency Proxy

Watch how to use the Dependency Proxy to help avoid Docker Hub rate limits.

In November 2020, Docker introduced rate limits on pull requests from Docker Hub. If your GitLab CI/CD configuration uses an image from Docker Hub, each time a job runs, it may count as a pull request. To help get around this limit, you can pull your image from the Dependency Proxy cache instead.

When you pull an image (by using a command like docker pull or, in a .gitlab-ci.yml file, image: foo:latest), the Docker client makes a collection of requests:

1. The image manifest is requested. The manifest contains information about how to build the image.
2. Using the manifest, the Docker client requests a collection of layers, also known as blobs, one at a time.

The Docker Hub rate limit is based on the number of GET requests for the manifest. The Dependency Proxy caches both the manifest and blobs for a given image, so when you request it again, Docker Hub does not have to be contacted.

How does GitLab know if a cached tagged image is stale?

If you are using an image tag like alpine:latest, the image changes over time. Each time it changes, the manifest contains different information about which blobs to request. The Dependency Proxy does not pull a new image each time the manifest changes; it checks only when the manifest becomes stale.

Docker does not count HEAD requests for the image manifest towards the rate limit. You can make a HEAD request for alpine:latest, view the digest (checksum) value returned in the header, and determine if a manifest has changed.

The Dependency Proxy starts all requests with a HEAD request. If the manifest has become stale, only then is a new image pulled.

For example, if your pipeline pulls node:latest every five minutes, the Dependency Proxy caches the entire image and only updates it if node:latest changes. So instead of having 360 requests for the image in six hours (which exceeds the Docker Hub rate limit), you only have one pull request, unless the manifest changed during that time.

Check your Docker Hub rate limit

If you are curious about how many requests to Docker Hub you have made and how many remain, you can run these commands from your runner, or even in a CI/CD script:

# Note, you must have jq installed to run this command
TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 | grep RateLimit
...

The output is something like:

RateLimit-Limit: 100;w=21600
RateLimit-Remaining: 98;w=21600

This example shows the total limit of 100 pulls in six hours, with 98 pulls remaining.

Check the rate limit in a CI/CD job

This example shows a GitLab CI/CD job that uses an image with jq and curl installed:

hub_docker_quota_check:
    stage: build
    image: alpine:latest
    tags:
        - <optional_runner_tag>
    before_script: apk add curl jq
    script:
      - |
        TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1