Geo proxying for secondary sites

Tier: Premium, Ultimate Offering: Self-managed
History
The availability of this feature is controlled by a feature flag. For more information, see the history. The geo_secondary_proxy_separate_urls feature flag is planned to be deprecated and removed in a future release. Support for read-only Geo secondary sites is proposed in issue 366810.

Secondary sites behave as full read-write GitLab instances. They transparently proxy all operations to the primary site, with some notable exceptions.

This behavior enables use-cases including:

  • Putting all Geo sites behind a single URL, to deliver a consistent, seamless, and comprehensive experience whichever site the user lands on. Users don’t need to juggle multiple GitLab URLs.
  • Geographically load-balancing traffic without worrying about write access.

For an overview, see Geo proxying for secondary sites.

For known issues, see proxying-related items in the Geo documentation.

Set up a unified URL for Geo sites

Secondary sites can transparently serve read-write traffic. Therefore, you can use a single external URL so that requests can hit either the primary Geo site or any secondary Geo sites. This delivers a consistent, seamless, and comprehensive experience whichever site the user lands on. Users don’t need to juggle multiple URLs or even be aware of the idea of multiple sites.

You can route traffic to Geo sites with:

  • Geo-location aware DNS. To route traffic to the closest Geo site, whether primary or secondary. For an example, follow Configure location-aware DNS.
  • Round-robin DNS.
  • A load-balancer. It must use sticky sessions to avoid authentication failures and cross-site request errors. DNS routing is inherently sticky so it does not share this caveat.

Configure location-aware DNS

Follow this example to route traffic to the closest Geo site, whether primary or secondary.

Prerequisites

This example creates a gitlab.example.com subdomain that automatically directs requests:

  • From Europe to a secondary site.
  • From all other locations to the primary site.

For this example, you need:

AWS Route53

In this example, you use a Route53 Hosted Zone managing your domain for the Route53 setup.

In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. To create a traffic policy:

  1. Go to the Route53 dashboard and select Traffic policies.
  2. Select Create traffic policy.
  3. Fill in the Policy Name field with Single Git Host and select Next.
  4. Leave DNS type as A: IP Address in IPv4 format.
  5. Select Connect to, then select Geolocation rule.
  6. For the first Location:
    1. Leave it as Default.
    2. Select Connect to, then select New endpoint.
    3. Choose Type value and fill it in with <your **primary** IP address>.
  7. For the second Location:
    1. Choose Europe.
    2. Select Connect to, then select New endpoint.
    3. Choose Type value and fill it in with <your **secondary** IP address>.

    Add traffic policy endpoints

  8. Select Create traffic policy.

  9. Fill in Policy record DNS name with gitlab.

    Create policy records with traffic policy

  10. Select Create policy records.

You have successfully set up a single host, like gitlab.example.com, which distributes traffic to your Geo sites by geolocation.

GCP

In this example, you create a GCP Cloud DNS zone managing your domain.

When creating Geo-Based record sets, GCP applies a nearest match for the source region when the source of the traffic doesn’t match any policy items exactly. To create a Geo-Based record set:

  1. Select Network Services > Cloud DNS.
  2. Select the Zone configured for your domain.
  3. Select Add Record Set.
  4. Enter the DNS Name for your Location-aware public URL, for example, gitlab.example.com.
  5. Select the Routing Policy: Geo-Based.
  6. Select Add Managed RRData.
    1. Select Source Region: us-central1.
    2. Enter your <**primary** IP address>.
    3. Select Done.
  7. Select Add Managed RRData.
    1. Select Source Region: europe-west1.
    2. Enter your <**secondary** IP address>.
    3. Select Done.
  8. Select Create.

You have successfully set up a single host, like gitlab.example.com, which distributes traffic to your Geo sites using a location-aware URL.

Configure each site to use the same external URL

After you have set up routing from a single URL to all of your Geo sites, follow the following steps if your sites use different URLs:

  1. On each GitLab site, SSH into each node running Rails (Puma, Sidekiq, Log-Cursor) and set the external_url to that of the single URL: 

    sudo -e /etc/gitlab/gitlab.rb

  2. Reconfigure the updated nodes for the change to take effect: 

    sudo gitlab-ctl reconfigure

  3. To match the new external URL set on the secondary Geo sites, the primary database needs to reflect this change.

    In the Geo administration page of the primary site, edit each Geo secondary that is using the secondary proxying and set the URL field to the single URL. Make sure the primary site is also using this URL.

    To allow the sites to talk to each other, make sure the Internal URL field is unique for each site.

In Kubernetes, you can use the same domain under global.hosts.domain as for the primary site.

Set up a separate URL for a secondary Geo site

You can use different external URLs per site. You can use this to offer a specific site to a specific set of users. Alternatively, you can give users control over which site they use, though they must understand the implications of their choice.

note
GitLab does not support multiple external URLs, see issue 21319. An inherent problem is there are many cases where a site needs to produce an absolute URL outside of the context of an HTTP request, such as when sending emails that were not triggered by a request.

Configure a secondary Geo site to a different external URL than the primary site

If your secondary site uses the same external URL as the primary site:

  1. On the secondary site, SSH into each node running Rails (Puma, Sidekiq, Log-Cursor) and set the external_url to the desired URL for the secondary site: 

    sudo -e /etc/gitlab/gitlab.rb

  2. Reconfigure the updated nodes for the change to take effect: 

    sudo gitlab-ctl reconfigure

  3. To match the new external URL set on the secondary Geo site, the primary database needs to reflect this change.

    In the Geo administration page of the primary site, edit the target secondary site and set the URL field to the desired URL.

    To allow the sites to talk to each other, make sure the Internal URL field is unique for each site. If the desired URL is unique to this site, then you can clear the Internal URL field. On save, it defaults to the external URL.

Behavior of secondary sites when the primary Geo site is down

Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary site is inaccessible:

  • UI and API traffic return the same errors as the primary (or fail if the primary is not accessible at all), since they are proxied.
  • For repositories that are fully up-to-date on the specific secondary site being accessed, Git read operations still work as expected, including authentication through HTTP(s) or SSH. However, Git reads performed by GitLab Runners will fail.
  • Git operations for repositories that are not replicated to the secondary site return the same errors as the primary site, since they are proxied.
  • All Git write operations return the same errors as the primary site, since they are proxied.

Features accelerated by secondary Geo sites

Most HTTP traffic sent to a secondary Geo site is proxied to the primary Geo site. With this architecture, secondary Geo sites are able to support write requests, and avoid read-after-write problems. Certain read requests are handled locally by secondary sites for improved latency and bandwidth nearby.

The following table details the components tested through the Geo secondary site Workhorse proxy. It does not cover all data types.

In this context, accelerated reads refer to read requests served from the secondary site, provided that the data is up to date for the component on the secondary site. If the data on the secondary site is determined to be out of date, the request is forwarded to the primary site. Read requests for components not listed in the table below are always automatically forwarded to the primary site.

Feature / component Accelerated reads? Notes
Project, wiki, design repository (using the web UI) No  
Project, wiki repository (using Git) Yes Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don’t exist locally on the Geo secondary throw a “not found” error.
Project, Personal Snippet (using the web UI) No  
Project, Personal Snippet (using Git) Yes Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don’t exist locally on the Geo secondary throw a “not found” error.
Group wiki repository (using the web UI) No  
Group wiki repository (using Git) Yes Git reads are served from the local secondary while pushes get proxied to the primary. Selective sync or cases where repositories don’t exist locally on the Geo secondary throw a “not found” error.
User uploads No  
LFS objects (using the web UI) No  
LFS objects (using Git) Yes  
Pages No Pages can use the same URL (without access control), but must be configured separately and are not proxied.
Advanced search (using the web UI) No  
Container registry No The container registry is only recommended for Disaster Recovery scenarios. If the secondary site’s container registry is not up to date, the read request is served with old data as the request is not forwarded to the primary site. Accelerating the container registry is planned, please upvote or comment in the issue to indicate your interest or ask your GitLab representative to do so on your behalf.
Dependency Proxy No Read requests to a Geo secondary site’s Dependency Proxy are always proxied to the primary site.
All other data No Read requests for components not listed in this table are always automatically forwarded to the primary site.

To request acceleration of a feature, check if an issue already exists in epic 8239 and upvote or comment on it to indicate your interest or ask your GitLab representative to do so on your behalf. If an applicable issue doesn’t exist, open one and mention it in the epic.

Disable secondary site HTTP proxying

Secondary site HTTP proxying is enabled by default on a secondary site when it uses a unified URL, meaning, it is configured with the same external_url as the primary site. Disabling proxying in this case tends not to be helpful due to completely different behavior being served at the same URL, depending on routing.

HTTP proxying is enabled by default in GitLab 15.1 on a secondary site even without a unified URL. If proxying needs to be disabled on all secondary sites, it is easiest to disable the feature flag:

  1. SSH into a node which is running Puma or Sidekiq on your primary Geo site and run: 

    sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)"

  2. Restart Puma on all nodes which are running it on your secondary Geo site: 

    sudo gitlab-ctl restart puma

In Kubernetes, you can run the same command in the toolbox pod. Refer to the Kubernetes cheat sheet for details.

Disable secondary site HTTP proxying per site

If there are multiple secondary sites, you can disable HTTP proxying on each secondary site separately, by following these steps:

Linux package (Omnibus)

  1. SSH into each application node (serving user traffic directly) on your secondary Geo site and add the following environment variable: 

    sudo -e /etc/gitlab/gitlab.rb

    gitlab_workhorse['env'] = {
  "GEO_SECONDARY_PROXY" => "0"
}

  2. Reconfigure the updated nodes for the change to take effect: 

    sudo gitlab-ctl reconfigure
Helm chart (Kubernetes)

You can use --set gitlab.webservice.extraEnv.GEO_SECONDARY_PROXY="0", or specify the following in your values file: 

gitlab:
  webservice:
    extraEnv:
      GEO_SECONDARY_PROXY: "0"

Disable secondary site Git proxying

It is not possible to disable forwarding of:

  • Git push over SSH
  • Git pull over SSH when the Git repository is out-of-date on the secondary site
  • Git push over HTTP
  • Git pull over HTTP when the Git repository is out-of-date on the secondary site