极狐 GitLab Logo

极狐GitLab 架构概览

软件交付#

极狐GitLab 有一个发行版-JH（GitLab-JH），包含基础版、专业版、旗舰版。极狐GitLab 在不同的订阅下可用。

极狐GitLab 的新版本从稳定分支发布，main 分支用于前沿开发。

发行版需要额外的组件。这些组件在组件详情部分中描述，并且都有自己的仓库。每个依赖组件的新版本通常是标记的，但保持在极狐GitLab 代码库的 main 分支上可以让你获得这些组件的最新稳定版本。新版本通常与极狐GitLab 的发布同步，除非被认为是关键的非正式安全更新。

组件#

极狐GitLab 的典型安装在 GNU/Linux 上，但越来越多的部署也使用 Kubernetes 平台。已知最大的极狐GitLab 实例在 JihuLab.com 上，使用我们的官方极狐GitLab Helm chart和官方 Linux 软件包进行部署。

典型的安装使用 NGINX 或 Apache 作为 Web 服务器，通过 极狐GitLab Workhorse 代理到 Puma 应用服务器。极狐GitLab 使用 Puma 应用服务器提供网页和 极狐GitLab API。它使用 Sidekiq 作为作业队列，后者使用 Redis 作为作业信息、元数据和传入作业的非持久性数据库后端。

默认情况下，Puma 和 Workhorse 之间的通信是通过 Unix 域套接字进行的，但也支持通过 TCP 转发请求。Workhorse 访问 gitlab/public 目录，绕过 Puma 应用服务器来提供静态页面、上传文件（例如头像或附件）和预编译资源。

极狐GitLab 应用使用 PostgreSQL 存储持久性数据库信息（例如用户、权限、议题或其他元数据）。极狐GitLab 将裸 Git 仓库存储在配置文件的 repositories: 部分中定义的位置。它还与裸仓库一起保存默认分支和挂钩信息。

在通过 HTTP/HTTPS 提供仓库时，极狐GitLab 使用极狐GitLab API 解决授权和访问问题并提供 Git 对象。

附加组件极狐GitLab Shell 通过 SSH 提供仓库。它在配置文件的 GitLab Shell 部分中定义的位置管理 SSH 密钥。该位置的文件不应被手动编辑。极狐GitLab Shell 通过 Gitaly 访问裸仓库以提供 Git 对象，并通过 Redis 提交作业给 Sidekiq 供极狐GitLab 处理。极狐GitLab Shell 查询极狐GitLab API 以确定授权和访问。

Gitaly 从极狐GitLab Shell 和极狐GitLab Web 应用执行 Git 操作，并为极狐GitLab Web 应用提供一个 API 以从 Git 获取属性（例如标题、分支、标签或其他元数据），以及获取 blobs（例如 diffs、提交或文件）。

适应现有组件和引入新组件#

在传统的 Linux 机器上安装应用程序与在容器化平台（如 Kubernetes）上安装应用程序时，应用程序的行为有根本的不同。

与我们的官方安装方法相比，一些显著的差异是：

  • 官方 Linux 软件包可以访问同一文件系统上不同服务的文件。共享文件对于在 Kubernetes 平台上运行的应用程序不是一个选项。
  • 官方 Linux 软件包默认具有可以访问共享配置和网络的服务。这对于在 Kubernetes 中运行的服务来说并不是这样，服务可能在完全隔离中运行，或者只能通过特定端口访问。

换句话说，在设计新功能和添加新组件时，需要仔细考虑服务之间的共享状态。需要访问相同文件的服务需要能够通过适当的 API 交换信息。只要可能，就不应该通过文件来完成。

由于采用 API 优先理念编写的组件与这两种方法兼容，因此所有新功能和服务必须首先考虑 Kubernetes 兼容性。

有关更多详细信息，请参阅添加新服务组件的流程。

简化的组件概述#

这是一个简化的架构图，可以用于理解极狐GitLab 的架构。

完整的架构图可以在我们的组件图中查看。

Rendering chart...

除非另有说明，否则所有连接都使用 Unix 套接字。

组件图#

Rendering chart...

组件图例#

  • ✅ - 默认安装
  • ⚙ - 需要额外配置
  • ⤓ - 需要手动安装
  • ❌ - 不支持或没有说明书
  • N/A - 不适用

组件状态链接到每个组件的配置文档。

组件列表#

组件描述Omnibus GitLab极狐GitLab 环境工具包 (GET)极狐GitLab chartminikube MinimalJihuLab.comSourceGDKJH
Certificate ManagementTLS 设置，Let's Encrypt✅✅✅⚙✅⚙⚙CE & EE
Consul数据库节点发现，故障转移⚙✅❌❌✅❌❌EE Only
Database Migrations数据库迁移✅✅✅✅✅⚙✅CE & EE
Elasticsearch改善极狐GitLab 内的搜索⤓⚙⤓⤓✅⤓⚙EE Only
Gitaly处理极狐GitLab 所有 Git 调用的 Git RPC 服务✅✅✅✅✅⚙✅CE & EE
GitLab Exporter生成各种极狐GitLab 指标✅✅✅✅✅❌❌CE & EE
GitLab Geo地理分布式极狐GitLab 站点⚙⚙❌❌✅❌⚙EE Only
GitLab Pages托管静态网站⚙⚙⚙❌✅⚙⚙CE & EE
GitLab agent以云原生方式集成 Kubernetes 集群⚙⚙⚙❌❌⤓⚙EE Only
GitLab self-monitoring: Alertmanager去重、分组和路由来自 Prometheus 的警报⚙⚙✅⚙✅❌❌CE & EE
GitLab self-monitoring: Grafana指标仪表板✅✅⚙⤓✅❌⚙CE & EE
GitLab self-monitoring: Jaeger查看由极狐GitLab 实例生成的跟踪❌⚙⚙❌❌⤓⚙CE & EE
GitLab self-monitoring: Prometheus时间序列数据库，指标收集和查询服务✅✅✅⚙✅❌⚙CE & EE
GitLab self-monitoring: Sentry跟踪极狐GitLab 实例生成的错误⤓⤓⤓❌✅⤓⤓CE & EE
GitLab Shell处理通过 SSH 会话进行的 git✅✅✅✅✅⚙✅CE & EE
GitLab Workhorse智能反向代理，处理大型 HTTP 请求✅✅✅✅✅⚙✅CE & EE
Inbound email (SMTP)接收消息以更新议题⤓⤓⚙⤓✅⤓⤓CE & EE
Jaeger integration已部署应用的分布式跟踪⤓⤓⤓⤓⤓⤓⚙EE Only
LDAP Authentication针对集中式 LDAP 目录对用户进行身份验证⤓⤓⤓⤓❌⤓⚙CE & EE
Mattermost开源的 Slack 替代方案⚙⚙⤓⤓⤓❌⚙CE & EE
MinIO对象存储服务⤓⤓✅✅✅❌⚙CE & EE
NGINX将请求路由到适当的组件，终止 SSL✅✅✅⚙✅⤓⚙CE & EE
Node Exporter具有系统指标的 Prometheus 端点✅✅N/AN/A✅❌❌CE & EE
Outbound email (SMTP)向用户发送电子邮件消息⤓⤓⚙⤓✅⤓⤓CE & EE
Patroni管理 PostgreSQL 高可用集群的领导者选择和复制⚙✅❌❌✅❌❌EE Only
PgBouncer Exporter具有 PgBouncer 指标的 Prometheus 端点⚙✅❌❌✅❌❌CE & EE
PgBouncer数据库连接池，故障转移⚙✅❌❌✅❌❌EE Only
PostgreSQL Exporter具有 PostgreSQL 指标的 Prometheus 端点✅✅✅✅✅❌❌CE & EE
PostgreSQL数据库✅✅✅✅✅⤓✅CE & EE
PraefectGit 客户端和 Gitaly 存储节点之间的透明代理。✅✅⚙❌❌⚙✅CE & EE
Puma (GitLab Rails)处理 Web 界面和 API 的请求✅✅✅✅✅⚙✅CE & EE
Redis Exporter具有 Redis 指标的 Prometheus 端点✅✅✅✅✅❌❌CE & EE
Redis缓存服务✅✅✅✅✅⤓✅CE & EE
Registry容器注册表，允许推送和拉取镜像⚙⚙✅✅✅⤓⚙CE & EE
Runner执行极狐GitLab CI/CD 作业⤓⤓✅⚙✅⚙⚙CE & EE
Sentry integration已部署应用的错误跟踪⤓⤓⤓⤓⤓⤓⤓CE & EE
Sidekiq后台作业处理器✅✅✅✅✅✅✅CE & EE
Token Revocation API接收和撤销泄露的密钥❌❌❌❌✅❌❌EE Only

组件详情#

本文档旨在供系统管理员和极狐GitLab 支持工程师使用，他们希望了解极狐GitLab 的内部工作原理及其如何协同工作。

在部署时，极狐GitLab 应被视为以下过程的集合。在故障排除或调试时，请尽可能具体地指明您所引用的组件。这应该能提高清晰度并减少混淆。

层

从过程的角度来看，极狐GitLab 可以被视为具有两个层：

  • 监控：来自此层的任何内容对于交付极狐GitLab 应用不是必需的，但允许管理员更深入了解其基础架构以及整个服务的运行情况。
  • 核心：任何对极狐GitLab 作为平台的交付至关重要的过程。如果这些过程中的任何一个停止，将导致极狐GitLab 停机。对于核心层，您可以进一步划分为：
    • 处理器：这些过程负责实际执行操作并提供服务。
    • 数据：这些服务存储/暴露极狐GitLab 服务的结构化数据。

Alertmanager#

  • 项目页面
  • 配置：
  • 层：监控
  • 过程：alertmanager
  • JihuLab.com

Alert manager 是 Prometheus 提供的一个工具，它_"处理由客户端应用程序（如 Prometheus 服务器）发送的警报。它负责去重、分组，并将它们路由到正确的接收方集成（如电子邮件、PagerDuty 或 Opsgenie）。它还负责消音和抑制警报。

Certificate management#

  • 项目页面：
    • Omnibus
    • Charts
  • 配置：
  • 层：核心服务（处理器）
  • JihuLab.com：密钥管理

Consul#

Consul 是一个用于服务发现和配置的工具。Consul 是分布式的，高度可用，并且极具可扩展性。

Database migrations#

Elasticsearch#

Elasticsearch 是一个为云构建的分布式 RESTful 搜索引擎。

Gitaly#

Gitaly 是极狐GitLab 设计的一个服务，用于消除我们在极狐GitLab 的分布式部署中对 Git 存储的 NFS 需求（想想极狐GitLab.com 或高可用性部署）。从 11.3.0 版本开始，此服务处理极狐GitLab 中的所有 Git 级别访问。你可以在项目的 README中阅读更多关于该项目的信息。

Praefect#

Praefect 是每个 Git 客户端和 Gitaly 之间的透明代理，协调将仓库更新复制到次节点。

GitLab Geo#

Geo 是一个专业版功能，旨在通过提供一个或多个主极狐GitLab 实例的只读镜像来加速分布式团队的开发。这个镜像（一个 Geo 次级站点）减少了克隆或获取大型仓库和项目的时间，或者可以成为灾难恢复解决方案的一部分。

GitLab Exporter#

GitLab Exporter 是一个在内部设计的过程，允许我们将极狐GitLab 应用程序内部的指标导出到 Prometheus。你可以在项目的 README中阅读更多内容。

GitLab agent#

极狐GitLab agent 是一个活跃的集群内组件，用于以安全和云原生的方式解决极狐GitLab 和 Kubernetes 集成任务。

你可以使用它来同步部署到你的 Kubernetes 集群。

GitLab Pages#

极狐GitLab Pages 是一个功能，允许你直接从极狐GitLab 中的仓库发布静态网站。

你可以将它用于个人或商业网站，如作品集、文档、宣言和商业演示。你还可以为你的内容指定任何许可证。

GitLab Runner#

极狐GitLab Runner 运行作业并将结果发送给极狐GitLab。

极狐GitLab CI/CD 是极狐GitLab 附带的开源持续集成服务，负责协调测试。该项目的旧名称是 极狐GitLab CI Multi Runner，但请从现在起使用 极狐GitLab Runner（不带 CI）。

GitLab Shell#

极狐GitLab Shell 是极狐GitLab 设计的一个程序，处理基于 SSH 的 git 会话，并修改授权密钥列表。极狐GitLab Shell 不是 Unix shell 也不是 Bash 或 Zsh 的替代品。

GitLab Workhorse#

极狐GitLab Workhorse 是极狐GitLab 设计的一个程序，用于帮助减轻 Puma 的压力。它被设计为一个智能反向代理，以帮助加速极狐GitLab 的整体性能。

Grafana#

Grafana 是一个开源的功能丰富的指标仪表板和图形编辑器，适用于 Graphite、Elasticsearch、OpenTSDB、Prometheus 和 InfluxDB。

Jaeger#

Jaeger 受 Dapper 和 OpenZipkin 启发，是一个分布式跟踪系统。它可用于监控基于微服务的分布式系统。

Logrotate#

  • 项目页面
  • 配置：
  • 层：核心服务
  • 过程：logrotate

极狐GitLab 由大量服务组成，这些服务都会记录日志。我们捆绑了我们自己的 Logrotate 以确保我们能够负责任地记录日志。这只是一个常见的开源产品的打包版本。

Mattermost#

  • 项目页面
  • 配置：
  • 层：核心服务（处理器）
  • JihuLab.com：Mattermost

Mattermost 是来自 https://mattermost.com 的开源私有云 Slack 替代品。

MinIO#

MinIO 是一个在 GNU AGPL v3.0 下发布的对象存储服务器。它兼容 Amazon S3 云存储服务。它最适合存储非结构化数据，如照片、视频、日志文件、备份和容器/虚拟机镜像。对象的大小可以从几 KB 到最大 5 TB 不等。

NGINX#

  • 项目页面：
    • Omnibus
    • Charts
  • 配置：
  • 层：核心服务（处理器）
  • 过程：nginx

NGINX 拥有一个入口端口，用于所有 HTTP 请求，并将它们路由到极狐GitLab 内的适当子系统。我们捆绑了一个未经修改的流行开源 Web 服务器版本。

Node Exporter#

  • 项目页面
  • 配置：
  • 层：监控
  • 过程：node-exporter

Node Exporter 是一个 Prometheus 工具，提供有关底层机器的指标（例如 CPU/磁盘/负载）。它只是 Prometheus 项目中常见开源产品的打包版本。

Patroni#

  • 项目页面
  • 配置：
  • 层：核心服务（数据）
  • 过程：patroni

PgBouncer#

Lightweight connection pooler for PostgreSQL.

PgBouncer Exporter#

Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics.

PostgreSQL#

GitLab packages the popular Database to provide storage for Application meta data and user information.

PostgreSQL Exporter#

postgres_exporter is the community provided Prometheus exporter that delivers data about PostgreSQL to Prometheus for use in Grafana Dashboards.

Prometheus#

Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service.

Redis#

Redis is packaged to provide a place to store:

  • session data
  • temporary cache information
  • background job queues

See our Redis guidelines for more information about how GitLab uses Redis.

Redis Exporter#

Redis Exporter is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana.

Registry#

The registry is what users use to store their own Docker images. The bundled registry uses NGINX as a load balancer and GitLab as an authentication manager. Whenever a client requests to pull or push an image from the registry, it returns a 401 response along with a header detailing where to get an authentication token, in this case the GitLab instance. The client then requests a pull or push auth token from GitLab and retries the original request to the registry. For more information, see token authentication.

An external registry can also be configured to use GitLab as an auth endpoint.

Sentry#

Sentry fundamentally is a service that helps you monitor and fix crashes in real time. The server is in Python, but it contains a full API for sending events from any language, in any application.

For monitoring deployed apps, see the Sentry integration docs

Sidekiq#

Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background.

Puma#

Starting with GitLab 13.0, Puma is the default web server.

Puma is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often this displays in process output as bundle or config.ru depending on the GitLab version.

LDAP Authentication#

Outbound Email#

Inbound Email#

GitLab by request type#

GitLab provides two "interfaces" for end users to access the service:

  • Web HTTP Requests (Viewing the UI/API)
  • Git HTTP/SSH Requests (Pushing/Pulling Git Data)

It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type.

GitLab Web HTTP request cycle#

When making a request to an HTTP Endpoint (think /users/sign_in) the request takes the following path through the GitLab Service:

  • NGINX - Acts as our first line reverse proxy.
  • GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Puma.
  • Puma - Since this is a web request, and it needs to access the application, it routes to Puma.
  • PostgreSQL/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data.

GitLab Git request cycle#

Below we describe the different paths that HTTP vs. SSH Git requests take. There is some overlap with the Web Request Cycle but also some differences.

Web request (80/443)#

Git operations over HTTP use the stateless "smart" protocol described in the Git documentation, but responsibility for handling these operations is split across several GitLab components.

Here is a sequence diagram for git fetch. All requests pass through NGINX and any other HTTP load balancers, but are not transformed in any way by them. All paths are presented relative to a /namespace/project.git URL.

Rendering chart...

The sequence is similar for git push, except git-receive-pack is used instead of git-upload-pack.

SSH request (22)#

Git operations over SSH can use the stateful protocol described in the Git documentation, but responsibility for handling them is split across several GitLab components.

No GitLab components speak SSH directly - all SSH connections are made between Git on the client machine and the SSH server, which terminates the connection. To the SSH server, all connections are authenticated as the git user; GitLab users are differentiated by the SSH key presented by the client.

Here is a sequence diagram for git fetch, assuming Fast SSH key lookup is enabled. AuthorizedKeysCommand is an executable provided by GitLab Shell:

Rendering chart...

The git push operation is very similar, except git receive-pack is used instead of git upload-pack.

If fast SSH key lookups are not enabled, the SSH server reads from the ~git/.ssh/authorized_keys file to determine what command to run for a given SSH session. This is kept up to date by an AuthorizedKeysWorker in Rails, scheduled to run whenever an SSH key is modified by a user.

SSH certificates may be used instead of keys. In this case, AuthorizedKeysCommand is replaced with an AuthorizedPrincipalsCommand. This extracts a username from the certificate without using the Rails internal API, which is used instead of key_id in the /api/internal/allowed call later.

GitLab Shell also has a few operations that do not involve Gitaly, such as resetting two-factor authentication codes. These are handled in the same way, except there is no round-trip into Gitaly - Rails performs the action as part of the internal API call, and GitLab Shell streams the response back to the user directly.

System layout#

When referring to ~git in the pictures it means the home directory of the Git user which is typically /home/git.

GitLab is primarily installed within the /home/git user home directory as git user. Within the home directory is where the GitLab server software resides as well as the repositories (though the repository location is configurable).

The bare repositories are located in /home/git/repositories. GitLab is a Ruby on rails application so the particulars of the inner workings can be learned by studying how a Ruby on rails application works.

To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in /home/git/gitlab-shell.

Installation folder summary#

To summarize here's the directory structure of the git user home directory.

Processes#

shell
ps aux | grep '^git'

GitLab has several components to operate. It requires a persistent database (PostgreSQL) and Redis database, and uses Apache httpd or NGINX to proxypass Puma. All these components should run as different system users to GitLab (for example, postgres, redis, and www-data, instead of git).

As the git user it starts Sidekiq and Puma (a simple Ruby HTTP server running on port 8080 by default). Under the GitLab user there are usually 4 processes: puma master (1 process), puma cluster worker (2 processes), sidekiq (1 process).

Repository access#

Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull uses the GitLab API and SSH cloning is handled by GitLab Shell (previously explained).

Troubleshooting#

See the README for more information.

Init scripts of the services#

The GitLab init script starts and stops Puma and Sidekiq:

plaintext
/etc/init.d/gitlab
Usage: service gitlab {start|stop|restart|reload|status}

Redis (key-value store/non-persistent database):

plaintext
/etc/init.d/redis
Usage: /etc/init.d/redis {start|stop|status|restart|condrestart|try-restart}

SSH daemon:

plaintext
/etc/init.d/sshd
Usage: /etc/init.d/sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status}

Web server (one of the following):

plaintext
/etc/init.d/httpd
Usage: httpd {start|stop|restart|condrestart|try-restart|force-reload|reload|status|fullstatus|graceful|help|configtest}

$ /etc/init.d/nginx
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}

Persistent database:

plaintext
$ /etc/init.d/postgresql
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]

Log locations of the services#

GitLab (includes Puma and Sidekiq logs):

  • /home/git/gitlab/log/ usually contains application.log, production.log, sidekiq.log, puma.stdout.log, git_json.log and puma.stderr.log.

GitLab Shell:

  • /home/git/gitlab-shell/gitlab-shell.log

SSH:

  • /var/log/auth.log auth log (on Ubuntu).
  • /var/log/secure auth log (on RHEL).

NGINX:

  • /var/log/nginx/ contains error and access logs.

Apache httpd:

  • Explanation of Apache logs.
  • /var/log/apache2/ contains error and output logs (on Ubuntu).
  • /var/log/httpd/ contains error and output logs (on RHEL).

Redis:

  • /var/log/redis/redis.log there are also log-rotated logs there.

PostgreSQL:

  • /var/log/postgresql/*

GitLab specific configuration files#

GitLab has configuration files located in /home/git/gitlab/config/*. Commonly referenced configuration files include:

  • gitlab.yml: GitLab Rails configuration
  • puma.rb: Puma web server settings
  • database.yml: Database connection settings

GitLab Shell has a configuration file at /home/git/gitlab-shell/config.yml.

Adding a new setting in GitLab Rails#

Settings which belong in gitlab.yml include those related to:

  • How the application is wired together across multiple services. For example, Gitaly addresses, Redis addresses, Postgres addresses, and Consul addresses.
  • Distributed Tracing configuration, and some observability configurations. For example, histogram bucket boundaries.
  • Anything that needs to be configured during Rails initialization, possibly before the Postgres connection has been established.

Many other settings are better placed in the app itself, in ApplicationSetting. Managing settings in UI is usually a better user experience compared to managing configuration files. With respect to development cost, modifying gitlab.yml often seems like a faster iteration, but when you consider all the deployment methods below, it may be a poor tradeoff.

When adding a setting to gitlab.yml:

  1. Ensure that it is also added to Omnibus.
  2. Ensure that it is also added to Charts, if needed.
  3. Ensure that it is also added to GDK.

Maintenance tasks#

GitLab provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See maintenance Rake tasks. In a nutshell, do the following:

shell
sudo -i -u git
cd gitlab
bundle exec rake gitlab:env:info RAILS_ENV=production
bundle exec rake gitlab:check RAILS_ENV=production

It's recommended to sign in to the git user using either sudo -i -u git or sudo su - git. Although the sudo commands provided by GitLab work in Ubuntu, they don't always work in RHEL.

GitLab.com#

The GitLab.com architecture is detailed for your reference, but this architecture is only useful if you have millions of users.