• Preparation
  • Generating certificates for internal use
    • Required certificate CN and SANs
• Configuration
• Result
• Troubleshooting

Use TLS between components of the GitLab chart

The GitLab charts can use transport-layer security (TLS) between the various components. This requires you to provide certificates for the services you want to enable, and configure those services to make use of those certificates and the certificate authority (CA) that signed them.

Preparation

Each chart has documentation regarding enabling TLS for that service, and the various settings required to ensure that appropriate configuration.

Generating certificates for internal use

For the purposes of this documentation, we provide a Proof of Concept script below, which makes use of Cloudflare’s CFSSL to produce a self-signed Certificate Authority, and a wildcard certificate that can be used for all services.

This script will:

• Generate a CA key pair.
• Sign a certificate meant to service all GitLab component service endpoints.
• Create two Kubernetes Secret objects:
  • A secret of type kuberetes.io/tls which has the server certificate and key pair.
  • A secret of type Opaque which only contains the public certificate of the CA as ca.crt as need by NGINX Ingress.

Prerequisites:

• Bash, or compatible shell.
• cfssl is available to your shell, and within PATH.
• kubectl is available, and configured to point to your Kubernetes cluster where GitLab will later be installed.
  • Be sure to have created the namespace you wish to have these certificates installed into before operating the script.

You may copy the content of this script to your computer, and make the resulting file executable. We suggest poc-gitlab-internal-tls.sh.

#!/bin/bash
set -e
#############
## make and change into a working directory
pushd $(mktemp -d)

#############
## setup environment
NAMESPACE=${NAMESPACE:-default}
RELEASE=${RELEASE:-gitlab}
## stop if variable is unset beyond this point
set -u
## known expected patterns for SAN
CERT_SANS="*.${NAMESPACE}.svc,${RELEASE}-metrics.${NAMESPACE}.svc,*.${RELEASE}-gitaly.${NAMESPACE}.svc"

#############
## generate default CA config
cfssl print-defaults config > ca-config.json
## generate a CA
echo '{"CN":"'${RELEASE}.${NAMESPACE}.internal.ca'","key":{"algo":"ecdsa","size":256}}' | \
  cfssl gencert -initca - | \
  cfssljson -bare ca -
## generate certificate
echo '{"CN":"'${RELEASE}.${NAMESPACE}.internal'","key":{"algo":"ecdsa","size":256}}' | \
  cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -profile www -hostname="${CERT_SANS}" - |\
  cfssljson -bare ${RELEASE}-services

#############
## load certificates into K8s
kubectl -n ${NAMESPACE} create secret tls ${RELEASE}-internal-tls \
  --cert=${RELEASE}-services.pem \
  --key=${RELEASE}-services-key.pem
kubectl -n ${NAMESPACE} create secret generic ${RELEASE}-internal-tls-ca \
  --from-file=ca.crt=ca.pem

The script expects two environment variables to be set:

1. NAMESPACE: The Kubernetes Namespace you will later install GitLab to. This defaults to default, as with kubectl.
2. RELEASE: The Helm Release name you will later use to install GitLab. This defaults to gitlab.

To operate this script, you may export the two variables, or prepend the script name with their values.

export NAMESPACE=testing
export RELEASE=gitlab

./poc-gitlab-internal-tls.sh

After the script has run, you will find the two secrets created, and the temporary working directory contains all certificates and their keys.

$ pwd
/tmp/tmp.swyMgf9mDs
$ kubectl -n ${NAMESPACE} get secret | grep internal-tls
testing-internal-tls      kubernetes.io/tls                     2      11s
testing-internal-tls-ca   Opaque                                1      10s
$ ls -1
ca-config.json
ca.csr
ca-key.pem
ca.pem
testing-services.csr
testing-services-key.pem
testing-services.pem

Required certificate CN and SANs

The various GitLab components speak to each other over their Service’s DNS names. The Ingress objects generated by the GitLab chart must provide NGINX the name to verify, when tls.verify: true (which is the default). As a result of this, each GitLab component should receive a certificate with a SAN including either their Service’s name, or a wildcard acceptable to the Kubernetes Service DNS entry.

• service-name.namespace.svc
• *.namespace.svc

Failure to ensure these SANs within certificates will result in a non-functional instance, and logs that can be quite cryptic, refering to “connection failure” or “SSL verification failed”.

You can make use of helm template to retrieve a full list of all Service object names, if needed. If your GitLab has been deployed without TLS, you can query Kubernetes for those names:

kubectl -n ${NAMESPACE} get service -lrelease=${RELEASE}

Configuration

Example configurations can be found in examples/internal-tls.

For the purposes of this documentation, we have provided shared-cert-values.yaml which configures the GitLab components to consume the certificates generated with the script above, in generating certificates for internal use.

Key items to configure:

1. Global Custom Certificate Authorities.
2. Per-component TLS for the service listeners. (See each chart’s documentation, under charts/)

This process is greatly simplified by making use of YAML’s native anchor functionality. A truncated snippet of shared-cert-values.yaml shows this:

.internal-ca: &internal-ca gitlab-internal-tls-ca
.internal-tls: &internal-tls gitlab-internal-tls

global:
  certificates:
    customCAs:
    - secret: *internal-ca
  workhorse:
    tls:
      enabled: true
gitlab:
  webservice:
    tls:
      secretName: *internal-tls
    workhorse:
       tls:
          verify: true # default
          secretName: *internal-tls
          caSecretName: *internal-ca

Result

When all components have been configured to provide TLS on their service listeners, all communication between GitLab components will traverse the network with TLS security, including connections from NGINX Ingress to each GitLab component.

NGINX Ingress will terminate any inbound TLS, determine the appropriate services to pass the traffic to, and then form a new TLS connection to the GitLab component. When configured as shown here, it will also verify the certificates served by the GitLab components against the CA.

This can be verified by connecting to the Toolbox pod, and querying the various compontent Services. One such example, connecting to the Webservice Pod’s primary service port that NGINX Ingress uses:

$ kubectl -n ${NAMESPACE} get pod -lapp=toolbox,release=${RELEASE}
NAME                              READY   STATUS    RESTARTS   AGE
gitlab-toolbox-5c447bfdb4-pfmpc   1/1     Running   0          65m
$ kubectl exec -ti gitlab-toolbox-5c447bfdb4-pfmpc -c toolbox -- \
    curl -Iv "https://gitlab-webservice-default.testing.svc:8181"

The output should be similar to following example:

*   Trying 10.60.0.237:8181...
* Connected to gitlab-webservice-default.testing.svc (10.60.0.237) port 8181 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=gitlab.testing.internal
*  start date: Jul 18 19:15:00 2022 GMT
*  expire date: Jul 18 19:15:00 2023 GMT
*  subjectAltName: host "gitlab-webservice-default.testing.svc" matched cert's "*.testing.svc"
*  issuer: CN=gitlab.testing.internal.ca
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: gitlab-webservice-default.testing.svc:8181

Troubleshooting

If your GitLab instance appears unreachable from the browser, by rendering an HTTP 503 error, NGINX Ingress is likely having a problem verifying the certificates of the GitLab components.

You may work around this by temporarily setting gitlab.webservice.workhorse.tls.verify to false.

The NGINX Ingress controller can be connected to, and will evidence a message in nginx.conf, regarding problems verifying the certificate(s).

Example content, where the Secret is not reachable:

# Location denied. Reason: "error obtaining certificate: local SSL certificate
  testing/gitlab-internal-tls-ca was not found"
return 503;

Common problems that cause this:

• CA certificate is not in a key named ca.crt within the Secret.
• The Secret was not properly supplied, or may not exist within the Namespace.