• Remote read/write
• Rules files
• External labels
• node_exporter
• Grafana dashboards

Prometheus Settings

Remote read/write

Prometheus supports reading and writing to remote services.

To configure a remote read or write service, you can include the following in gitlab.rb.

prometheus['remote_write'] = [
  {
    url: 'https://some-remote-write-service.example.com',
    basic_auth: {
      password: 'remote write secret password'
    }
  }
]
prometheus['remote_read'] = [
  {
    url: 'https://some-remote-write-service.example.com'
  }
]

For more documentation on the options available, see the remote write and remote read sections of the official documentation.

Rules files

Prometheus allows for recording and alerting rules.

Omnibus includes some default rules files that are stored in /var/opt/gitlab/prometheus/rules/.

To override the default rules, you can change the default list in gitlab.rb..

No rules:

prometheus['rules_files'] = []

Custom list:

prometheus['rules_files'] = ['/path/to/rules/*.rules', '/path/to/single/file.rules']

External labels

To set external labels:

prometheus['external_labels'] = {
    'region' => 'us-west-2',
    'source' => 'omnibus',
}

No external labels are set by default.

node_exporter

The node_exporter provides system level metrics.

Additional metrics collectors are enabled by default. For example, mountstats is used to collect metrics about NFS mounts.

To disable the mountstats collector, adjust gitlab.rb with the following setting and run gitlab-ctl reconfigure:

node_exporter['flags'] = {
  'collector.mountstats' => false,
}

For more information on available collectors, see the upstream documentation.

Grafana dashboards

Grafana is a powerful dashboard software for presenting Prometheus metrics data. Omnibus GitLab >= 11.9 includes an embedded copy.

See the embedded Grafana documentation for more information.