• What is end-to-end testing?
• How do we test GitLab?
  • Testing nightly builds
  • Testing staging
  • Testing code in merge requests
    • Using the package-and-qa job
    • How does it work?
    • With Pipeline for Merged Results
      • Running custom tests
    • Using the review-qa-all jobs
• How do I run the tests?
  • Running tests that require special setup
• How do I write tests?
  • Consider not writing an end-to-end test
• Where can I ask for help?

End-to-end Testing

What is end-to-end testing?

End-to-end testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.

How do we test GitLab?

We use Omnibus GitLab to build GitLab packages and then we test these packages using the GitLab QA orchestrator tool, which is a black-box testing framework for the API and the UI.

Testing nightly builds

We run scheduled pipelines each night to test nightly builds created by Omnibus. You can find these pipelines at https://gitlab.com/gitlab-org/quality/nightly/pipelines (need Developer access permissions). Results are reported in the #qa-nightly Slack channel.

Testing staging

We run scheduled pipelines each night to test staging. You can find these pipelines at https://gitlab.com/gitlab-org/quality/staging/pipelines (need Developer access permissions). Results are reported in the #qa-staging Slack channel.

Testing code in merge requests

Using the package-and-qa job

It is possible to run end-to-end tests for a merge request, eventually being run in a pipeline in the gitlab-org/gitlab-qa-mirror project, by triggering the package-and-qa manual action in the qa stage (not available for forks).

This runs end-to-end tests against a custom EE (with an Ultimate license) Docker image built from your merge request’s changes.

Manual action that starts end-to-end tests is also available in gitlab-org/omnibus-gitlab merge requests.

How does it work?

Currently, we are using multi-project pipeline-like approach to run end-to-end pipelines.

1. In the gitlab-org/gitlab pipeline:
   1. Developer triggers the package-and-qa manual action (available once the build-qa-image and build-assets-image jobs are done), that can be found in GitLab merge requests. This starts a chain of pipelines in multiple projects.
   2. The script being executed triggers a pipeline in gitlab-org/build/omnibus-gitlab-mirror and polls for the resulting status. We call this a status attribution.
2. In the gitlab-org/build/omnibus-gitlab-mirror pipeline:
   1. Docker image is being built and pushed to its Container Registry.
   2. Finally, the Trigger:qa-test job triggers a new end-to-end pipeline in gitlab-org/gitlab-qa-mirror and polls for the resulting status.
3. In the gitlab-org/gitlab-qa-mirror pipeline:
   1. Container for the Docker image stored in the gitlab-org/build/omnibus-gitlab-mirror registry is spun-up.
   2. End-to-end tests are run with the gitlab-qa executable, which spin up a container for the end-to-end image from the gitlab-org/gitlab registry.
4. The result of the gitlab-org/gitlab-qa-mirror pipeline is being propagated upstream (through polling from upstream pipelines), through gitlab-org/build/omnibus-gitlab-mirror, back to the gitlab-org/gitlab merge request.

Please note, we plan to add more specific information about the tests included in each job/scenario that runs in gitlab-org/gitlab-qa-mirror.

With Pipeline for Merged Results

In a Pipeline for Merged Results, the pipeline runs on a new ref that contains the merge result of the source and target branch. However, this ref is not available to the gitlab-qa-mirror pipeline.

For this reason, the end-to-end tests on a Pipeline for Merged Results would use the head of the merge request source branch.

Running custom tests

The existing scenarios that run in the downstream gitlab-qa-mirror pipeline include many tests, but there are times when you might want to run a test or a group of tests that are different than the groups in any of the existing scenarios.

For example, when we dequarantine a flaky test we first want to make sure that it’s no longer flaky. We can do that using the ce:custom-parallel and ee:custom-parallel jobs. Both are manual jobs that you can configure using custom variables. When clicking the name (not the play icon) of one of the parallel jobs, you are prompted to enter variables. You can use any of the variables that can be used with gitlab-qa as well as these:

For now manual jobs with custom variables don’t use the same variable when retried, so if you want to run the same test(s) multiple times, specify the same variables in each custom-parallel job (up to as many of the 10 available jobs that you want to run).

Using the review-qa-all jobs

On every pipeline during the test stage, the review-qa-smoke job is automatically started: it runs the QA smoke suite against the Review App.

You can also manually start the review-qa-all: it runs the full QA suite against the Review App.

This runs end-to-end tests against a Review App based on the official GitLab Helm chart, itself deployed with custom Cloud Native components built from your merge request’s changes.

See Review Apps for more details about Review Apps.

How do I run the tests?

If you are not testing code in a merge request, there are two main options for running the tests. If you want to run the existing tests against a live GitLab instance or against a pre-built Docker image, use the GitLab QA orchestrator. See also examples of the test scenarios you can run via the orchestrator.

On the other hand, if you would like to run against a local development GitLab environment, you can use the GitLab Development Kit (GDK). Please refer to the instructions in the QA README and the section below.

Running tests that require special setup

Learn how to perform tests that require special setup or consideration to run on your local environment.

How do I write tests?

In order to write new tests, you first need to learn more about GitLab QA architecture. See the documentation about it.

Once you decided where to put test environment orchestration scenarios and instance-level scenarios, take a look at the GitLab QA README, the GitLab QA orchestrator README, and the already existing instance-level scenarios.

Consider not writing an end-to-end test

We should follow these best practices for end-to-end tests:

• Do not write an end-to-end test if a lower-level feature test exists. End-to-end tests require more work and resources.
• Troubleshooting for end-to-end tests can be more complex as connections to the application under test are not known.

Continued reading:

• Beginner’s Guide
• Style Guide
• Best Practices
• Testing with feature flags
• Flows
• RSpec metadata/tags

Where can I ask for help?

You can ask question in the #quality channel on Slack (GitLab internal) or you can find an issue you would like to work on in the gitlab issue tracker, or the gitlab-qa issue tracker.