What is end-to-end testing?
End-to-end testing is a strategy used to check whether your application works as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together.
How do we test GitLab?
Testing nightly builds
We run scheduled pipeline each night to test nightly builds created by Omnibus.
You can find these nightly pipelines at
(need Developer access permissions). Results are reported in the
#qa-nightly Slack channel.
We run scheduled pipeline each night to test staging.
You can find these nightly pipelines at
(need developer access permissions). Results are reported in the
#qa-staging Slack channel.
Testing code in merge requests
It is possible to run end-to-end tests for a merge request, eventually being run in
a pipeline in the
by triggering the
package-and-qa-manual manual action in the
test stage (not
available for forks).
This runs end-to-end tests against a custom Omnibus package built from your merge request's changes.
Manual action that starts end-to-end tests is also available in merge requests in Omnibus GitLab.
Below you can read more about how to use it and how does it work.
How does it work?
Currently, we are using multi-project pipeline-like approach to run QA pipelines.
graph LR A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2 B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 subgraph "gitlab-foss/gitlab pipeline" A1[`test` stage<br>`package-and-qa-manual` job] end subgraph "omnibus-gitlab pipeline" A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end subgraph "gitlab-qa pipeline" A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1 end
Developer triggers a manual action, that can be found in CE / EE merge requests. This starts a chain of pipelines in multiple projects.
The script being executed triggers a pipeline in Omnibus GitLab and waits for the resulting status. We call this a status attribution.
GitLab packages are being built in the Omnibus GitLab pipeline. Packages are then pushed to its Container Registry.
When packages are ready, and available in the registry, a final step in the Omnibus GitLab pipeline, triggers a new GitLab QA pipeline (those with access can view them at
https://gitlab.com/gitlab-org/gitlab-qa/pipelines). It also waits for a resulting status.
GitLab QA pulls images from the registry, spins-up containers and runs tests against a test environment that has been just orchestrated by the
The result of the GitLab QA pipeline is being propagated upstream, through Omnibus, back to the CE / EE merge request.
On every pipeline during the
test stage, the
review-qa-smoke job is
automatically started: it runs the QA smoke suite against the
You can also manually start the
review-qa-all: it runs the full QA suite
against the Review App.
See Review Apps for more details about Review Apps.
How do I run the tests?
There are two main options for running the tests. If you simply want to run the existing tests against a live GitLab instance or against a pre-built docker image you can use the GitLab QA orchestrator. See also examples of the test scenarios you can run via the orchestrator.
On the other hand, if you would like to run against a local development GitLab environment, you can use the GitLab Development Kit (GDK). Please refer to the instructions in the QA README and the section below.
How do I write tests?
In order to write new tests, you first need to learn more about GitLab QA architecture. See the documentation about it.
Once you decided where to put test environment orchestration scenarios and instance-level scenarios, take a look at the GitLab QA README, the GitLab QA orchestrator README, and the already existing instance-level scenarios.