Browser Performance Testing (PREMIUM)
If your application offers a web interface and you're using GitLab CI/CD, you can quickly determine the rendering performance impact of pending code changes in the browser.
NOTE: You can automate this feature in your applications by using Auto DevOps.
GitLab uses Sitespeed.io, a free and open source
tool, for measuring the rendering performance of web sites. The
Sitespeed plugin that GitLab built outputs
the performance score for each page analyzed in a file called
this data can be shown on Merge Requests.
Consider the following workflow:
- A member of the marketing team is attempting to track engagement by adding a new tool.
- With browser performance metrics, they see how their changes are impacting the usability of the page for end users.
- The metrics show that after their changes, the performance score of the page has gone down.
<head>, which affects loading page speed.
- They ask for help from a front end developer, who sets the library to load asynchronously.
- The frontend developer approves the merge request, and authorizes its deployment to production.
How browser performance testing works
First, define a job in your
.gitlab-ci.yml file that generates the
Browser Performance report artifact.
GitLab then checks this report, compares key performance metrics for each page
between the source and target branches, and shows the information in the merge request.
For an example Performance job, see Configuring Browser Performance Testing.
If the Browser Performance report has no data to compare, such as when you add the
Browser Performance job in your
.gitlab-ci.yml for the very first time,
the Browser Performance report widget doesn't show. It must have run at least
once on the target branch (
master, for example), before it displays in a
merge request targeting that branch.
Configuring Browser Performance Testing
First, set up GitLab Runner with a Docker-in-Docker build.
Configure the default Browser Performance Testing CI/CD job as follows in your
include: template: Verify/Browser-Performance.gitlab-ci.yml performance: variables: URL: https://example.com
In GitLab 14.0 and later, the job is scheduled to be renamed
The above example:
- Creates a
performancejob in your CI/CD pipeline and runs sitespeed.io against the webpage you defined in
URLto gather key metrics.
- Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster,
- Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using GitLab 12.3 or earlier, you must add the configuration manually.
The template uses the GitLab plugin for sitespeed.io, and it saves the full HTML sitespeed.io report as a Browser Performance report artifact that you can later download and analyze. This implementation always takes the latest Browser Performance artifact available. If GitLab Pages is enabled, you can view the report directly in your browser.
You can also customize the jobs with CI/CD variables:
SITESPEED_IMAGE: Configure the Docker image to use for the job (default
sitespeedio/sitespeed.io), but not the image version.
SITESPEED_VERSION: Configure the version of the Docker image to use for the job (default
SITESPEED_OPTIONS: Configure any additional sitespeed.io options as required (default
nil). Refer to the sitespeed.io documentation for more details.
For example, you can override the number of runs sitespeed.io makes on the given URL, and change the version:
include: template: Verify/Browser-Performance.gitlab-ci.yml performance: variables: URL: https://www.sitespeed.io/ SITESPEED_VERSION: 13.2.0 SITESPEED_OPTIONS: -n 5
Configuring degradation threshold
Introduced in GitLab 13.0.
You can configure the sensitivity of degradation alerts to avoid getting alerts for minor drops in metrics.
This is done by setting the
DEGRADATION_THRESHOLD CI/CD variable. In the example below, the alert only shows up
Total Score metric degrades by 5 points or more:
include: template: Verify/Browser-Performance.gitlab-ci.yml performance: variables: URL: https://example.com DEGRADATION_THRESHOLD: 5
Performance testing on Review Apps
The above CI YAML configuration is great for testing against static environments, and it can be extended for dynamic environments, but a few extra steps are required:
performancejob should run after the dynamic environment has started.
- In the
- Generate a URL list file with the dynamic URL.
- Save the file as an artifact, for example with
echo $CI_ENVIRONMENT_URL > environment_url.txtin your job's
- Pass the list as the URL environment variable (which can be a URL or a file containing URLs)
- You can now run the sitespeed.io container against the desired hostname and paths.
.gitlab-ci.yml file would look like:
stages: - deploy - performance include: template: Verify/Browser-Performance.gitlab-ci.yml review: stage: deploy environment: name: review/$CI_COMMIT_REF_SLUG url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN script: - run_deploy_script - echo $CI_ENVIRONMENT_URL > environment_url.txt artifacts: paths: - environment_url.txt only: - branches except: - master performance: dependencies: - review variables: URL: environment_url.txt
GitLab versions 13.2 and earlier
Browser Performance Testing has gone through several changes since its introduction. In this section we detail these changes and how you can run the test based on your GitLab version:
In 13.2 the feature was renamed from
Browser Performancewith additional template CI/CD variables.
In GitLab 12.4 a job template was made available.
For 11.5 to 12.3 no template is available and the job has to be defined manually as follows:
performance: stage: performance image: docker:git variables: URL: https://example.com SITESPEED_VERSION: 14.1.0 SITESPEED_OPTIONS: '' services: - docker:stable-dind script: - mkdir gitlab-exporter - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js - mkdir sitespeed-results - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:$SITESPEED_VERSION --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL $SITESPEED_OPTIONS - mv sitespeed-results/data/performance.json performance.json artifacts: paths: - performance.json - sitespeed-results/ reports: performance: performance.json
For 11.4 and earlier the job should be defined as follows:
performance: stage: performance image: docker:git variables: URL: https://example.com services: - docker:stable-dind script: - mkdir gitlab-exporter - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js - mkdir sitespeed-results - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL - mv sitespeed-results/data/performance.json performance.json artifacts: paths: - performance.json - sitespeed-results/
Upgrading to the latest version and using the templates is recommended, to ensure you receive the latest updates, including updates to the sitespeed.io versions.