Testpilot GitLab CI/CD Integration

You can check out an example repo with our Gitlab integration for more details.

Getting Started

Prerequisites

Before you begin, make sure you have:

A GitLab project with CI/CD enabled
At least one Testpilot test file (.pilot.yaml) in your repository
Access to GitLab runners (SaaS or self-hosted)

Step 1: Create Your First Pipeline

Let’s start by creating a simple GitLab CI pipeline that installs Testpilot and runs a test. Create or edit your .gitlab-ci.yml file in your project root:

# Include the Testpilot component
include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install

# Define pipeline stages
stages:
  - install
  - test

# Your test job
run-my-tests:
  stage: test
  needs: ["testpilot-install-linux-amd64"] # Wait for Testpilot installation
  script:
    - ./bin/testpilot test your_test.pilot.yaml # Replace with your test file
  # preserve artifacts and reports
  artifacts:
    reports:
      junit: testpilot-out/*.xml
    paths:
      - testpilot-out
    when: always
    expire_in: 1 week

Step 2: Commit and Run

Now that you have your pipeline defined, let’s run it:

Save your .gitlab-ci.yml file
Commit and push to your GitLab repository
Navigate to your project’s CI/CD > Pipelines section
Watch your pipeline run!

You should see two jobs:

testpilot-install-linux-amd64 - Downloads and sets up Testpilot
run-my-tests - Runs your Testpilot tests

Step 3: View your Artifacts

The artifacts section of the job above will preserve a static report of your Testpilot tests, including images and videos. You can view it by browsing to the Artifacts section of your Gitlab project. Testpilot also generates a JUnit report that can be directly reported to your merge requests

Step 4: Understanding What Happened

The Testpilot component automatically:

Downloaded the Testpilot installer
Installed Testpilot and its dependencies (Playwright, Node.js)
Created a ./bin/testpilot launcher script
Cached everything for faster future runs
Made Testpilot available to your test job

Platform-Specific Setup

Testpilot supports running your tests on Linux and macOS runners. You can easily configure your pipeline to handle both platforms:

Setting Up for Linux (Default)

The component works out-of-the-box with GitLab’s standard Linux runners. No additional configuration needed!

include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install

stages:
  - install
  - test

run-tests:
  stage: test
  needs: ["testpilot-install-linux-amd64"]
  script:
    - ./bin/testpilot test your_test.pilot.yaml

Setting Up for macOS

Since macOS runners can’t (currently) run Docker Images directly, they require a slightly different setup. Here’s how to configure your pipeline to use GitLab’s SaaS macOS runners: Step 1: Define your macOS runner configuration at the top of your .gitlab-ci.yml:

# macOS runner configuration
.macos_saas_runners:
  tags:
    - saas-macos-medium-m1 # Use GitLab's SaaS macOS runners

Step 2: Include the component with macOS-specific settings:

# Include the Testpilot component for macOS
include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install # Install Testpilot in the 'install' stage
      platform: macos # Target the macOS platform
      runner-extends: .macos_saas_runners # Use the macOS SaaS runner config

# Define pipeline stages
stages:
  - install
  - test

# Run your tests on macOS
run-tests-macos:
  stage: test
  extends: .macos_saas_runners
  needs: ["testpilot-install-macos"]
  script:
    - ./bin/testpilot test your_test.pilot.yaml # Replace with your test file

Setting up Custom Runners

If you’re using self-hosted runners, you can configure Testpilot to run on them using a configuration like the following

# Custom runner configurations
.arm_runner:
  tags:
    # custom runner configuration goes here

include:
  # ARM64 runner
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install
      platform: linux-arm64
      runner-extends: .arm_runner

stages:
  - install
  - test

test-arm:
  stage: test
  extends: .arm_runner
  needs: ["testpilot-install-linux-arm64"]
  script:
    - ./bin/testpilot test tests/

Multi-Platform Testing

Want to test on both Linux and macOS? Here’s how:

# Runner configurations
.macos_saas_runners:
  tags:
    - saas-macos-medium-m1

# Install Testpilot on both platforms
include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install
      # platform: linux-amd64 is the default
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install
      platform: macos
      runner-extends: .macos_saas_runners

stages:
  - install
  - test

# Run tests on Linux
test-linux:
  stage: test
  needs: ["testpilot-install-linux-amd64"]
  script:
    - ./bin/testpilot test tests/

# Run tests on macOS
test-macos:
  stage: test
  extends: .macos_saas_runners
  needs: ["testpilot-install-macos"]
  script:
    - ./bin/testpilot test tests/

Sharding your Tests and Running in Parallel

Testpilot can shard your test files across multiple runners using the --shard flag, which is useful for parallelizing and speeding up your test runs. Note that after sharding and running your tests in parallel, you will probably want to merge the reports back together using testpilot report merge. You can see an example of sharding testpilot files in our example repo. An example gitlab.yaml file is below:

include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install
      xdg_cache_home: .cache/jetify

stages:
  - install
  - test
  - merge

testpilot-test:
  stage: test
  needs: ["testpilot-install-linux-amd64"]
  parallel: 2
  # Use built in CI_NODE environment variables to shard across runners
  script: ./bin/testpilot test --outdir=testpilot-out-${CI_NODE_INDEX} --shard=${CI_NODE_INDEX}/${CI_NODE_TOTAL} --reporters=junit
  artifacts:
    reports:
      junit: testpilot-out/*.xml
    paths:
      - testpilot-out-${CI_NODE_INDEX}/
    when: always
    expire_in: 1 week

# Merge your test reports together into a single artifact
testpilot_merge:
  stage: merge
  needs:
    - "testpilot-install-linux-amd64"
    - job: "testpilot-test"
      artifacts: true
      optional: true
  script: ./bin/testpilot report merge --outdir=testpilot-out testpilot-out-*
  artifacts:
    paths:
      - testpilot-out/
    when: always
    expire_in: 1 week

Understanding the Component

Behind the Scenes

When you include the Testpilot component, it creates a job that:

Downloads the installer from https://get.jetify.com/testpilot
Installs Testpilot and its dependencies (Playwright, Node.js)
Creates a launcher script at ./bin/testpilot that handles environment setup
Caches everything so future pipeline runs are faster
Shares the installation with your test jobs via GitLab artifacts

Component Configuration Options

The component accepts these parameters:

Parameter	Required	Default	Description	Example
`stage`	✅ Yes	-	Which stage to install Testpilot in	`install`
`platform`	No	`linux-amd64`	Target platform	`macos`, `linux-arm64`
`runner-extends`	No	-	Custom runner configuration	`.macos_saas_runners`
`xdg_cache_home`	No	`.cache`	Cache directory location	`.custom_cache`

Caching and Performance

Caching Strategy

The component automatically caches:

The Testpilot binary and dependencies
Your ./bin/testpilot launcher script

This means the second time your pipeline runs, Testpilot installation will be much faster!

Custom Cache Directory

If you need to change where files are cached:

include:
  - component: gitlab.com/jetify-com/gitlab-components/testpilot-install@main
    inputs:
      stage: install
      xdg_cache_home: .my_custom_cache

run-tests:
  stage: test
  needs: ["testpilot-install-linux-amd64"]
  script:
    - ./bin/testpilot test tests/

Best Practices Summary

✅ Do:

Always use needs: to ensure proper job dependencies
Leverage the built-in caching for faster pipelines
Use descriptive job names that reflect what you’re testing
Test on multiple platforms if your application supports them
Structure your tests in logical directories

❌ Don’t:

Forget to specify the correct job name in needs:
Modify the cache directory unless necessary
Run tests without waiting for the installation job
Use the same job name for multiple platforms

Overview

Getting Started

Guides

Integrations

Advanced

Evaluation

Reference

Best Practices

Changelog

Testpilot GitLab CI/CD Integration

Getting Started

Prerequisites

Step 1: Create Your First Pipeline

Step 2: Commit and Run

Step 3: View your Artifacts

Step 4: Understanding What Happened

Platform-Specific Setup

Setting Up for Linux (Default)

Setting Up for macOS

Setting up Custom Runners

Multi-Platform Testing

Sharding your Tests and Running in Parallel

Understanding the Component

Behind the Scenes

Component Configuration Options

Caching and Performance

Caching Strategy

Custom Cache Directory

Best Practices Summary

​Getting Started​

​Prerequisites​

​Step 1: Create Your First Pipeline​

​Step 2: Commit and Run​

​Step 3: View your Artifacts​

​Step 4: Understanding What Happened​

​Platform-Specific Setup​

​Setting Up for Linux (Default)​

​Setting Up for macOS​

​Setting up Custom Runners​

​Multi-Platform Testing​

​Sharding your Tests and Running in Parallel​

​Understanding the Component​

​Behind the Scenes​

​Component Configuration Options​

​Caching and Performance​

​Caching Strategy​

​Custom Cache Directory​

​Best Practices Summary​

Getting Started

Prerequisites

Step 1: Create Your First Pipeline

Step 2: Commit and Run

Step 3: View your Artifacts

Step 4: Understanding What Happened

Platform-Specific Setup

Setting Up for Linux (Default)

Setting Up for macOS

Setting up Custom Runners

Multi-Platform Testing

Sharding your Tests and Running in Parallel

Understanding the Component

Behind the Scenes

Component Configuration Options

Caching and Performance

Caching Strategy

Custom Cache Directory

Best Practices Summary