CI/CD Integration

Many teams execute their automated tests as part of their CI/CD pipeline or during nightly builds on their build server. It's easy to submit your automated test results to Testmo from your CI/CD and build workflow by using Testmo's powerful CLI tool. You can also use the same approach to submit your test results from any manually launched test automation suites.

Tools and guides

In addition to the documentation on this page, we've also prepared some guides and resources on how to use the Testmo CLI for specific integrations such as integrating with GitHub Actions or GitLab CI/CD. You can learn more here:

• Bitbucket
• CircleCI
• GitHub Actions
• GitLab CI/CD
• Jenkins
• Custom/other

Pipeline and workflow integration

Regardless of the CI/CD tool or build system you use, the integration is similar in all environments. You would simply follow these steps in most cases:

During your CI/CD pipeline run or build, you install the Testmo CLI tool. This is usually done with a single command using npm and the CLI is easily installed on any OS and in any environment. You can learn more about installing the CLI tool.

  You run your automated tests as usual and tell your automation tool or framework to generate a JUnit XML report file (practically any tool and framework supports this file format or can convert to it). Alternatively you can tell our CLI to start your tests; this way we can also capture the full console output, record exit codes and measure test times. 

You then use our CLI tool to submit your test results. Testmo automatically reads all tests, creates the test run and submits the results.

Testmo also supports advanced workflows such as parallel CI/CD test jobs. It's also possible to include additional fields, links and artifacts such as a link to your Git commit or CI tool. You can find examples illustrations of common CI/CD pipelines and the typical Testmo integration below.

Basic testing workflow

In a typical basic CI/CD pipeline, you usually run your tests after building or setting up your code. You can then use Testmo's CLI tool to submit your test results with its simple automation:run:submit command. This command reads your automation result file, creates a new test run in Testmo and submits all results. It can optionally also report the console output, exit code and execution times of your automated tests.

If you run multiple automation test suites during your build, such as separate frontend and backend tests for example, you would also submit multiple runs to Testmo.

You can learn more about using the CLI to submit test runs.

Parallel testing workflow

Testmo also fully supports advanced CI/CD workflows such as parallel testing. To submit test runs with multiple parallel test jobs, you would first create the run, then pass the run ID to the separate jobs, submit the jobs' results (with a separate thread) and finally mark the run as completed.

You can learn more about submitting parallel test jobs.

Important security considerations

When working with the Testmo API and CLI to submit test automation results to Testmo, you need to specify an API key for the authentication. You can learn more about generating API keys in Testmo here.

When implementing Testmo in your CI/CD pipeline, it is important not to store the API key in your code repository. If you store the API key in your code repository (e.g., in Git), everyone with access to the repository would be able to access the key. You should never add and store your Testmo API key as part of your code repository.

Instead, all CI/CD tools provide a way to store such secrets and make them available as environment variables during the pipeline run. This way you cannot directly access the secret API key in the code repository.

We include examples on how to securely store the Testmo API key in the various tools in the above articles. And you can learn more about secret management in the different tools here:

• GitHub Actions secret management
• CircleCI secret management
• GitLab CI/CD project variables
• BitBucket variables and secrets
• Jenkins credentials