Automation Jobs

Automation Launching is a Testmo Enterprise feature.
If you’re not sure which plan your Testmo account is on or want to learn more about Testmo Enterprise, please feel free to contact our team!

Many teams execute their automated tests as part of their GitHub Actions or GitLab CI/CD pipelines. With automation launching, Testmo makes it easy to trigger automated test jobs in those pipelines at any time, directly from the Testmo UI.

Tools and Guides

In addition to the documentation on this page, we’ve also prepared some guides & resources on how to use Testmo Automation Launching with specific targets. You can learn more here:

• Announcement: Automation Launching
• Guide: Launching GitHub Actions from Testmo
• Guide: Launching GitLab CI/CD from Testmo
• Docs: Automation Jobs
• Docs: GitHub Automation Jobs
• Docs: GitLab Automation Jobs

Target and Job Configuration

The steps to set up automation launching are similar for both GitHub Actions & GitLab CI/CD. You can simply follow the steps below:

1. Configure the desired integration(s) in the Testmo Admin > Integrations area. Both GitHub and GitLab require secured integrations as automation launching targets, and each tool differs slightly in how the integration is configured. Learn more about how to set up the GitHub and GitLab integrations in our documentation area.

2. In the Testmo Admin > Automation area, add the automation job you want to launch from Testmo as a target. Again, the specifics for configuring a GitHub Target and a GitLab Target are slightly different and you can find full details of how to configure them as automation launching targets in our documentation area. Note that the target configuration relies on a working GitHub or GitLab integration, meaning step 1 above is a prerequisite.

3. Once the target has been successfully added, automation jobs can be launched from the Jobs view in your Testmo project. Simply navigate to the Jobs view, add a new job, or duplicate an existing historic job against the target(s) you previously created.

   Note: To use automation launching, you must already have set up workflows or pipelines in your CI/CD tool.

Basic Automation Launching Workflow

The automation launching workflow assumes test execution occurs in the context of a GitHub or GitLab CI/CD pipeline. In our documentation we assume the Testmo CLI tool is also being used to submit test results to Testmo once the tests have completed, though this isn’t strictly necessary. However, since the automation launching feature provides testers with complete control over when and how test automation is executed, ensuring test results are reported to Testmo ensures a unified view of testing outcomes can be made visible to testers for real time tracking and progress reporting.

 

Automation Launching Targets

Automation targets are the foundation of automation jobs. Automation targets give you a way to specify pipelines, branches, or tags that you would like to be able to trigger ahead of time. While setting up automation targets, you can also include unique parameters, inputs, or variables to trigger pipelines for particular environments, datasets, etc.

As mentioned in the configuration steps above, GitHub and GitLab configuration specifics have a few differentiating features which are called out in the linked documentation, but are comprised of the following building blocks:

• The target name - a unique identifier for the automation job to be launched, on the Jobs view
• The target Connection - the CI/CD platform to be used for the job
• The target settings - the project or repository identifier, workflow & any additional references required (in the case of GitHub)
• The target inputs (for GitHub) or variables (for GitLab) - used to supply additional parameters to the job target. See below for more information.

Automation Launching Parameters

Automation parameters (references in GitHub, variables in GitLab) may be used to supply additional information to the target when triggering a job in Testmo.

The parameter to be used should be specified in the target configuration, either as an input for Github, or as a variable for Gitlab. Once the target is configured, the parameter will be passed to the automation job once the job is triggered via the jobs view. You can learn more about the specific configurations for each platform here:

• GitHub Actions
• GitLab CI/CD

Automation Jobs FAQs

When launching automation jobs from Testmo, your CI provider may return various response codes depending on the outcome of your job. We aim to answer frequently asked questions relating to those responses below.

Testmo will usually display an error message in the log to make it easy to understand what the cause is, however there might be times when this isn’t made perfectly clear from there. Remember that you can also always review the error responses straight from the GitHub or GitLab documentation to complement the information provided by Testmo.
 

GitHub Actions

• If you get this error code with the message “This installation has been suspended”, please ask your GitHub admin to Unsuspend the Testmo App on GitHub.
• If the message is "Resource not accessible by integration", you need to approve the new permissions for GitHub Integration App and reauthorize the connection in Testmo to allow Automation Launching.
  • Navigate to the Testmo GitHub app via https://github.com/apps/testmo-app and hit the Configure button to see which GitHub repositories the app is installed for.
  • Approve the permission change for each repository you want to use the app in.

• You may also need to re-authorize the GitHub connection in Testmo:

  • Navigate to the Admin > Integrations view and click the "Edit" button on your existing connection(s)
  • Click the "Authorize with GitHub" button to re-authorize the app with the new permissions.

GitHub returned error code 404

• If you make a request to access a repository or workflow you don’t have access to or are not mapped in Testmo, you will receive a 404 response.
• Make sure that the repository you are trying to reach is available and is selected in the integration mapping settings in Admin > Integrations > GitHub.
• This error code might also mean that the Testmo GitHub App has been uninstalled. If none of the steps above work, please contact your GitHub admin to verify that the Testmo App is correctly installed on GitHub.

GitHub returned error code 422

• If you omit a required parameter, mistype the parameter name, or input a wrong parameter, reference, or value, you may receive a 422 response. The message will have the details of the specific case.
   

GitLab CI/CD

Testmo will usually display an error message in the log to make it easy to understand what the cause is, however there might be times when this isn’t made perfectly clear from there. Remember that you can also always review the error responses straight from the GitLab documentation to complement the information.

GitLab returned error code 400

• Triggering a pipeline using an incorrect reference (wrong branch name), which will have GitLab return a 400. In Testmo this will be displayed as {"message":{"base”:[ “Reference not found"]}}
• You may have accidentally used main for the branch name in a project that uses a different branch name for its default branch, such as master.

GitLab returned error code 401

• This can be caused by an issue with your connection, rather than the trigger. You can review the status of your connection to GitLab under Admin > Integrations > GitLab, by opening the connection being used by the target and clicking “Test Connection”.

GitLab returned error code 403

• The {"message":"403 Forbidden"} is commonly caused by a permissions issue, such as when the user that owns the application (the one connected to the GitLab integration in Testmo) has any role below the Developer role (Guest, Planner, Reporter).
• If modifying the role of that user isn't possible, you should switch the application in use by the connection to an Owner or Maintainer of the Project.

GitLab returned error code 404

• The {"message":"404 Project Not Found"} response can be caused by pointing to a non-existent or mispelled group or project name/ID. Double-check your project group and/or project name/ID in the corresponding Automation target you configured in the Admin > Automation view.
   

Further FAQ's

• Does the pre-existing connection remove the need to create a token?
  • Yes in terms of tokens specific to triggering pipelines, workflows, etc in your tools, but remember that the standard connection requires either a token or an integration to establish the connection, which in turn is used for your target.
• How is an automation target added?
  • The target is added under Admin -> Automation. In the case of GitLab/Github for example, assuming you’ve set up a connection to the instance in our Integrations tab, then you can proceed to adding a target. It’s at this stage that you’ll need information specific to your workflow or pipeline to properly create the target (in the case of GitLab this would be your group/project name and reference).
• In the GitLab target creation panel, what are the “project path” and the “reference”?
  • The project path is the GitLab group and name of the project/ID (Ex: group/myproject), while the reference is either a branch or tag name (Ex: main or master).
• Is it possible to relaunch past jobs?
  • Yes, we have a Quick Launch function at the top right of the Jobs view in your project, where the latest targets used will be presented.
• Why can I only delete one job at a time?
  • It’s definitely possible to delete more than one job. The way to accomplish this is by selecting multiple jobs, and then using the dropdown options for any of the jobs. You’ll be presented with the “Delete jobs” function!
• I think something went wrong, I’m not seeing the run show up in Testmo.
  • If an issue occurs with one of your jobs, whenever possible Testmo will display the log for you to review the potential cause. Simply click the job itself or on “Details” to see what the log presents.
• My job ran correctly, however I’m not seeing my run. What could be causing this?
  • Though the log can reflect issues with the connection, from the actuation of the job to the triggering, issues with the test or submission of the results to Testmo will not be displayed. To understand the potential causes you can review the tests themselves, and if the configuration of your result file submission process.

For any other questions or issues, please feel free to contact our support team at contact@testmo.com.