Automation Metrics

The automation metrics report provides users with a high level overview of a project’s test automation executions and results. The results of automation runs can be viewed from different perspectives, including results timelines, timing trends and test count statistics.

By default the report will provide listings of automation runs in the reports scope, configurable to more or less granular levels of detail (cards & tables, respectively). By selecting an appropriate time frame and grouping the results by day, week or month, users can see how automation results have trended or changed over different time periods.

The report configuration panel consists of 3 main areas, covered in each section below. Many of the configuration options correspond with controls in our existing PDF reports, which you can read more about in the guide: Flexible Print & PDF Reports - Export, Share & Review - Testmo

Generating the Report

The Options section controls the scope of the report to be generated, starting with project selection and gradually drilling down to further selections enabling fine-grained control over objects to be included in the report.

Project

Use this dropdown to select which single project the report will cover. Reports can be generated for one project at a time, and cannot retrieve data from multiple projects in the same report.

Milestone

Use this dropdown to select which milestones the report will cover. Unlike other Testmo reports, multiple milestones may be selected. If a milestone has children and you wish to include automation runs linked to them in the report, you should select those also, since sub-milestones won’t automatically be included based on the selection of a parent.

Source

Use the Source dropdown to filter from which sources you wish to include automation runs. By default, the report will include automation runs from all sources.

Runs

The Runs dropdown enables you to make a quick selection as to whether you wish to include all runs (the default setting), all successful or all failed runs in the scope of your report. Alternatively, you can apply a filter to your runs by selecting the Filter option and clicking the filter icon. Note that filter settings applied in the Runs filter may be overridden by other option settings on the report.

Time frame

Use this dropdown to make a quick selection as to the period the report should cover. By default, the report includes automation runs completed in the last 30 days. If you need a more custom time frame, apply a custom date range in the Runs filter and leave the Time frame field blank.

Group by

The Group by dropdown should be used to determine how you wish data to be grouped in the report charts and tables, by day, week or month. Metrics will be aggregated in chart columns and table rows accordingly.

Average test & thread counts, and run times

The average test counts, average run times, and average thread counts checkboxes may be used to control whether these metrics are displayed on the Run Metrics chart and Automation Statistics table. Unselecting or selecting them will remove or add chart elements and table columns accordingly.

Content

How much information the report will contain can be controlled using the Content settings. Check or uncheck the following controls depending on whether you want to display or omit the information accordingly:

• Timeline chart: Controls whether the automation runs across time chart is displayed. The Timeline chart shows successful and/or failed automation runs over the selected Time frame, grouped by day, week or month according to the Group by setting. The timeline chart will contain columns for each day, week or month on which automation runs were completed.

• Metrics chart: Controls whether the run, thread and duration averages chart is displayed. The Run metrics chart provides a different perspective on your automation runs, supplying information about the average number of tests and threads for the period and grouping options you’ve selected. The chart will also show you the average time taken for your runs to complete, for each group during the period. You can enable or disable the metrics individually in the content panel, or disable the entire chart using the Metrics chart checkbox.

• Automation statistics: Controls whether the detailed automation run stats table is displayed. The Automation statistics table provides a detailed breakdown of the automation runs included in your report, grouped according to your Control panel setting for the selected Time frame. The table will indicate where there are days without runs on separate rows. Table columns should be read as follows:
  • Day: The date on which the automation runs started. Note that this means automation runs in a state of “Running” may be included in the report.
  • Runs: This column is comprised of 3 components:
    • Always shows the total number of runs completed on the day.
    • How many of those runs passed or failed. By default, the column will show how many runs passed, in brackets with a green square. If you select All failed runs from the Runs dropdown however, you’ll be shown how many tests failed instead, with a red square (indicating failures rather than successes). If you select All runs, or apply a custom filter including both passed and failed runs, you’ll be shown the number of successful runs.
• Tests per run: Shows the average number of tests per run and an indicator as to whether the number has increased or decreased since the last average was calculated (usually on the previous row). Increases are rendered in green, decreases in red.
• Run times: Shows the average time taken to execute the automation runs, on the specified day.
• Threads: Shows the average number of threads across which automation runs were executed, on the the specified day.

• Automation runs: Controls whether a more granular breakdown for each run is displayed, in either card or table format, in addition to the maximum allowed number of runs. Automation runs will be grouped according to the Options setting (by day, week or month).
  • Display as: Controls whether runs are displayed as individual cards or as lists.
    • Cards: If you want more granular information about each automation run, choose this option in order to display all status metrics and attributes associated with the run.
    • Lists: If you’re ok with a higher level view of your automation run information, only displaying the name, source, status & timestamp for each automation run, choose this view.
  • Maximum runs: Limits the number of automation runs to the selected maximum. The limit is set to 200 by default.

Using the Report

Example: Did we increase the number of passing automated UI tests during the last sprint?

You can follow the example steps below to create a report which shows the progression of your automated test metrics over a period of time (e.g. a sprint).

Creating the report

On the Automation metrics report page:

1. Select your project.
2. Apply a source filter so it constrains the report to only the UI tests (or tests of interest if something other than the UI, e.g. API tests).
3. Select the Runs > Filter runs option and apply a custom date range matching the start & end dates of the sprints you want to compare across (e.g. a 4 week period covering the start and end dates of your last 2 sprints).

When applying a custom date range, the Time frame field will automatically be set to All time, enabling the custom period to override the setting. Changing it to another time frame will override the custom duration. You can revert to the custom duration by setting the Time frame option back to All time.

1. Set the Group by option to Day (the default option).
2. Click the Generate button.

Information about your automation tests can be derived from various places on the report:

• The Run Timeline by Day chart will show how many automation runs were executed each day for the selected, period grouped by whether the runs passed or failed.
• The Run Metrics by Day chart will show the average number of tests per run, per day, along with their average durations and thread counts. We’re looking for an upward trend in the average test count here.
• The Automation Statistics chart will unequivocally answer the question we’re asking. The Tests per run count should be increasing over the period. We can also see from the Tests per run +/- value whether the number of tests has increased (in green) or decreased (in red).

Report View Settings

The report view settings can be used to enable or disable components of the report so they can more easily be customised and shared to suit different use cases.

Show Links

By default, reports include links to the relevant pages in Testmo. For example, you can quickly navigate to tests by clicking their names. Or you can navigate to a test run by clicking its ID. Alternatively, if you want to share reports with persons who don’t have access to Testmo or shouldn’t even know where the tests are, you can completely disable and remove all links, so you can share reports without any links back to your Testmo account, by unchecking this option.

Show Persons

A major benefit of using Testmo for your projects is that it’s easy to track and understand which person on your team made changes to test cases or added results. Testmo also helps improve collaboration and task management by allowing you to assign tests to team members. If you don’t want to share that information in your report however, you can easily hide all personal names and avatars by unchecking this option.

Show Header & Show Footer

Don’t want to include a header or Testmo branded footer in your report? No problem, you can also disable those by unchecking this option.

Exporting & Sharing the Report

Once your report includes all the data and content you wish to include based on the settings above, you can share it with your team and stakeholders by clicking on the Print & PDF button.

Testmo utilises the built-in capabilities of your web browser to generate the Print or PDF version of your report, which you can send to your printer or save to a PDF using your browser controls.