Requirements Coverage & Traceability

The Issue Coverage & Traceability Report enables you to report on how requirements are covered by test cases, plans, and tests.

Requirements, use cases, or other scenarios that you may need to test against often exist in the form of issues, epics, stories, bugs, or other ticket types logged in issue trackers like Jira, GitHub Issues, or GitLab. You can link those requirements to test cases, test runs, milestones, and sessions in Testmo by adding the issue ID from your external issue tracker to the Issues field on the related objects in Testmo. This allows you to trace your requirements through your test cycle and ensure you are covering the full scope of what needs to be tested.

This page explains how to use the Requirements Coverage & Traceability Report to show how your issues are covered by planned testing activities, and what the latest results of those tests are.

Note: The Coverage & Traceability report includes issues linked to test cases, test runs, milestones, and sessions. Issues linked to test run results or automation test comments are not included in the Issue Coverage & Traceability report. A dedicated test automation report will be made available in a future release.

The report configuration panel consists of 3 main areas, covered in each section below. Many of the configuration options correspond with controls in our existing PDF reports, which you can read more about in the guide: https://www.testmo.com/blog/print-pdf-reports/

Generating the Report

The Options section controls the scope of the report to be generated, starting with project selection and gradually drilling down to further selections enabling fine-grained control over objects to be included in the report.

Project

Use this dropdown to select which single project the report will cover. Reports can be generated for one project at a time, and cannot retrieve data from multiple projects in the same report.

Issues

Use this dropdown to determine the level of issue coverage for the report. You have 3 options to choose from:

1. All linked issues: If you want to include all issues which are linked to cases, results, runs, sessions & milestones in the scope of your report, choose this option.
2. Select trackers: If you use more than one issue tracker in your project, and wish to limit the scope of your report by one or more trackers, use this option (by clicking the window button). Note that your selection here is overridden if you select option 1 “All linked issues”, or option 2 by entering specific issue IDs.
3. Enter issue ID’s: If you wish to constrain your report to a specified list of issue ID’s, you may enter them (one per line) in the Enter issues dialog box (by clicking the window button).

Cases

The cases dropdown enables you to select which cases should be included in the scope of your report. You again have 3 options to select from:

1. All cases: If you want to include all cases from your project in the scope of your report, choose this option.
2. Filter cases: If you want to define and apply a filter to your cases, select this option and define the filter in the dialog box (by clicking the filter button).
3. Select cases: If you want to select specific cases for your report, select this option and apply your selections in the dialog (by clicking the window button).

Runs

The runs dropdown enables you to select which runs should be included in the scope of your report. You have 4 options to select from:

1. All runs: Choose this option if you want to include all the runs from your project in your report. Note that selecting this option will include runs marked as closed as well as open/active runs.
2. All active runs: Choose this option if you only want to include open/active runs and exclude closed runs from your report.
3. All closed runs: Choose this option if you only want to include closed runs and exclude open runs from your report.
4. Filter runs: Choose this option if you want to define a custom filter for which runs are included in the scope of your report (by clicking the filter button).

Sessions

The sessions dropdown enables you to select which sessions should be included in the scope of your report. You have 4 options to select from:

1. All sessions: Choose this option if you want to include all the sessions from your project in your report. Note that selecting this option will include sessions marked as closed as well as open/active sessions.
2. All active sessions: Choose this option if you only want to include open/active sessions and exclude closed sessions from your report.
3. All closed sessions: Choose this option if you only want to include closed sessions and exclude open sessions from your report.
4. Filter sessions: Choose this option if you want to define a custom filter for which sessions are included in the scope of your report (by clicking the filter button).

Milestone

This field may be used to configure a single milestone filter which is applied across both runs and sessions, rather than doing so for each individually. Note that this filter will override any filters which have already been applied on the run & session filter configuration. Only a single milestone may be selected at a time using this controller, versus multiple milestones using the run &/or session filters. Note that only runs & sessions from the selected milestone will be included in the report, not those from sub milestones. (If you want to specify sub milestones to include, leave this setting blank and use run/session filters instead).

Created

This field may be used to configure a single creation-date filter which is applied across both runs and sessions, rather than doing so for each individually. Note that this filter will override any filters which have already been applied on the run & session filter configuration.

Using the Report

The report content controls determine what major sections will be included in the generated report. Each section can be toggled off or on by unchecking or checking the relevant checkbox. By default, all sections are included in the finished report.

Key Metrics

This initial section of the report contains high level metrics for the configured scope, in summary form. Depending on the options selected, the key metrics will display total numbers in the report for:

• Issues
• Test cases
• Runs
• Sessions

Coverage chart

This section displays a stacked bar chart view of issue coverage across the selected report options (issues, cases, runs, sessions), making it easy for consumers of the report to see how well issues are covered.

Coverage statistics:

This section displays a tabular version of the issue coverage metrics, with counts and percentages. If you have more than one issue tracker integrated with your project, you’ll see a breakdown of issues by tracker as well.

Issues

The issues table contains a detailed breakdown of all issues in the scope of the report along with details of how they’re covered by cases, runs and sessions (depending on selected report options).

Show status column

Gauge whether your issues are ready to ship, or not, by displaying the latest status of the tests covering them.
 

Example: reporting coverage for a list of issues from Jira

You can follow example steps below to create a requirements coverage & traceability report of your Jira issues.

Jira issues have test case coverage and results traceability

Prerequisite steps

1. Ensure you’ve setup the necessary integration with Jira first, by following the steps here: Jira Cloud Integration Guide
2. Create the necessary issues in Jira. Typically, these will take the form of Epics, Stories, Tasks or Bugs. For this example we’ll assume you’re using Jira Stories as placeholders for requirements (AKA some feature or improvement in your product).
3. Add the Jira issue references to the test cases, sessions or runs, which cover your Stories.
4. Execute your tests.

Creating the report with all issues included

Once the prerequisite steps have been completed, you can create a requirements coverage report for your Jira issues, as follows. On the Requirements coverage & traceability report page:

1. Select your project.
2. If you have more than one issue tracker integrated with your project, click the Issues dropdown, click the Select trackers option, click the dialogue icon and select only the Jira tracker.
3. Click the Generate report button.

The report will show you coverage for all Jira issues which are linked to your cases, runs & sessions:

High level metrics:

• Numbers of issues, test cases, runs, sessions covering your issues
• Chart breakdown
• Total issues and issues with coverage
• Issue trackers in scope (Jira, in this case)

Detailed metrics:

• A breakdown of case, run & session coverage by issue - with the issue ID on the left, the entity type, case, run or session name, and latest test status (if you’re using the default “Show status column” setting

Creating the report with explicitly specified issues

If you want to take the report a step further, and show which Jira issues aren’t covered by your cases, runs or sessions, we need to apply a slightly different approach:

1. Click the Issues dropdown and select the Enter issue IDs option.
2. Click the dialogue button in the Issues dropdown.
3. Enter the issue IDs you wish to report on, in a list, then click Save.
4. Once done, click the Generate report button.

With the issues explicitly specified, Testmo can report on whether or not all of them are covered by your cases, runs or sessions. Thereby helping you to identify potential gaps in your testing:

• The number of issues without coverage are highlighted in the graph, and in the coverage statistics.
• Issues without coverage are also reported on in a separate table in an additional Issues without coverage table.

Reporting coverage for a list of issues from Jira

Example: reporting on coverage for automated tests

You can follow example steps below to create a requirements coverage report for your automated tests.

Prerequisite steps

In order to create a requirements coverage report for your automated tests, you’ll need to follow similar steps as already outlined above, with an additional automation linking step.

Assuming that you’re still using Jira as the source of your requirements:

1. Ensure you’ve setup the necessary integration with Jira first, by following the steps here: Jira Cloud Integration Guide
2. Create the necessary issues in Jira.
3. Add the Jira issue references to the test cases, sessions or runs, which cover your Stories.
4. Link your test cases to the automated tests which cover them. Follow the instructions here: Automation Linking
5. Execute your automated tests.

Creating the report with all issues included

Once the prerequisite steps have been completed, you can create a requirements coverage report for your Jira issues and your automation coverage, as follows.

On the Requirements coverage & traceability report page:

1. Select your project.
2. If you have more than one issue tracker integrated with your project, click the Issues dropdown, click the Select trackers option, click the dialogue icon and select only the Jira tracker.
3. Click the Cases dropdown, select Filter cases and click the Filter icon to open the filter dialogue. Select the Automation dropdown, and set it to Yes. Applying this setting means that only cases with automation coverage will be visible in the report.
4. Uncheck the Runs & Sessions Options.
5. Click the Generate report button.

The report will show you coverage for all Jira issues which are linked to your cases, and which also have automated test coverage.

If you want to see which test cases have requirements but no automation test coverage, you can simply change the Cases > Automation filter to No, instead of Yes.

Report View Settings

The report view settings can be used to enable or disable components of the report so they can more easily be customised and shared to suit different use cases.

Show Links

By default, reports include links to the relevant pages in Testmo. For example, you can quickly navigate to tests by clicking their names. Or you can navigate to a test run by clicking its ID. Alternatively, if you want to share reports with persons who don’t have access to Testmo or shouldn’t even know where the tests are, you can completely disable and remove all links, so you can share reports without any links back to your Testmo account, by unchecking this option.

Show Persons

A major benefit of using Testmo for your projects is that it’s easy to track and understand which person on your team made changes to test cases or added results. Testmo also helps improve collaboration and task management by allowing you to assign tests to team members. If you don’t want to share that information in your report however, you can easily hide all personal names and avatars by unchecking this option.

Show Header & Show Footer

Don’t want to include a header or Testmo branded footer in your report? No problem, you can also disable those by unchecking this option.
 

Exporting & Sharing the Report

Once your report includes all the data and content you wish to include based on the settings above, you can share it with your team and stakeholders by clicking on the Print & PDF button.

Testmo utilises the built-in capabilities of your web browser to generate the Print or PDF version of your report, which you can send to your printer or save to a PDF using your browser controls.