Testmo Automation Linking CLI Extension

Automatically link your test automation results to test cases in your Testmo repository with the Testmo automation linking tool.

About Automation Linking

When you submit test automation results to Testmo, your test results appear in automation runs but are not automatically connected to your test cases in the repository. The automation linking tool solves this by automatically creating links between your automation test results and your repository test cases based on configurable rules.

This makes it easy to:

• Track automation coverage - See which test cases have automated tests linked to them
• Monitor test execution history - View the latest automation results directly in your test case repository
• Identify gaps - Quickly find test cases without automation coverage
• Maintain traceability - Keep your test repository synchronized with your automation suite

The automation linking tool works seamlessly with your existing Testmo CLI workflow. You continue to submit your test results with the standard testmo command, and then run the linking tool as a separate step to create the connections.

Supported languages and frameworks

Language Support: The automation linking tool is language-agnostic and works with any programming language and test framework combination, as long as your tests can export results in JUnit XML format.

Supported Languages (for annotation scanning):

• Python (pytest, unittest, nose)
• Java (JUnit, TestNG)
• JavaScript/TypeScript (Jest, Mocha, Jasmine, Vitest)
• C# (.NET, NUnit, xUnit, MSTest)
• Ruby (RSpec, Minitest)
• Go (go test)
• Rust (cargo test)
• PHP (PHPUnit)
• Kotlin (JUnit)
• Scala (ScalaTest)
• Any other language with text-based comments

Framework Requirement: Your test framework must be able to export test results in JUnit XML format. Most popular testing frameworks support this either natively or via plugins.

Design goals

The automation linking tool is designed to support various use cases and scenarios:

• Works as a post-submission step after your existing Testmo CLI workflow
• Supports multiple linking methods: source code annotations, explicit configuration mappings, and pattern matching
• Integrates with any CI/CD and build system such as GitHub Actions, GitLab CI, Jenkins, and more
• Language-agnostic: Works with any test automation framework that generates JUnit XML reports
• Configurable linking rules via YAML configuration file
• Supports dry-run mode to preview links before creating them
• Provides clear reporting on matched and unmatched tests
• Non-interactive and suitable for automated CI/CD pipelines

Installation & getting started

The automation linking tool is distributed as an NPM package, similar to the Testmo CLI. You can install it globally or locally in your project.

Installation

$ npm install -g @testmo/testmo-link
$ testmo-link --help
Usage: testmo-link [options]

Options:
  --instance <url>                 Testmo instance URL (required)
  --project-id <id>                Project ID (required)
  --run-id <id>                    Automation run ID (required)
  --config <file>                  Configuration file path (required)
  --dry-run                        Preview links without creating them
  --verbose                        Enable detailed output
  --output <file>                  Write detailed output to file instead of stdout
  --conflict-resolution <mode>     How to handle conflicting case IDs (error, warn, silent)
  --fail-on-unmatched              Exit with error if any tests remain unmatched
  -h, --help                       Display help

Most systems and environments such as CI/CD containers usually already come with NPM installed. If your system is missing NPM/Node.js, just follow their guide on installing these tools.

Please note that you do not need to use Node.js/JavaScript yourself for your projects or as your automation programming language. We just use NPM to conveniently deploy the tool on any platform.

How automation linking works

Automation linking is a two-step process that works after you submit your test results:

Step 1: Submit test results (existing workflow)

First, you run your tests and submit the results to Testmo using the standard Testmo CLI, exactly as you do today:

$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
    --instance https://<your-name>.testmo.net \
    --project-id 1 \
    --name "Backend Tests" \
    --source "backend-unit" \
    --results results/*.xml

The Testmo CLI outputs the automation run ID as a plain string upon successful completion:

456

This is consistent with testmo automation:run:create command behavior. You'll need to capture this run ID for the next step.

Step 2: Link tests to cases (new step)

After submitting your results, you run the linking tool with the run ID:

$ testmo-link \
    --instance https://<your-name>.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml

The tool then:

1. Retrieves the test results from your automation run
2. Applies the linking rules from your configuration file
3. Creates links between automation tests and repository test cases
4. Reports a summary of linked and unmatched tests

That's it! Your automation results are now linked to your test cases and will appear in your test repository.

Complete workflow example

Here's a full example showing how to integrate automation linking into your existing workflow:

Before (existing workflow)

#!/bin/bash

# Run tests
npm test

# Submit results to Testmo
export TESTMO_TOKEN=********
testmo automation:run:submit \
    --instance https://example.testmo.net \
    --project-id 1 \
    --name "Build #$CI_BUILD_NUMBER" \
    --source "backend-unit" \
    --results results/*.xml

After (with automation linking)

#!/bin/bash

# Run tests
npm test

# Submit results to Testmo and capture run ID (plain string output)
export TESTMO_TOKEN=********
RUN_ID=$(testmo automation:run:submit \
    --instance https://example.testmo.net \
    --project-id 1 \
    --name "Build #$CI_BUILD_NUMBER" \
    --source "backend-unit" \
    --results results/*.xml)

# Link automation results to test cases
testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id $RUN_ID \
    --config testmo-linking.yml

echo "✅ Tests submitted and linked successfully"

Configuration file

The automation linking tool uses a YAML configuration file to define how test results should be linked to test cases. The configuration supports three different linking methods.

Basic configuration

Create a file called testmo-linking.yml in your project:

# Linking method precedence (which methods to try and in what order)
precedence:
  - annotation    # Try source code annotations first
  - config        # Then explicit mappings
  - pattern       # Finally pattern matching

# Method 1: Scan source code for annotations
annotations:
  enabled: true
  paths:
    - "test/**/*.py"
    - "test/**/*.java"
  patterns:
    - "@TestmoId:(\\d+)"

# Method 2: Explicit test-to-case mappings
config_mappings:
  - test_name: "testLoginWithUser"
    case_id: 42
  - test_name: "testCheckoutFlow"
    case_id: 100

# Method 3: Pattern matching rules
patterns:
  - rule: "test name matches case title (case-insensitive)"
    enabled: true

Linking methods explained

The automation linking tool supports three methods for linking tests to cases. You can use one, two, or all three methods together. The tool tries each method in the order specified by the precedence setting.

Requirements by Linking Method

Before choosing a linking method, verify your environment meets the requirements:

Legend:

• ✅ Required: Method won't work without this
• ⚠️ Conditionally needed: Required for certain scenarios
• ❌ Not needed: Method works without this
• ⬇️ Low risk: Explicit configuration prevents mis-linking
• ⬆️ Higher risk: Auto-matching can create wrong links if not validated

How to check if your framework includes file and line attributes:

Run your tests and inspect the JUnit XML output:

<!-- ✅ GOOD - Includes both file and line -->
<testcase name="test_login" classname="tests.auth" file="tests/auth.py" line="42" time="0.5"/>

<!-- ⚠️ PARTIAL - File only (annotations limited to single-test-per-file) -->
<testcase name="test_login" classname="tests.auth" file="tests/auth.py" time="0.5"/>

<!-- ❌ MISSING - No file attribute (annotations won't work reliably) -->
<testcase name="test_login" classname="tests.auth" time="0.5"/>

Common frameworks:

• Include file + line: pytest (default), JUnit 5 (with configuration), Jest (with configuration), NUnit, xUnit
• Include file only: Some older configurations
• May lack file: Some custom reporters, minimal JUnit implementations

Method 1: Annotations

Link tests to cases by adding special comments in your test source code.

Language Support: Annotation scanning is language-agnostic and works with any programming language that uses text-based comments or decorators. The tool performs regex pattern matching on source files, so it supports Python, Java, JavaScript, TypeScript, C#, Ruby, Go, Rust, PHP, and any other language where you can add text annotations.

Example in Python:

def test_login_with_user():
    """
    Test user login with valid credentials
    @TestmoId:42
    """
    # Test implementation
    pass

Example in Java:

/**
 * Test user login with valid credentials
 * @TestmoId:42
 */
@Test
public void testLoginWithUser() {
    // Test implementation
}

Example in JavaScript/TypeScript:

/**
 * Test user login with valid credentials
 * @TestmoId:42
 */
test('login with user', () = {
    // Test implementation
});

Example in C#:

/// <summary>
/// Test user login with valid credentials
/// @TestmoId:42
/// </summary>
[Test]
public void TestLoginWithUser() {
    // Test implementation
}

Example in Go:

/// <summary>
/// Test user login with valid credentials
/// @TestmoId:42
/// </summary>
[Test]
public void TestLoginWithUser() {
    // Test implementation
}

Configuration:

annotations:
  enabled: true
  paths:
    - "test/**/*.py"       # Python tests
    - "test/**/*.java"     # Java tests
    - "src/**/*.test.ts"   # TypeScript tests
    - "src/**/*.test.js"   # JavaScript tests
    - "tests/**/*.cs"      # C# tests
    - "*_test.go"          # Go tests
    - "spec/**/*.rb"       # Ruby tests
    # Add any file patterns for your language
  patterns:
    - "@TestmoId:(\\d+)"       # Standard format (always supported)
    - "@TestCaseId:(\\d+)"     # Alternative format (optional, configure if needed)

Note: Additional annotation aliases like @TestCaseId or custom formats are supported when configured in the patterns list. The standard @TestmoId format is recommended for consistency.

Benefits:

• Test case ID lives right next to the test code
• Easy for developers to see which case a test implements
• Survives test name changes
• Self-documenting

Requirements:

• Test source code files must be present in the environment where testmo-link runs
• In CI/CD: Requires repository checkout (e.g., actions/checkout@v3 in GitHub Actions)
• Cannot be used if only test results (XML files) are available

Best Practices for Annotation Placement:

To ensure reliable annotation matching, follow these guidelines:

1. Place annotations immediately ABOVE test function definitions:

   # @TestmoId:42
   def test_login_with_user():
       pass

   ✅ Required - Annotation must be above (not inside or below) the test definition

2. Multiple tests per file fully supported:

   # File: test_auth.py
   
   # @TestmoId:42
   def test_login():
       pass
   
   # @TestmoId:43
   def test_logout():
       pass
   
   # @TestmoId:44
   def test_password_reset():
       pass

   ✅ Works reliably with line-based proximity matching

3. Avoid incorrect placement:

# ❌ WRONG - Annotation inside function
def test_login():
    # @TestmoId:42  ← Won't match (below test definition)
    pass

# ❌ WRONG - Annotation after function
def test_login():
    pass
# @TestmoId:42  ← Won't match (below test definition)

How annotation matching works:

• The tool scans files for @TestmoId:(\d+) patterns using text-based regex
• Test names and line numbers come from JUnit XML (via API), not from parsing function definitions
• Primary approach (when line numbers available): For each test, the tool finds the annotation with the closest line number above the test definition
• Fallback (when line numbers unavailable):
  • File with exactly one test + one annotation → Automatic match
  • File with multiple tests/annotations → Error (use explicit config mappings)

Requirements for annotations:

• ✅ JUnit XML must include file attribute - Most frameworks support this
• ✅ JUnit XML should include line attribute - Required for multi-test files (pytest, JUnit, Jest, NUnit, etc.)
• ⚠️ If line attribute missing: Only single-test-per-file will work; multi-test files require explicit config mappings

Use this method when:

• Your team has access to test source code
• CI/CD workspace includes the full repository checkout
• You want to maintain links directly in the code
• You can structure tests with clear annotation placement

Method 2: Explicit Mappings

Link tests to cases using a configuration file with explicit test name to case ID mappings.

Configuration:

config_mappings:
  # Authentication tests
  - test_name: "testLoginWithUser"
    case_id: 42
  - test_name: "testLoginWithout2FA"
    case_id: 43

  # Checkout tests
  - test_name: "testAddToCart"
    case_id: 100
  - test_name: "testRemoveFromCart"
    case_id: 101

Benefits:

• No source code access required
• Centralized configuration
• Easy to bulk-edit mappings
• Works with any test framework or language
• Can be maintained by QA team

Requirements:

• Only requires the configuration file (no source code access needed)
• Works in any CI/CD environment, even with only test results available

Use this method when:

• You don't have access to test source code in your CI/CD pipeline
• Test results are generated in isolated containers without source
• You prefer centralized configuration management
• Multiple teams need to maintain mappings without touching test code

Method 3: Pattern Matching

Automatically link tests to cases based on naming conventions and rules.

Configuration:

patterns:
  - rule: "test name matches case title (case-insensitive)"
    enabled: true
  - rule: "test file path contains case key"
    enabled: false

Example matching:

• Test name: testLoginWithUser
• Case title: "Test Login With User"
• Result: Automatically linked ✅

Benefits:

• Zero manual configuration if naming is consistent
• Automatically works for all tests
• Scales to large test suites
• No maintenance overhead

Requirements:

• Only requires test results and Testmo case repository access (no source code needed)
• Naming conventions must be consistent between tests and cases

Use this method when:

• Your team follows consistent naming conventions between test names and test case titles
• You want zero-configuration automated linking
• Test naming standards are well-established and enforced

⚠️ Safety considerations:

Pattern matching can create incorrect links if naming is ambiguous. To use this method safely:

1. Always run with --dry-run first to review matches before creating actual links:

   testmo-link link --run-id 456 --config testmo-link.yml --dry-run

2. Watch for ambiguous matches:

   ⚠️ Warning: Multiple cases match test "test_login":
     - Case 42: "Test Login"
     - Case 100: "Test Login With 2FA"
     - Case 150: "Test Login With SSO"
   
   Result: Test remains UNMATCHED (ambiguous)

   By default, the tool treats ambiguous matches as unmatched (safe behavior). This prevents wrong links.

3. Common scenarios that cause wrong links:
   • Duplicate case titles: "Test Login" appears in multiple test suites
   • Similar names: "Login Test", "Test Login", "User Login Test" all normalize similarly
   • Legacy tests: Old tests may match unintended cases

4. Recommended configuration for safety:

   patterns:
     - rule: "test name matches case title (case-insensitive)"
       enabled: true
   
   settings:
     # Treat ambiguous pattern matches as unmatched (default: true)
     fail_on_ambiguous_pattern_match: true
   
     # Run in dry-run mode to verify all matches first
     dry_run: true  # Remove after verification
   

5. Best practice workflow:

   # Step 1: Preview matches
   testmo-link link --run-id 456 --config testmo-link.yml --dry-run
   
   # Step 2: Review output for wrong/ambiguous matches
   # Look for warnings about multiple matches
   
   # Step 3: Fix ambiguous cases with explicit mappings
   # Add to config:
   config_mappings:
     - test_name: "test_login"
       case_id: 42  # Explicit mapping removes ambiguity
   
   # Step 4: Run for real
   testmo-link link --run-id 456 --config testmo-link.yml

Risk level: ⬆️ Higher risk of incorrect links compared to annotations or explicit mappings. Use pattern matching as a supplement to explicit methods, not as the primary linking strategy for critical tests.

Using multiple methods together

You can combine all three methods for maximum flexibility:

precedence:
  - annotation    # Try annotations first (most explicit)
  - config        # Then explicit mappings (for special cases)
  - pattern       # Finally automatic matching (for the rest)

annotations:
  enabled: true
  paths:
    - "test/**/*.py"
  patterns:
    - "@TestmoId:(\\d+)"

config_mappings:
  # Only define exceptions or special cases here
  - test_name: "testLegacyAuthFlow"
    case_id: 999

patterns:
  - rule: "test name matches case title (case-insensitive)"
    enabled: true

How it works: For each test, the tool tries methods in order until a match is found. This means:

• Tests with @TestmoId annotations are linked first (highest priority)
• Tests in the config mappings are linked next
• Remaining tests are automatically matched by pattern
• Unmatched tests are reported for review

Handling conflicts

When a test matches multiple linking methods with different case IDs, the tool detects this as a conflict. You can configure how conflicts are handled:

settings:
  conflict_resolution: error  # Options: error (default), warn, silent

precedence:
  - annotation
  - config
  - pattern

Conflict resolution modes:

1. error (default, recommended): Stops execution and reports all conflicts
   • Best for catching configuration mistakes early
   • Ensures explicit intent

   • Example error:

     ❌ Error: Conflicting case IDs for test "test_login"
       - Annotation (@TestmoId): case 42
       - Config mapping: case 99
     
     Please resolve by ensuring both methods reference the same case ID,
     or remove one of the conflicting methods.

2. warn: Logs warnings but continues execution using precedence order
   • Good for gradual migration scenarios
   • Conflicts are visible but don't block CI/CD

   • Example warning:

     ⚠️  Warning: Conflicting case IDs for test "test_login"
       - Using annotation: case 42 (from @TestmoId)
       - Ignoring config: case 99

3. silent: Uses precedence order without warnings
   • Not recommended for production use
   • Only use if you intentionally have overlapping methods
   • Conflicts can be reviewed manually in Testmo by removing incorrect automation links

Example conflict scenario:

# test_login.py
def test_login():
    """
    @TestmoId:42  # Points to case 42
    """
    pass

# testmo-linking.yml
config_mappings:
  - test_name: "test_login"
    case_id: 99  # Points to case 99 - CONFLICT!

With conflict_resolution: error, the tool will report this mismatch and fail. With conflict_resolution: warn, it will use case 42 (annotation has higher precedence) and warn you about case 99 being ignored.

Overriding settings with command-line flags

All behavioral settings that are available in the YAML configuration file can also be specified as command-line flags. Command-line flags always override YAML settings, allowing you to change behavior without modifying the configuration file.

Available overrides:

Example: Testing different conflict modes

Your config file has conflict_resolution: error, but you want to test warn mode first:

# Use warn mode temporarily without changing the config file
$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml \
    --conflict-resolution warn

Example: Strict CI/CD validation

Your config allows unmatched tests, but you want production CI to fail if any tests are unmatched:

# Production pipeline enforces complete coverage
$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml \
    --fail-on-unmatched

Example: Development vs. production settings

# Development: Preview with warnings
$ testmo-link --config testmo-linking.yml --dry-run --verbose --conflict-resolution warn

# Production: Strict mode with failures on issues
$ testmo-link --config testmo-linking.yml --conflict-resolution error --fail-on-unmatched

Why this is useful:

• Test different modes without editing the config file
• Different settings for different environments (dev vs. prod)
• Quick overrides for debugging or experimentation
• Team-wide config with per-developer overrides

Writing output to a file

When processing large test suites with verbose logging enabled, the command-line output can overwhelm CI/CD logs. Use the --output flag to write detailed execution information to a file while displaying only a summary to stdout.

When to use file output:

• Processing hundreds or thousands of tests with --verbose enabled
• CI/CD environments where log storage has size limits
• Debugging complex linking scenarios without cluttering CI logs
• Generating detailed reports for team review or auditing

Example: Save detailed output to file

# Write all verbose output to file, display only summary to stdout
$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml \
    --verbose \
    --output testmo-linking-report.log

What gets written to the file:

• Progress updates for each test processed
• Detailed matching information (which method matched each test)
• All warnings and errors with full context
• Complete conflict reports (if any)
• API request/response details
• Performance metrics and timing information

What gets displayed to stdout:

• Final summary statistics (tests processed, linked, failed, unmatched)
• Critical errors that prevent execution
• Exit status message

Example: Use in CI/CD pipeline

# GitHub Actions example
- name: Link automation tests to cases
  run: |
    testmo-link \
      --instance ${{ secrets.TESTMO_URL }} \
      --project-id 1 \
      --run-id ${{ steps.testmo-submit.outputs.run_id }} \
      --config .testmo/linking.yml \
      --verbose \
      --output testmo-linking-report.log

- name: Upload linking report
  if: always()
  uses: actions/upload-artifact@v3
  with:
    name: testmo-linking-report
    path: testmo-linking-report.log

File output with YAML configuration:

You can also specify the output file in your configuration:

settings:
  verbose: true
  output_file: logs/testmo-linking.log  # Relative or absolute path

The CLI flag --output overrides the YAML setting, allowing you to change the output location per-run without editing the config file.

Parameterized / data-driven tests

Overview

Parameterized tests (also called data-driven tests) are tests that run multiple times with different input values. Most modern testing frameworks support this pattern, where one test function executes multiple times with varying parameters, producing multiple test results.

Common names across frameworks:

• pytest: @pytest.mark.parametrize
• JUnit 5: @ParameterizedTest with @ValueSource, @MethodSource, etc.
• TestNG: @DataProvider
• NUnit: [TestCase] or [TestCaseSource]
• xUnit: [Theory] with [InlineData]
• Jest/Mocha: test.each() or it.each()

What's Supported

✅ Supported: Linking all parameterized test instances to the same test case

❌ Not Supported (yet): Linking each parameterized instance to a different test case

The approach matches the most common use case: one logical test with multiple data points should link to one test case in Testmo.

How Parameterized Tests Appear in JUnit XML

When you run parameterized tests, your test framework generates multiple <testcase> entries in the JUnit XML output, one for each parameter combination. The framework typically appends parameter identifiers to the test name to distinguish between executions.

Example: pytest with @pytest.mark.parametrize

# @TestmoId:42
@pytest.mark.parametrize("username,password", [
    ("user1", "pass1"),
    ("user2", "pass2"),
    ("admin", "admin123"),
])
def test_login(username, password):
    """Test login with various user credentials"""
    assert login(username, password) == True

Resulting JUnit XML:

<testsuite name="authentication_tests">
  <testcase name="test_login[user1-pass1]" classname="tests.auth" file="tests/auth.py" line="15" time="0.123"/>
  <testcase name="test_login[user2-pass2]" classname="tests.auth" file="tests/auth.py" line="15" time="0.098"/>
  <testcase name="test_login[admin-admin123]" classname="tests.auth" file="tests/auth.py" line="15" time="0.105"/>
</testsuite>

Notice:

• Three separate <testcase> entries (one per parameter combination)
• Test names include parameter suffixes: [user1-pass1], [user2-pass2], [admin-admin123]
• All instances point to the same file and line number (line 15)
• One @TestmoId:42 annotation in source code

Method 1: Annotations (Recommended)

How it works: When using annotations with parameterized tests, the linking tool automatically links all parameterized instances to the same test case.

Configuration:

annotations:
  enabled: true
  paths:
    - "tests/**/*.py"
  patterns:
    - "@TestmoId:(\\d+)"

  # Optional: Enable parameterized test name matching
  parameterized_test_matching:
    enabled: true  # Default: true
    strip_parameters: true  # Remove parameter suffixes when matching

Matching algorithm:

1. Tool scans source file and finds @TestmoId:42 at line 14
2. Retrieves test results from API:
   • test_login[user1-pass1] at line 15 → Strip parameters → test_login
   • test_login[user2-pass2] at line 15 → Strip parameters → test_login
   • test_login[admin-admin123] at line 15 → Strip parameters → test_login
3. Line-based proximity matching: All three tests on line 15 → Annotation on line 14 (closest above)
4. Result: All three parameterized instances link to Case ID 42

Before/After in Testmo:

Before linking:

Automation Run #456
├─ test_login[user1-pass1] ✅ PASSED (0.123s) - Not linked
├─ test_login[user2-pass2] ✅ PASSED (0.098s) - Not linked
└─ test_login[admin-admin123] ✅ PASSED (0.105s) - Not linked

After linking:

Test Case #42: "Test login with various credentials"
├─ Latest Result: ✅ PASSED (from run #456)
├─ Linked Automation:
   ├─ test_login[user1-pass1] ✅ PASSED (0.123s)
   ├─ test_login[user2-pass2] ✅ PASSED (0.098s)
   └─ test_login[admin-admin123] ✅ PASSED (0.105s)

When any instance fails:

Test Case #42: "Test login with various credentials"
├─ Latest Result: ❌ FAILED (from run #456)
├─ Linked Automation:
   ├─ test_login[user1-pass1] ✅ PASSED (0.123s)
   ├─ test_login[user2-pass2] ❌ FAILED (0.098s) ← Failure shown
   └─ test_login[admin-admin123] ✅ PASSED (0.105s)

Method 2: Explicit Mappings (Fine-Grained Control)

If you need precise control over which parameterized instances link to which cases, use explicit mappings with full test names including parameters.

Use case 1: Link all instances to one case (same as annotations)

config_mappings:
  - test_name_pattern: "test_login\\[.*\\]"  # Regex matching all parameter variants
    case_id: 42

Use case 2: Link specific instances to different cases (not common, but supported)

config_mappings:
  - test_name: "test_login[user1-pass1]"
    case_id: 42  # Regular user test case
  - test_name: "test_login[user2-pass2]"
    case_id: 42  # Same case
  - test_name: "test_login[admin-admin123]"
    case_id: 50  # Admin-specific test case

Use case 3: Link parameterized test base name (matches all variants)

config_mappings:
  - test_name: "test_login"  # Matches "test_login" and all "test_login[...]" variants
    case_id: 42
    match_prefix: true  # Enable prefix matching for parameterized tests

Method 3: Pattern Matching (Automatic Discovery)

Pattern matching works with parameterized tests by stripping parameter suffixes before matching test names to case titles.

Configuration:

config_mappings:
  - test_name: "test_login"  # Matches "test_login" and all "test_login[...]" variants
    case_id: 42
    match_prefix: true  # Enable prefix matching for parameterized tests

Example:

• Test name: test_login[user1-pass1]
• Normalized: test_login → Test Login
• Case title in Testmo: "Test Login"
• Result: Match found → All test_login[...] variants link to the same case

Framework-Specific Examples

Java (JUnit 5)

// @TestmoId:42
@ParameterizedTest
@ValueSource(strings = {"user1", "user2", "admin"})
void testLogin(String username) {
    assertTrue(authenticate(username));
}

JUnit XML output:

<testcase name="testLogin(String)[1]" .../>
<testcase name="testLogin(String)[2]" .../>
<testcase name="testLogin(String)[3]" .../>

JavaScript (Jest)

// @TestmoId:42
test.each([
  ['user1', 'pass1'],
  ['user2', 'pass2'],
  ['admin', 'admin123'],
])('login with %s', (username, password) => {
  expect(login(username, password)).toBe(true);
});

JUnit XML output:

<testcase name="login with user1" .../>
<testcase name="login with user2" .../>
<testcase name="login with admin" .../>

C# (NUnit)

// @TestmoId:42
[TestCase("user1", "pass1")]
[TestCase("user2", "pass2")]
[TestCase("admin", "admin123")]
public void TestLogin(string username, string password)
{
    Assert.IsTrue(Authenticate(username, password));
}

JUnit XML output:

<testcase name="TestLogin(&quot;user1&quot;,&quot;pass1&quot;)" .../>
<testcase name="TestLogin(&quot;user2&quot;,&quot;pass2&quot;)" .../>
<testcase name="TestLogin(&quot;admin&quot;,&quot;admin123&quot;)" .../>

All examples above: One annotation → All instances link to Case ID 42 ✅

Recommended Approach

For most teams:

1. Use annotations with parameterized tests
2. Place @TestmoId annotation above the parameterized test function
3. All parameter variants automatically link to the same test case
4. This matches the typical use case: "One logical test with multiple data points = One test case"

Example:

# @TestmoId:123
@pytest.mark.parametrize("input,expected", [
    (1, 2),
    (2, 4),
    (3, 6),
])
def test_multiply_by_two(input, expected):
    assert multiply(input, 2) == expected

Result: All three test executions link to Case #123 in Testmo.

Handling Test Case Status with Parameterized Tests

Question: If one parameterized instance fails, what status does the test case show?

Answer: Testmo's automation linking shows:

• Overall status: The "worst" result (failed > passed > skipped)
• Individual results: All parameterized instances remain visible
• Execution details: You can drill down to see which specific parameter combination failed

Example:

Test Case #42: "Login with various users"
├─ Status: ❌ FAILED (1 of 3 failed)
├─ test_login[user1-pass1]: ✅ Passed
├─ test_login[user2-pass2]: ❌ Failed ← This caused overall failure
└─ test_login[admin-admin123]: ✅ Passed

Troubleshooting Parameterized Tests

Issue 1: Parameterized tests not matching annotations

Symptom:

Warning: Found annotation @TestmoId:42 but no matching test
Test: test_login[user1-pass1] - No annotation found
Test: test_login[user2-pass2] - No annotation found

Solution: Enable parameterized test name stripping:

annotations:
  parameterized_test_matching:
    strip_parameters: true  # Strip [parameter] suffixes when matching

Issue 2: Partial matches (some instances link, others don't)

Symptom:

Linked: test_login[user1-pass1] → Case 42 ✅
Linked: test_login[user2-pass2] → Case 42 ✅
Unmatched: test_login_special[admin] ❌

Cause: Test name changed for one variant (e.g., test_login_special vs test_login)

Solution: Add explicit mapping for the special case:

config_mappings:
  - test_name: "test_login_special[admin]"
    case_id: 42

Issue 3: Wrong test case status (expected failed, shows passed)

Symptom: One parameterized instance failed, but test case shows "Passed"

Cause: Framework may report individual instances separately; check Testmo run details

Solution: Verify in Testmo automation run that all instances are linked correctly. Status aggregation happens at the test case level.

Best Practices

1. One test case per logical test: Use parameterization for data variations of the same test, not for testing different functionality
2. Document parameters in test case: Add test case description explaining what parameter combinations are tested
3. Use descriptive parameter names: [valid_user], [invalid_password] instead of [test1], [test2]
4. Annotations for new tests: Prefer annotations for new parameterized tests (easier to maintain)
5. Explicit mappings for migration: Use explicit mappings when migrating parameterized tests from other tools

Migration Scenarios

Moving from Another Test Management Tool

If you're migrating from another test management tool to Testmo, automation linking makes it easy to reconnect your existing test automation to your newly imported test cases.

The Challenge

When you migrate test cases from another tool:

• Your automation tests still have the same names/identifiers
• But your test cases have new Testmo case IDs
• You need a way to map: old test name → new Testmo case ID

The Solution: Explicit Mappings

Use the config_mappings section to create an explicit mapping file:

settings:
  precedence:
    - config_mappings  # Prioritize explicit mappings during migration
    - annotations
    - pattern_matching

config_mappings:
  - test_name: "testLoginWithUser"
    case_id: 42
  - test_name: "testLogoutFlow"
    case_id: 43
  - test_name: "testCheckoutFlow"
    case_id: 100
  # ... hundreds or thousands of mappings

Migration Workflow

1. Export from previous tool - Get list of test case IDs and names
2. Import to Testmo - Test cases get new Testmo case IDs
3. Create mapping CSV - Spreadsheet with: test_name, old_id, new_testmo_case_id
4. Convert to YAML - Transform CSV into testmo-linking.yml config file
5. Run linking - Execute testmo-link to create all links

Large Migration Support

The testmo-link tool efficiently handles large migrations:

• 10,000+ mappings supported
• Fast processing: 5,000 mappings in < 5 seconds
• Memory efficient: Streaming YAML parser
• Progress reporting: Use --verbose to track progress

Example with large file:

# testmo-linking.yml with 5,000 mappings (500KB file)
testmo-link run \
  --instance https://your-instance.testmo.net \
  --run-id 123 \
  --config testmo-linking.yml \
  --verbose

# Output:
# Loaded 5,000 config mappings from testmo-linking.yml
# Processing 3,842 automation tests...
# Matched 3,756 tests via config mappings (97.8%)
# Created 3,756 links to test cases
# Completed in 4.2 seconds

Gradual Migration Path

You don't need to migrate all at once:

1. Phase 1: Use config_mappings for all existing tests
2. Phase 2: Add @TestmoId annotations to new tests
3. Phase 3: Gradually refactor old tests to use annotations
4. Phase 4: Remove mappings as tests are updated

The precedence system handles this gracefully:

• Annotations take precedence over config (new tests)
• Config mappings take precedence over patterns (legacy tests)
• Everything just works during the transition

Preview mode (dry-run)

Before creating actual links, you can preview what the tool will do using dry-run mode:

$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml \
    --dry-run

Example output:

✓ Retrieved 500 tests from automation run 456
✓ Loaded configuration from testmo-linking.yml

Applying linking rules...

📝 Preview of links to be created:

  Via annotations:
    ✓ testLoginWithUser → Case 42 (Test Login With User)
    ✓ testPasswordReset → Case 44 (Test Password Reset)
    ... (120 tests)

  Via config mappings:
    ✓ testLegacyAuthFlow → Case 999 (Legacy Authentication Flow)
    ... (15 tests)

  Via pattern matching:
    ✓ testCheckoutFlow → Case 100 (Test Checkout Flow)
    ✓ testAddToCart → Case 101 (Test Add To Cart)
    ... (350 tests)

  Unmatched:
    ⚠ testNewFeatureX (no annotation, not in config, no pattern match)
    ⚠ testExperimentalY (no annotation, not in config, no pattern match)
    ... (15 tests)

Summary:
  ✅ Would link: 485 tests
  ⚠️  Unmatched: 15 tests

[DRY RUN] No links were created. Remove --dry-run to create links.

This is useful for:

• Testing your configuration before going live
• Identifying unmatched tests that need attention
• Verifying pattern matching is working correctly

CI/CD integration examples

GitHub Actions

name: Test and Link

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run tests
        run: npm test

      - name: Submit results to Testmo
        id: testmo-submit
        run: |
          RUN_ID=$(testmo automation:run:submit \
            --instance ${{ secrets.TESTMO_URL }} \
            --project-id 1 \
            --name "Build ${{ github.run_number }}" \
            --source "backend-unit" \
            --results "results/*.xml")
          echo "run_id=$RUN_ID" >> $GITHUB_OUTPUT
        env:
          TESTMO_TOKEN: ${{ secrets.TESTMO_TOKEN }}

      - name: Link automation results
        run: |
          testmo-link \
            --instance ${{ secrets.TESTMO_URL }} \
            --project-id 1 \
            --run-id ${{ steps.testmo-submit.outputs.run_id }} \
            --config .testmo/linking.yml
        env:
          TESTMO_TOKEN: ${{ secrets.TESTMO_TOKEN }}

GitLab CI

test-and-link:
  stage: test
  script:
    # Run tests
    - npm test

    # Submit to Testmo
    - >
      RUN_ID=$(testmo automation:run:submit
      --instance $TESTMO_URL
      --project-id 1
      --name "Build $CI_PIPELINE_ID"
      --source "backend-unit"
      --results "results/*.xml")

    # Link results
    - >
      testmo-link
      --instance $TESTMO_URL
      --project-id 1
      --run-id $RUN_ID
      --config testmo-linking.yml

Jenkins Pipeline

pipeline {
    agent any

    environment {
        TESTMO_TOKEN = credentials('testmo-token')
        TESTMO_URL = 'https://example.testmo.net'
    }

    stages {
        stage('Test') {
            steps {
                sh 'npm test'
            }
        }

        stage('Submit to Testmo') {
            steps {
                script {
                    def runId = sh(
                        script: """
                            testmo automation:run:submit \
                                --instance ${TESTMO_URL} \
                                --project-id 1 \
                                --name "Build ${BUILD_NUMBER}" \
                                --source "backend-unit" \
                                --results "results/*.xml" \
                                | grep -oP '(?<="id":)\\d+'
                        """,
                        returnStdout: true
                    ).trim()

                    env.RUN_ID = runId
                }
            }
        }

        stage('Link Results') {
            steps {
                sh """
                    testmo-link \
                        --instance ${TESTMO_URL} \
                        --project-id 1 \
                        --run-id ${RUN_ID} \
                        --config testmo-linking.yml
                """
            }
        }
    }
}

Exit codes and CI/CD integration

The automation linking tool returns standard exit codes to work seamlessly with CI/CD pipelines:

• Exit code 0: Success - all tests processed, links created successfully
• Exit code 1: Error - authentication failed, configuration invalid, or API errors

By default, the tool exits with code 0 even if some tests remain unmatched (they're just reported in the summary). If you want your CI/CD pipeline to fail when tests cannot be linked, use the configuration option:

settings:
  fail_on_unmatched: true  # Exit with code 1 if any tests are unmatched

This is useful for enforcing 100% automation coverage requirements.

Viewing linked results in Testmo

After running the linking tool, your automation results will be visible in your test repository:

1. Repository View: Navigate to your test case repository in Testmo
2. Automation Column: You'll see an "Automation" column showing which test cases have automated tests
3. Latest Status: The "Status (latest)" column shows the most recent automation result for each case
4. Test History: Click on any test case to view its full automation execution history
5. Coverage Report: Use filters to identify test cases without automation coverage

Handling unmatched tests

When the linking tool cannot find a matching test case for a test result, it reports these as "unmatched" tests:

Example output:

Summary:
  ✅ Successfully linked: 487 tests
  ⚠️  Unmatched: 13 tests
  ❌ Errors: 0

Unmatched tests:
  - testNewFeatureX (no annotation, not in config)
  - testExperimentalY (no annotation, not in config)
  - test_legacy_auth_method (no matching case found)
  ...

Why tests might be unmatched

• No @TestmoId annotation in source code
• Not listed in config mappings
• Test name doesn't match any case title
• Test case doesn't exist in Testmo repository
• Test case is in a different project

How to handle unmatched tests

1. Add annotations - Add @TestmoId:123 to test source code
2. Update config - Add explicit mapping in config_mappings
3. Create test case - Create the corresponding test case in Testmo repository
4. Review naming - Align test names with case titles for pattern matching
5. Accept as unmatched - Some tests may intentionally not map to cases

Advanced configuration

Verbose output

Enable detailed logging to see exactly how the tool processes each test:

$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config testmo-linking.yml \
    --verbose

Custom configuration location

You can store your configuration file anywhere:

$ testmo-link \
    --instance https://example.testmo.net \
    --project-id 1 \
    --run-id 456 \
    --config .testmo/custom-linking-config.yml

Environment variables

Like the Testmo CLI, the linking tool reads authentication from the TESTMO_TOKEN environment variable:

$ export TESTMO_TOKEN=your-api-token-here
$ testmo-link --instance https://example.testmo.net --project-id 1 --run-id 456 --config testmo-linking.yml

Using HTTP proxies

If you need to connect through an HTTP proxy (common in corporate environments), set the standard HTTP_PROXY or HTTPS_PROXY environment variables:

# Set proxy for HTTP and HTTPS connections
$ export HTTP_PROXY=http://proxy.company.com:8080
$ export HTTPS_PROXY=http://proxy.company.com:8080

# Or with authentication
$ export HTTP_PROXY=http://username:password@proxy.company.com:8080
$ export HTTPS_PROXY=http://username:password@proxy.company.com:8080

# Then run the tool normally
$ testmo-link --instance https://example.testmo.net --project-id 1 --run-id 456 --config testmo-linking.yml

The tool uses an HTTP client library that respects these standard proxy environment variables, consistent with Testmo CLI behavior (which uses proxy-agent via the @testmo/testmo-api package).

Complete configuration reference

Here's a fully documented configuration file with all available options:

# Settings (optional)
# Note: All settings below can be overridden via command-line flags
settings:
  conflict_resolution: error  # How to handle conflicting case IDs
                              # Options: error (default), warn, silent
                              # error: Fail with detailed conflict report
                              # warn: Log warning, use precedence order
                              # silent: Use precedence order without warnings
                              # CLI override: --conflict-resolution <mode>
  output_file: null           # Write detailed output to file
                              # CLI override: --output <file>

# Linking method precedence (required)
# Defines which linking methods to use and in what order
precedence:
  - annotation    # Try annotations first
  - config        # Then explicit mappings
  - pattern       # Finally pattern matching

# Method 1: Annotation scanning (optional)
annotations:
  enabled: true                          # Enable/disable annotation scanning
  paths:                                 # Directories/files to scan
    - "test/**/*.py"                     # Python files
    - "test/**/*.java"                   # Java files
    - "test/**/*.js"                     # JavaScript files
    - "test/**/*.ts"                     # TypeScript files
  patterns:                              # Regex patterns to search for
    - "@TestmoId:(\\d+)"                 # Standard (recommended): @TestmoId:42
    - "@TestCaseId:(\\d+)"               # Alternative alias (optional): @TestCaseId:42
    - "TC-(\\d+)"                        # Custom short form (optional): TC-42

# Method 2: Explicit mappings (optional)
config_mappings:
  # Add as many mappings as needed
  - test_name: "testLoginWithUser"       # Exact test name
    case_id: 42                          # Testmo case ID
  - test_name: "testPasswordReset"
    case_id: 44
  - test_name: "testCheckoutFlow"
    case_id: 100

# Method 3: Pattern matching (optional)
patterns:
  - rule: "test name matches case title (case-insensitive)"
    enabled: true                        # Enable this rule
  - rule: "test file path contains case key"
    enabled: false                       # Disable this rule

# Optional settings (can be overridden via CLI flags)
settings:
  dry_run: false                         # true = preview only, false = create links
                                         # CLI override: --dry-run
  verbose: false                         # true = detailed logging
                                         # CLI override: --verbose
  output_file: null                      # Write detailed output to file (path)
                                         # CLI override: --output <file>
  fail_on_unmatched: false               # true = exit 1 if any unmatched tests
                                         # CLI override: --fail-on-unmatched
  batch_size: 100                        # Number of links per API PATCH request (max 100)
                                         # Note: Read pagination (GET) supports 100-1000 per page

Best practices

1. Start with dry-run mode

Always test your configuration with --dry-run first:

# Test configuration
testmo-link --run-id 456 --config testmo-linking.yml --dry-run

# Once verified, run for real
testmo-link --run-id 456 --config testmo-linking.yml

2. Use version control for configuration

Store your testmo-linking.yml file in version control alongside your test code:

project/
├── test/
├── testmo-linking.yml  ← Version controlled
└── .github/
    └── workflows/

3. Start with one method, add more later

Begin with the simplest method that works for your team:

For teams with source access:

precedence:
  - annotation

annotations:
  enabled: true
  paths: ["test/**/*.py"]
  patterns: ["@TestmoId:(\\d+)"]

For teams without source access:

precedence:
  - config

config_mappings:
  - test_name: "testLoginFlow"
    case_id: 42

4. Use pattern matching for scale

If you have consistent naming conventions, pattern matching requires zero maintenance:

precedence:
  - pattern

patterns:
  - rule: "test name matches case title (case-insensitive)"
    enabled: true

5. Monitor unmatched tests

Review unmatched tests regularly and decide whether to:

• Add missing test cases to Testmo
• Add annotations or config mappings
• Accept some tests as intentionally unlinked

Performance and limits

The automation linking tool is designed to handle large test suites efficiently:

• Retrieval: Processes 1,000 test results in under 2 seconds
• Linking: Creates 500 links in under 30 seconds
• Batch processing: Groups links into batches of 100 for efficient API calls
• Total time: Complete workflow for 1,000 tests takes less than 2 minutes

Troubleshooting

"No tests found for run ID"

Cause: The run ID doesn't exist or has no test results.

Solution: Verify the run ID is correct and that test results were successfully submitted.

"Configuration file not found"

Cause: The specified configuration file path is incorrect or missing.

Solution: Check the file path and ensure the file exists:

ls -la testmo-linking.yml

Note: The configuration file is required for the linking tool to work. If you need to link tests without a configuration file, you must create at least a minimal configuration specifying which linking methods to use.

"Authentication failed"

Cause: Invalid or missing TESTMO_TOKEN environment variable.

Solution: Verify your API token:

echo $TESTMO_TOKEN
# Should output your token (not empty)

"Invalid annotation pattern"

Cause: Regex pattern in configuration is malformed.

Solution: Test your regex pattern separately or use the standard pattern:

patterns:
  - "@TestmoId:(\\d+)"  # Note: double backslash in YAML

Many unmatched tests

Cause: Linking methods aren't finding matches.

Solutions:

1. Run with --verbose to see why tests aren't matching
2. Use --dry-run to preview without creating links
3. Add more linking methods to your configuration
4. Review test names and case titles for consistency

"Invalid case ID" or "Case not found" errors

Cause: Configuration references case IDs that don't exist in the Testmo project.

When validation occurs: Invalid case IDs are detected and rejected by the Testmo API during link creation (PATCH request). The tool will report which case IDs failed.

Solutions:

1. Verify case IDs exist in your Testmo project
2. Check for typos in annotations or config mappings
3. Ensure you're using the correct project ID
4. Run with --dry-run first to catch invalid IDs before creating links

Example error:

❌ Error creating links: API returned 400 Bad Request
  - Case ID 99999 not found in project
  - Case ID 12345 not found in project

Failed to create 2 of 50 links.

Annotations not matching (multi-test files)

Symptoms:

• Tool reports "File has multiple tests and annotations"
• Error message says "Use explicit config mappings"
• Annotations work in some files but not others

Cause: JUnit XML is missing line attributes, so the tool cannot use line-based proximity matching.

Check your JUnit XML: Inspect your test framework's XML output:

# Find your JUnit XML file
cat test-results.xml | grep testcase | head -5

# ✅ GOOD - Has line attribute:
# <testcase name="test_login" file="tests/auth.py" line="42" .../>

# ❌ BAD - Missing line attribute:
# <testcase name="test_login" file="tests/auth.py" .../>

Solutions:

1. Configure your test framework to output line numbers (recommended):

   pytest (default behavior - should work out of the box):

   pytest --junitxml=test-results.xml

   Jest (configure in jest.config.js):

   module.exports = {
     reporters: [
       'default',
       ['jest-junit', {
         outputDirectory: './test-results',
         outputName: 'junit.xml',
         addFileAttribute: 'true',  // Adds file and line
       }]
     ]
   };

   JUnit 5 (add to pom.xml or build.gradle):

   <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <configuration>
       <properties>
         <property>
           <name>junit.jupiter.execution.parallel.enabled</name>
           <value>false</value>
         </property>
       </properties>
     </configuration>
   </plugin>

2. OR: Use single test per file (fallback if line numbers unavailable):

   test_login_user.py  # One test, one annotation
   test_logout_user.py # One test, one annotation

3. OR: Use explicit config mappings for multi-test files:

   config_mappings:
     - test_name: "test_login"
       case_id: 42
     - test_name: "test_logout"
       case_id: 43

JUnit XML readiness check

Before relying on annotations, verify your test framework outputs the required metadata:

# 1. Run your tests
pytest --junitxml=test-results.xml  # or your test command

# 2. Check for file attribute (required)
grep 'file=' test-results.xml | wc -l
# Should be > 0 (shows count of tests with file attribute)

# 3. Check for line attribute (needed for multi-test files)
grep 'line=' test-results.xml | wc -l
# Should match test count if you want multi-test file support

# 4. Inspect a sample
grep '<testcase' test-results.xml | head -3

What you should see:

<!-- ✅ IDEAL: Both file and line -->
<testcase name="test_login" classname="tests.auth" file="tests/auth.py" line="42" time="0.5"/>

<!-- ⚠️ LIMITED: File only (annotations work for single-test-per-file only) -->
<testcase name="test_login" classname="tests.auth" file="tests/auth.py" time="0.5"/>

<!-- ❌ WON'T WORK: No file attribute (use explicit mappings or pattern matching instead) -->
<testcase name="test_login" classname="tests.auth" time="0.5"/>