When you search your Testmo repository through an MCP-connected assistant, it uses Testmo's Test Selection engine. You describe what you're testing in natural language and it returns a ranked, scored list of the most relevant test cases — blending meaning-based and keyword-based search so you get the best of both.
How it works: two searches, blended
Traditional search matches the exact words you type. Test Selection runs two complementary searches at the same time and merges them into a single ranking — an approach called hybrid search:
| Method | What it's good at | Everyday analogy |
|---|---|---|
| Semantic search | Meaning. It knows "login failure" and "authentication error" are related even when the words differ — so synonyms and paraphrases still match. | A librarian who understands concepts |
| Lexical (keyword) search | Exact terms — error codes (ERR_5012), API paths, ticket IDs (SIS-695). |
Ctrl-F / exact keyword match |
Neither is enough on its own: semantic search misses exact codes, and keyword search misses paraphrases. Running both and combining them gives the strongest result — the reason you'll sometimes see this called "semantic search" is that the meaning-based half is what makes it feel smart, but the keyword half is doing just as much work.
When to use it
- Coverage discovery — "What do we already test around subscription renewals?"
- Finding related cases before a change — surface everything relevant to the area you're about to modify.
- Exploring an unfamiliar project — get oriented in a large repository by asking about a topic rather than guessing folder names.
Writing a good prompt
Precise, well-structured prompts produce noticeably better results. A helpful way to think about it is the search bar on Amazon or Apple Music: searching "shoes" or "jazz" returns thousands of loosely related results, but "men's waterproof trail-running shoes, size 10" or "1960s modal jazz, saxophone-led" gets you what you actually want. The same is true here — the more specific your query, the more useful the ranking.
A strong prompt usually names the feature or area, the scenario you care about, and any specifics — platform, error codes, ticket IDs. Some examples:
- "Find test cases for checkout and payment flows for the Black Friday release — the priority is refunds and declined cards on web."
- "Find test cases for password reset and account recovery across Project Alpha and Project Beta." — searches more than one project in a single query.
- "Find test cases for login failures and authentication errors, including locked accounts and expired sessions."
-
"Find test cases covering the
ERR_5012payment-declined response on the orders API." — exact codes and IDs are picked up by the keyword half of the search.
Understanding the relevance score
Every returned case gets a relevance score from 0 to 100 — higher means more relevant — and results come back sorted best-first.
The score reflects how much the two searches agree:
- A case that ranks highly in both the meaning search and the keyword search scores near 100 — both methods strongly agree it's relevant.
- A case found by only one of the two searches scores around 50 — one method likes it, the other didn't surface it (often a wording mismatch).
- Weaker and single-method matches taper down from there.
In plain terms: the more the two searches agree that a case matters, the higher its score. Agreement is the signal — which is why a case that's #1 in one search but missing from the other (≈50) scores lower than a case sitting mid-pack in both.
How to read the score
| Score range | How to read it |
|---|---|
| 90–100 | Top ~10 in both searches. Very strong agreement — review these first. |
| 70–90 | Top ~30 in both, or top-ranked in one with a moderate rank in the other. |
| 50–70 | Moderate across both searches, or very strong in just one. |
| 30–50 | Weaker — low ranks in both, or a moderate rank in only one. |
| Below 30 | Low relevance — only marginally matched by either search. |
What the score does not mean
The 0–100 range invites a few natural but wrong readings — worth heading off before they cause confusion:
| It's tempting to assume… | Actually… |
|---|---|
| "A score of 85 means an 85% chance this test will find a bug." | No. The score says nothing about bug-finding probability. |
| "A score of 70 means the engine is 70% confident this is the right test." | No. It is not a confidence level. |
| "Anything above X is good, anything below is noise." | Not quite. The score is relative within a single query. Use it to order and prioritise, not as a universal pass mark. |
A few more things worth knowing:
- Scores compare within one query, not across queries. A 72 and a 65 in the same result set are meaningfully rankable against each other; a 72 in one query and a 72 in a different query are not. Don't build a fixed cross-query threshold and expect it to behave consistently.
- The score is stable across pages. A case shows the same score no matter which page it appears on.
- Nothing is hidden. The engine returns everything it found, including weak matches — it's up to you where to stop reading.
- Results are deterministic. The same query over the same data returns the same cases, with the same scores, in the same order.
Paging through results
A query can match hundreds or thousands of cases, so results are returned a page at a time — the top matches first, then "show me more." Paging is fast and stable: the ranking is worked out once and then held steady, so results don't shuffle, skip, or duplicate as you page through them, and later pages come back near-instantly even in very large repositories.
Security & privacy
How Your Data Is Used
To match by meaning, Test Selection processes the text of your test cases — titles, descriptions, and steps — through the Sembi IQ processing pipeline. Your data is treated as Customer Content: it is never shared with other customers and is not used to train underlying models, and all transfers are encrypted in transit. Run results, defects, user information, source code, and secrets are not processed.