Testmo will generate forecasts for test runs and milestones based on estimates and other metrics gathered during the course of test executions. You can find the forecasts on the Test Run > Status view in the sidebar Progress section, and in the Progress section of the Milestone > Status view for your projects.
Test Run Forecasts
The Progress section containing your Run forecasts can be found in the Status area of your Test Run and is made up of 3 metrics:
- Elapsed: The total elapsed time across all of the tests in the run which have been completed so far.
- Forecast: The total forecasted time Testmo expects all your tests will take to be completed.
- Done: The total time it should have taken to execute the complete tests based on the test forecasts.
Milestone Forecasts
The Progress section, containing your Milestone forecasts for all in-scope runs and sessions can be found in the Status area of your Milestone (or Sub-Milestones) and is made up of 4 metrics:
- Ideal forecast: The ideal burndown for the remaining tests in the milestone based on the test estimates and actual durations so far.
- Completed todos: A daily count of the completed todos, in the scope of the milestone, displayed as bars.
- Remaining todos: A daily count of the remaining todos, in the scope of the milestone, displayed as a line graph.
- Remaining effort: The remaining effort associated with the incomplete tests in the scope of the milestone, displayed as a hour units along a line graph.
The Progress section also displays the Due date for the milestone and the forecasted completion date based on the calculations above, so you can see how on track you are with your target completion.
Forecast Generation
Sometimes, instead of the expected forecasts, you’ll see “None” instead. This may be the result of several factors:
Testmo doesn’t have enough information to work with. Testmo needs most (ideally all) of the test cases to have estimates in order to generate Forecast and Done metrics. Testmo also needs users to record the amount of time taken to complete the tests, in order to update the Elapsed metric. Without this information, Testmo can’t carry out the necessary forecasting calculations reliably.
Testmo hasn’t run the calculation yet. Testmo generates forecasting calculations periodically throughout the day. If you’ve added the necessary information, recorded some timings for your test cases but still aren’t seeing any updates in the Test Run or Milestone Progress sections, it’s probably because the calculation hasn’t been carried out yet.
Whenever a test run is created or modified, Testmo identifies the run as having been updated, and therefore within the scope of its forecasting calculations. Since various different actions may affect the forecasts, including estimations, assignment and velocity, and the calculations are somewhat complex as a result, Testmo carries out the forecasting algorithms periodically in the background.
This means that you won’t usually see an update as a result of a specific action or datapoint. It also helps ensure the best user experience by optimizing for Testmo performance.
Common Questions
We’re often asked questions about how forecasting works. The guide above provides an overview of the functionality and expected usage. We’ve included some additional FAQ’s below to help clarify a few further points:
Q. What events trigger the forecast calculation? (What are the necessary conditions for that calculation to be made?)
A. The necessary conditions for an update are that a run has been created or updated. This doesn’t necessarily mean that the calculation will be triggered immediately however. As mentioned above, the forecasting calculations are executed periodically throughout the day.
Q. What is the frequency of the forecasting calculation updates?
A. Approximately every 6 hours. Timings may change however depending on your specific instance, the size and scope of the calculation and other factors.
Q. Why don’t the Forecast and Done progress statistics update instantly as the Elapsed statistic does?
A. Elapsed is a relatively simple metric to calculate (being the sum of all the elapsed timings for the tests in the scope of the run or milestone Elapsed Progress metric). Forecast & Done statistics require additional processing and are therefore carried out on a schedule, as described above.
Q. What are the actual calculations made in the backend to get the values for the Run & Milestone forecasts?
A. The forecast of a run is mainly based on the forecasts of the test cases included in a run. Testmo falls back to the test case estimates or average forecast/estimate across the case selection in the event that a test case does not have a forecast. For the completion stats of a run, Testmo looks at the percentage of completed tests and compares that to the forecast.
The forecast (and burndown chart) of a milestone is based on the burndown statistics of the runs & sessions in that milestone (and potentially sub-milestones). Testmo looks at the remaining effort & amount of tests across the runs/sessions and tries to compute a good forecast for the milestone based on those and the velocity (how fast tests are being completed). In practice, the computation is relatively complex because Testmo differentiates between effort (time-based) and to-dos (count-based) and can also use different start dates to compute the velocity (depending on the milestone settings).
Q. What are the minimum requirements for Milestone forecast calculation?
A. Testmo does not show a forecast if the number of completed tests is smaller than a certain threshold. The reason for this is that the velocity (and therefore forecast) gets more accurate the more tests you have completed, and it therefore doesn't make sense to show a forecast below a certain limit. In general, the forecast is more useful and more accurate for longer running milestones compared to shorter milestones; ideally in over a week or more rather than a few days