Reliability of Tests in an Execution Plan

Problem

In a continuous integration environment tests are ideally executed at least once per day for testing the daily build and ensuring the quality of your application under test. To understand how reliable your test set is for measuring the quality of your AUT it is inevitable to sometimes have a look at how the results change over time. For example you might have tests in your test set that frequently change status, therefore being no real measure for quality.

Solution

Use the data mart view RV_TestStatusExtended to create a report that lists the results for a specific test in the context of a specific execution plan. This allows you to see how this test’s results have changed over time. For convenience, we will narrow the list of results down to those related to tagged builds, thus looking at specific milestone builds of the application under test only. This report collects test result data for tests in the context of execution plans and builds. In the following query we:

• Select the columns we want to display from this view.
• Narrow the result down by the ID of the test we want to investigate and the ID of the execution plan in which the test belongs.
• Add a constraint to consider tagged builds only.

SELECT TestName, ExecutionPlanName, VersionName, BuildName, TestRunID, 
  PassedCount, FailedCount, NotExecutedCount
FROM RV_TestStatusExtended
WHERE TestID = ${TESTID|1|Test ID} AND ExecutionPlanID = ${EXECUTIONPLANID|1|Execution Plan ID} AND BuildIsTagged = 1
ORDER BY BuildOrderNumber