Description of Test Reports and Metrics

Reports are related to a Test Plan (defined in top of navigator). This Test Plan could differs from the current Test Plan for execution. You can also select Report format:

The print button activate print of a report only (without navigation).

There are several seperate reports to choose from, their purpose and function are explained below.

General Test Plan Metrics

This page shows you only the most current status of a Test plan by test suite, owner, and keyword. The most "current status" is determined by the most recent build test cases were executed on. For instance, if a test case was executed over multiple builds, only the latest result is taken into account.

"Last Test Result" is a concept used in many reports, and is determined as follows:

The following tables are displayed:

The Overall Build Status

Lists the execution results for every build. For each build, the total test cases, total pass, % pass, total fail, % fail, blocked, % blocked, not run, %not run. If a test case has been executed twice on the same build, the most recent execution will be taken into account.

Query Metrics

This report consists of a query form page, and a query results page which contains the queried data. Query Form Page presents with a query page with 4 controls. Each control is set to a default which maximizes the number of test cases and builds the query should be performed against. Altering the controls allows the user to filter the results and generate specific reports for specific owner, keyword, suite, and build combinations.

Press the "submit" button to proceed with the query and display the output page.

Query Report Page will display:

  1. the query parameters used to create report
  2. totals for the entire test plan
  3. a per suite breakdown of totals (sum / pass / fail / blocked / not run) and all executions performed on that suite. If a test case has been executed more than once on multiple builds - all executions will be displayed that were recorded against the selected builds. However, the summary for that suite will only include the "Last Test Result" for the selected builds.

Blocked, Failed, and Not Run Test Case Reports

These reports show all of the currently blocked, failing, or not run test cases. "Last test Result" logic (which is described above under General Test Plan Metrics) is again employed to determine if a test case should be considered blocked, failed, or not run. Blocked and failed test case reports will display the associated bugs if the user is using an integrated bug tracking system.

Test Report

View status of every test case on every build. The most recent execution result will be used if a test case was executed multiple times on the same build. It is recommended to export this report to Excel format for easier browsing if a large data set is being used.

Charts - General Test Plan Metrics

"Last test Result" logic is used for all four charts that you will see. The graphs are animated to help the user visualize the metrics from the current test plan. The four charts provide are :

The bars in the bar charts are colored such that the user can identify the approximate number of pass, fail, blocked, and not run cases.

This report page requires your browser have a flash plugin (by http://www.maani.us) to display results in a graphical format.

Total Bugs For Each Test Case

This report shows each test case with all of the bugs filed against it for the entire project. This report is only available if a Bug Tracking System is connected.