Description of Test Reports and Metrics
Reports are related to a Test Plan (defined in top of navigator). This Test Plan could differs from the
current Test Plan for execution. You can also select Report format:
- Normal - report is displayed in web page
- MS Excel - report exported to Microsoft Excel
- HTML Email - report is emailed to user's email address
The print button activate print of a report only (without navigation).
There are several seperate reports to choose from, their purpose and function are explained below.
General Test Plan Metrics
This page shows you only the most current status of a Test plan by test suite, owner, and keyword.
The most "current status" is determined by the most recent build test cases were executed on. For
instance, if a test case was executed over multiple builds, only the latest result is taken into account.
"Last Test Result" is a concept used in many reports, and is determined as follows:
- The order in which builds are added to a Test Plan determines which build is most recent. The results
from the most recent build will take precendence over older builds. For example, if you mark a test as
"fail" in build 1, and mark it as "pass" in build 2, it's latest result will be "pass".
- If a test case is executed mulitple times on the same build, the most recent execution will take
precedence. For example, if build 3 is released to your team and tester 1 marks it as "pass" at 2PM,
and tester 2 marks it as "fail" at 3PM - it will appear as "fail".
- Test cases listed as "not run" against a build are not taken into account. For example, if you mark
a case as "pass" in build 1, and don't execute it in build 2, it's last result will be considered as
"pass".
The following tables are displayed:
- Results by top level Test Suites
Lists the results of each top level suite. Total cases, passed, failed, blocked, not run, and percent
completed are listed. A "completed" test case is one that has been marked pass, fail, or block.
Results for top level suites include all children suites.
- Results By Keyword
Lists all keywords that are assigned to cases in the current test plan, and the results associated
with them.
- Results by owner
Lists each owner that has test cases assigned to them in the current test plan. Test cases which
are not assigned are tallied under the "unassigned" heading.
The Overall Build Status
Lists the execution results for every build. For each build, the total test cases, total pass,
% pass, total fail, % fail, blocked, % blocked, not run, %not run. If a test case has been executed
twice on the same build, the most recent execution will be taken into account.
Query Metrics
This report consists of a query form page, and a query results page which contains the queried data.
Query Form Page presents with a query page with 4 controls. Each control is set to a default which
maximizes the number of test cases and builds the query should be performed against. Altering the controls
allows the user to filter the results and generate specific reports for specific owner, keyword, suite,
and build combinations.
- keyword 0->1 keywords can be selected. By default - no keyword is selected. If a keyword is not
selected, then all test cases will be considered regardless of keyword assignments. Keywords are assigned
in the test specification or Keyword Management pages. Keywords assigned to test cases span all test plans,
and span across all versions of a test case. If you are interested in the results for a specific keyword
you would alter this control.
- owner 0->1 owners can be selected. By default - no owner is selected. If an owner is not selected,
then all test cases will be considered regardless of owner assignment. Currently there is no functionality
to search for "unassigned" test cases. Ownership is assigned through the "Assign Test Case execution" page,
and is done on a per test plan basis. If you are interested in the work done by a specific tester you would
alter this control.
- top level suite 0->n top level suites can be selected. By default - all suites are selected.
Only suites that are selected will be queried for result metrics. If you are only intested in the results
for a specific suite you would alter this control.
- Builds 1->n builds can be selected. By default - all builds are selected. Only executions
performed on builds you select will be taken into account when producing metrics. For example - if you
wanted to see how many test cases were executed on the last 3 builds - you would alter this control.
Keyword, owner, and top level suite selections will dictate the number of test cases from your test plan
are used to computate per suite and per test plan metrics. For example, if you select owner = "Greg",
Keyword="Priority 1", and all available test suites - only Priority 1 test cases assigned to Greg will be
considered. The "# of Test Cases" totals you will see on the report will be influenced by these 3 controls.
Build selections will influence if a case is considered "pass", "fail", "blocked", or "not run". Please
refer to "Last Test Result" rules as they appear above.
Press the "submit" button to proceed with the query and display the output page.
Query Report Page will display:
- the query parameters used to create report
- totals for the entire test plan
- a per suite breakdown of totals (sum / pass / fail / blocked / not run) and all executions performed
on that suite. If a test case has been executed more than once on multiple builds - all executions will be
displayed that were recorded against the selected builds. However, the summary for that suite will only
include the "Last Test Result" for the selected builds.
Blocked, Failed, and Not Run Test Case Reports
These reports show all of the currently blocked, failing, or not run test cases. "Last test Result"
logic (which is described above under General Test Plan Metrics) is again employed to determine if
a test case should be considered blocked, failed, or not run. Blocked and failed test case reports will
display the associated bugs if the user is using an integrated bug tracking system.
Test Report
View status of every test case on every build. The most recent execution result will be used
if a test case was executed multiple times on the same build. It is recommended to export this report
to Excel format for easier browsing if a large data set is being used.
Charts - General Test Plan Metrics
"Last test Result" logic is used for all four charts that you will see. The graphs are animated to help
the user visualize the metrics from the current test plan. The four charts provide are :
- Pie chart of overall pass / fail / blocked / and not run test cases
- Bar chart of Results by Keyword
- Bar chart of Results By Owner
- Bar chart of Results By Top Level Suite
The bars in the bar charts are colored such that the user can identify the approximate number of
pass, fail, blocked, and not run cases.
This report page requires your browser have a flash plugin (by http://www.maani.us) to display
results in a graphical format.
Total Bugs For Each Test Case
This report shows each test case with all of the bugs filed against it for the entire project.
This report is only available if a Bug Tracking System is connected.