Reliability Testing

Definition

Reliability testing is the system testing of an integrated, blackbox application against its reliability requirements.

Goals

The typical goals of reliability testing are to:

Determine if the application meets its reliability requirements.

Objectives

The typical objectives of reliability testing are to:

Attempt to cause the application to fail under normal conditions a sufficient number of times so that the reliability of the application can be estimated.
Partially validate the application (i.e., to determine if it fulfills its reliability requirements).
Determine how often the application fails.
Report these failures to the development teams so that the associated defects can be fixed.
Help determine the extent to which the application is ready for launch.
Provide input to the defect trend analysis effort.

Examples

Typical examples include reliability testing of an application that is:

A system including software, hardware (that may fail), and data components.
Embeded within another system (e.g., flight-control software, cruise-control software).
Software only.
Client/server or n-tier distributed.
Business-critical or safety-critical.

Preconditions

Reliability testing can typically begin when the following preconditions hold:

The reliability requirements to be tested have been specified.
The relevant software components have passed unit testing.
Software integration has occurred (i.e., although reliability testing can begin prior to the distribution of the software components onto the hardware components, accurate estimates require).
The relevant system components have passed system integration testing.
The independent test team is adequately staffed and trained in reliability testing.
The test environment is ready.

Completion Criteria

Reliability testing is typically complete when sufficient testing has occurred to provide statistically valid evidence enabling the probability of failure to be estimated.

Tasks

Reliability testing typically involves the independent test team performing the following testing tasks using the following techniques:

Test Planning
- Determine the application’s usage profile.
- Determine the number of test cases representative of the application’s usage profile that are sufficient to provide statistically valid evidence to show whether the application’s reliability requirements will be met.
- Determine the equipment, tools, and support software required to produce the test inputs, the expected outputs, and to compare the expected outputs with the actual outputs.
Test Reuse
Test Design
- For all tests, determine the:
  - Length of time of each test.
  - Time-related function describing how each test input will vary over the test period. Where appropriate, include the effects of instrument response times, signal noise, etc.
  - Statistical distribution of the initial values for each test input.
Test Implementation
- For each test, randomly select the test inputs.
Test Execution
Test Reporting

Environments

Load testing is typically performed on the following environments using the following tools:

Test Environment:
- Test Harness
- Test Case Generation Tools

Work Products

Reliability testing typically results in the production of all or part of the following work products from the test work product set:

Documents:
Software and Data:

Phases

Reliability testing typically consists of the following tasks being performed during the following phases:

PHASE → TASK ↓	Business Strategy (*)	Business Optimization	Initiation	Construction	Delivery	Usage	Retirement
Test Planning	Not Applicable	Not Applicable	Completed	Optional Regression	Not Applicable	Not Applicable	Not Applicable
Test Reuse	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Not Applicable	Not Applicable
Test Design	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Implementation	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Execution	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Reporting	Not Applicable	Not Applicable	Not Applicable	Completed	Not Applicable	Optional Regression	Not Applicable

(*) Optional reliability testing of COTS software components during the technology analysis and technology vendor selection tasks.
(**) Optional reliability testing of the executable architecture as well as the COTS components during the vendor and tool evaluation and vendor and tool selection tasks.

Guidelines

Accurate estimation of application reliability requires deployment of unit and integrated software components onto the actual hardware components. If reliability testing is to occur in a test environment, then to the extent practical the test environment should use the same hardware and the same operating system.
The iterative and incremental development cycle implies that reliability testing is regularly performed in an iterative and incremental manner.
Reliability testing must be automated (where practical) if adequate testing is to occur.
Reliability testing can elicit failures prior to launch.
To the extent practical, reuse functional test cases as reliability test cases.