Robustness Testing

Definition

Robustness testing is the system testing of an integrated, blackbox application against its robustness requirements.

Goals

The typical goals of robustness testing are to:

• Cause the application to fail under invalid conditions so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.

Objectives

The typical objectives of robustness testing are to:

• Partially validate the application (i.e., to determine if it fulfills its robustness requirements):
  • Operational requirements specified by exceptional use case paths.
  • Operational availability requirements.
  • Reliability requirements.
  • Robustness requirements.
• To help ensure that the application will not fail catastrophically when the unexpected occurs by determining how well it:
  • Handles failures of its:
    • Hardware components.
    • Software components.
    • Dependent applications (e.g., ISP provider or back-end legacy system) or databases (e.g., unavailability).
    • Human users or operators (e.g., invalid data input).
  • Gracefully performs:
    • Rollover and failover.
    • Disaster recovery.
• Cause failures concerning the exceptional operational requirements and robustness requirements that help identify defects that may not have been found during unit and integration testing.
• Report these failures to the development teams so that the associated defects can be fixed.
• Help determine the extent to which the application is ready for launch.
• Provide input to the defect trend analysis effort.

Examples

Typical examples include robustness testing of an application that is:

• Software only.
• A system including software, hardware (that may fail), and data components.
• Able to deal with bad inputs.
• Embeded within another system (e.g., flight-control software, cruise-control software).
• Client/server or n-tier distributed.
• A research prototype that will not be placed into service.
• Business-critical or safety-critical.

Preconditions

Robustness testing can typically begin when the following preconditions hold:

• The exceptional operational requirements and the robustness requirements to be tested have been specified.
• The relevant software components have passed unit testing.
• Software integration has occurred (i.e., robustness testing can begin prior to the distribution of the software components onto the hardware components).
• The relevant system components have passed system integration testing.
• The independent test team is adequately staffed and trained in robustness testing.
• The test environment is ready.

Completion Criteria

Robustness testing is typically complete when at least one test case exists and executes successfully for every important, probable system failure mode exists:

• Exceptional use case path.
• External server application:
  • Each legacy application individually removed from the network to determine if the failure would be handled gracefully.
  • Each ISP individually removed from the network to determine if another would take over.
• Hardware failure:
  • Commercial power supply disabled to determine if the Uninterruptible Power Supply (UPS) would take over until the generator supplies power.
  • Primary UPS disabled to determine if the secondary would take over.
  • Primary Power Distribution Unit (PDU) disabled to determine if the secondary would take over.
  • Primary network disabled to determine if the secondary would take over.
  • Workstation(s) removed from network to determine if the others would take over.
  • Primary router removed from network to determine if the secondary would take over.
  • Primary firewall removed from network to determine if the secondary would take over.
  • Primary load balancer removed from network to determine if the secondary would take over.
  • One switch removed from network to determine if the others would take over.
  • Web server(s) removed from network to determine if load balancing and failover would occur.
  • Application server(s) removed from network to determine if load balancing and failover would occur.
  • Database server(s) removed from network to determine if load balancing and failover would occur.
  • Disk/tape library removed from network to determine if load balancing and failover would occur.
• Software component failure.
  • Each COTS component is forced to fail.
  • Each database is forced to fail.
• Disaster recovery:
  • Operator responses to disasters are evaluated.

Tasks

Robustness testing typically involves the independent test team performing the following testing tasks using the following techniques:

• Test Planning
• Test Reuse
• Test Design:
  • Use case based testing
• Test Implementation
• Test Execution
  • Regression testing
• Test Reporting

Environments

Load testing is typically performed on the following environments using the following tools:

• Test Environment:
  • Test Harness
  • Use Case Tools
  • Record and Playback Tools

Work Products

Robustness testing typically results in the production of all or part of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Test Procedures
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Harness
  • Test Scripts
  • Test Suites
  • Test Cases
  • Test Data

Phases

Robustness testing typically consists of the following tasks being performed during the following phases:

Guidelines

• The iterative and incremental development cycle implies that robustness testing is regularly performed in an iterative and incremental manner.
• Robustness testing must be automated (where practical) if adequate regression testing is to occur.
• Robustness testing can elicit failures prior to launch.
• To the extent practical, reuse functional test cases as robustness test cases.
• Robustness testing may overlap slightly with usability testing in that disaster recovery may involve manual procedures performed by members of the operational organization.