Contention Testing

Definition

Contention testing is the system testing of an integrated, blackbox application that attempts to cause failures involving actual or simulated concurrency.

Goals

The typical goals of contention testing are to:

• Cause the application to fail with regard to actual or simulated concurrency so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.

Objectives

The typical objectives of contention testing are to:

• Cause failures involving concurrency (e.g., excessive locking, deadlock, livelock, starvation, race conditions, priority inversion, and lack of thread safety in shared software components or data) that help identify defects that are not efficiently found during unit and integration testing.
• Report these failures to the development teams so that the associated defects can be fixed.
• Help determine the extent to which the system is ready for launch.
• Provide input to the defect trend analysis effort.

Examples

Typical examples include contention testing of an application that has:

• Only a single thread on a single process.
• Multiple processes (i.e., multiple processors or multiple computers).
• Multiple processes, each with multiple threads.

Preconditions

Contention testing can typically begin when the following preconditions hold:

• The relevant software components have passed unit testing.
• Software integration testing has started. However, contention testing can begin prior to the distribution of the software components onto the hardware components.
• The relevant system components have passed system integration testing.
• The independent test team is adequately staffed and trained in contention testing.
• The test environment is ready.

Completion Criteria

Contention testing is typically complete when the following postconditions hold:

• At least one configuration test suite exists for each process or thread.
• All scheduled contention test suites execute successfully on the appropriate configuration.

Tasks

Contention testing typically involves the independent test team performing the following testing tasks:

• Test Planning
• Test Reuse
• Test Design:
  • Concurrency Analysis
• Test Implementation
• Test Execution
• Test Reporting

Environments

Contention testing is typically performed on the following environments using the following tools:

• Test Environment:
  • Test Harness

Work Products

Contention testing typically results in the production of all or part of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Test Procedures
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Harness
  • Test Scripts
  • Test Suites
  • Test Cases
  • Test Data

Phases

Contention testing typically consists of the following tasks being performed during the following phases:

Guidelines

• The iterative and incremental development cycle implies that contention testing is regularly performed in an iterative and incremental manner.
• Contention testing must be automated if adequate regression testing is to occur.
• To the extent practical, reuse functional test cases as contention test cases.