Testability Requirements

Definition

A testability requirement is a developer-oriented quality requirement that specifies a required amount of testability, which is a quality factor that is defined as follows:

Testability

the degree to which something facilitates the creation and execution of successful tests (i.e., tests that cause failures due to underlying defects).

Objectives

The typical objectives of an testability requirement are to:

• Ensure that an application or component:
  • Is feasible to test.
  • Can be tested effectively and efficiently.
• Thereby minimize the cost of testing.
• Maximize the defects that can be found by testing.

Examples

The following are typical examples of testability requirements:

• “The component shall provide a [potentially separate] test interface that enables its:
  • State to be controlled.
  • State to be observed.”
• “The application shall provide a human interface that enables the human tester role to perform his/her application testing tasks.”
• “The component shall include built-in self-test software that automatically and continuously tests the component while it is in operation.”
• “The component shall be delivered with associated test software and test data that is sufficient to enable regression testing of all of its associated functional requirements.”

Guidelines

The following guidelines have been found to be useful when producing testability requirements:

• The scope of a testability requirement can be:
  • A business enterprise
  • An application
  • A component
  • A center
  • A document
• Testability is usually defined in terms of:
  • Controllability, which is the ease with which the application or component can be:
    • Placed into the proper pretest state.
    • Stimulated with the test message or data.
  • Observability, which is the ease with which the application or component can be observed:
    • To be in the proper pretest state.
    • To provide the proper output to its clients, peers, and servers (e.g., returned values, output messages, output requests for data).
    • be in the proper posttest state.
• Testability requirements are more likely to be needed when the application or component to be specified will naturally have limited controllability or observability. Thus, expect testability requirements when applications or components tend to have minimal interfaces combined with significant internal complexity.
• When testability is increased, coupling is increased and information hiding is decreased. For example, a separate testing interface that enables testers and testing software to directly manipulate hidden state can be misused to enable malicious access and modification of secure information or to enable accidental changes to the state that would place the component into an inconsistent or inappropriate state. Because increasing testability may thus decreases maintainability and security, care should be taken to properly prioritize these potentially conflicting quality requirements. For example, security requirements should specify that only testers and test software are authorized to use test interfaces and only for testing purposes.
• Object-oriented software is typically less testable (but more maintainable and secure) because:
  • It minimizes coupling.
  • It maximizes information hiding.
  • The parameters of input test stimuli (messages) and the associated outputs can be complex objects with hidden state rather than simple data types with visible values.
• Avoid unnecessarily specifying testing mechanisms (e.g., built-in tests, separate test interface) as testing constraints, and clearly differentiate testability requirements from testing constraints.
• Care should be taken to avoid accidentally misidentifying testing-related process requirements (i.e., how testing tasks shall be performed) as testability product requirements (i.e., how deliverable products shall be testable).