Testability

Definition

Testability

the development-oriented quality factor representing the degree of the easiness to which a system, application, component, or model facilitates the creation and execution of successful tests (i.e., tests that cause failures due to underlying defects)

Classification

As illustrated in the preceding figure, Testability is part of the following inheritance hierarchy:

• Type: Abstract
• Superclass: Development-Oriented Quality Factor
• Subclasses:
  • System Testability
  • Application Testability
  • Component Testability:
    • Data Component Testability
    • Hardware Component Testability
    • Personnel Component Testability
    • Software Component Testability
  • Model Testability

Responsibilities

The typical responsibilities of testability are to:

• Model the degree to which an application or component facilitates the creation and execution of successful tests.
• Support the analysis and specification of testability requirements.
• Provide a foundation for evaluating the quality of an architecture.

Subfactors

Testability is typically decomposed into the following aggregation hierarchy of subfactors:

• Controllability,
  which is the ease with which the application or component can be:
  • Placed into the proper pretest state.
  • Stimulated with the test message or data.
• Observability,
  which is the ease with which the application or component can be observed to:
  • Be in the proper pretest state.
  • Provide the proper output to its clients, peers, and servers (e.g., returned values, output messages, output requests for data).
  • In the proper posttest state.

Measures

Testability is typically measured in terms of:

• Controllability
  Percentage of states that can be directly set via external interfaces:
  • Object Under Test
  • Inputs (e.g., messages received, exceptions handled)
  • Outputs (e.g., messages sent, exceptions thrown)
• Observability
  Percentage of states that are directly observable via external interfaces:
  • Object Under Test
  • Inputs (e.g., messages received, exceptions handled)
  • Outputs (e.g., messages sent, exceptions thrown)
• Effort
  Average/estimated person-hours required to produce a complete set of test cases:
  • Message coverage
  • Exception coverage
  • State coverage:
    • Object Under Test (OUT) State Coverage
      Every event (e.g., message and exception) in every OUT state
    • Input State Coverage
      Every state of every input event (e.g., message and exception)
    • Output State Coverage
      Every state of every output event (e.g., message and exception)
  • Branch coverage
• Tool Support
  Percentage of tests that can be automated with existing tools

Mechanisms

Typical mechanisms for implementing testability include:

• Built-in tests
• Instrumentation
• Refactoring
• Separate test interfaces
• Tool support

Guidelines

The following guidelines have been found to be useful regarding testability:

• Testability is more likely to be a concern when the application or component to be specified will naturally have limited controllability or observability. Thus, expect testability to be important when applications or components tend to have minimal interfaces combined with significant internal complexity.
• When testability is increased, coupling is increased and information hiding is decreased. For example, a separate testing interface that enables testers and testing software to directly manipulate hidden state can be misused to enable malicious access and modification of secure information or to enable accidental changes to the state that would place the component into an inconsistent or inappropriate state. Because increasing testability may thus decreases maintainability and security, care should be taken to properly prioritize these potentially conflicting quality requirements. For example, security requirements should specify that only testers and test software are authorized to use test interfaces and only for testing purposes.
• Object-oriented software is typically less testable (but more maintainable and secure) because:
  • It minimizes coupling.
  • It maximizes information hiding.
  • The parameters of input test stimuli (messages) and the associated outputs can be complex objects with hidden state rather than simple data types with visible values.