Configuration Testing

Definition

Configuration testing is the system testing of different variations of an integrated, blackbox application against its configurability requirements.

The typical goals of configuration testing are to:

Cause the application to fail to meet its configurability requirements so that the underlying defects can be identified, analyzed, fixed, and prevented in the future.

The typical objectives of configuration testing are to:

Partially validate the application (i.e., to determine if it fulfills its configurability requirements).
Cause failures concerning the configurability requirements that help identify defects that are not efficiently found during unit and integration testing:
- Functional Variants.
- Internationalization (e.g., multiple languages, currencies, taxes and tariffs, time zones, etc.).
- Personalization
Report these failures to the development teams so that the associated defects can be fixed.
Determine the effect of adding or modifying hardware resources such as:
- Memory
- Disk and tape resources
- Processors
- Load balancers
Determine an optimal system configuration.

Typical examples include configuration testing of an application that must:

Configuration testing can typically begin when the following preconditions hold:

The configurability requirements to be tested have been specified.
Multiple variants of the application exist.
The relevant software components have passed unit testing.
Software integration testing has started. However, configuration testing can begin prior to the distribution of the software components onto the hardware components.
The relevant system components have passed system integration testing.
The independent test team is adequately staffed and trained in configuration testing.
The test environment is ready.

Configuration testing is typically complete when the following postconditions hold:

At least one configuration test suite exists for each configurability requirement.
The test suites for every scheduled configurability requirement execute successfully on the appropriate configuration.

Configurability testing typically involves the independent test team performing the following testing tasks:

Configuration testing is performed on the following environments using the following techniques:

Configuration testing typically results in the production of all or part of the following work products from the test work product set:

Configuration testing typically consists of the following tasks being performed during the following phases:

PHASE → TASK ↓	Business Strategy (*)	Business Optimization	Initiation	Construction	Delivery	Usage	Retirement
Test Planning	Not Applicable	Not Applicable	Completed	Optional Regression	Not Applicable	Not Applicable	Not Applicable
Test Reuse	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Not Applicable	Not Applicable
Test Design	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Implementation	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Execution	Not Applicable	Not Applicable	Optionally Started (**)	Completed	Not Applicable	Optional Regression	Not Applicable
Test Reporting	Not Applicable	Not Applicable	Not Applicable	Completed	Not Applicable	Optional Regression	Not Applicable

(*) Optional configuration testing of COTS software components during the technology analysis and technology vendor selection tasks.
(**) Optional configuration testing of the executable architecture as well as the COTS components during the vendor and tool evaluation and vendor and tool selection tasks.

The iterative and incremental development cycle implies that configuration testing is regularly performed in an iterative and incremental manner.
Configuration testing must be automated if adequate regression testing is to occur.
To the extent practical, reuse functional test cases as configuration test cases.