ࡱ> }xyz{|q  %bjbjt+t+ ^AAf] X @8@c?X<#R#R#R#R# -0???????$@B6?-`2R#R#`2`26?4R#R#444`2"R#R#?..< .j .`2?44;f-|?R#pti Project Test Plan (PTP) Version Initial Draft Produced for: Produced by: Revision History DateVersionDescriptionAuthorTBD0.1Initial draftTBD0.20.3 Table of Contents  TOC \o "1-5" 1 Introduction  PAGEREF _Toc499001562 \h 6 1.1 Project Test Plan Goals  PAGEREF _Toc499001563 \h 6 1.2 Project Test Plan Objectives  PAGEREF _Toc499001564 \h 6 1.3 Intended Audience  PAGEREF _Toc499001565 \h 6 1.4 References  PAGEREF _Toc499001566 \h 6 1.5 Project Test Plan Overview  PAGEREF _Toc499001567 \h 7 2 Test Philosophy  PAGEREF _Toc499001568 \h 9 2.1 Essential Concepts  PAGEREF _Toc499001569 \h 9 2.2 Observations Regarding Testing  PAGEREF _Toc499001570 \h 10 2.3 Testing Goals  PAGEREF _Toc499001571 \h 11 2.4 Testing Objectives  PAGEREF _Toc499001572 \h 11 3 Test Work Products  PAGEREF _Toc499001573 \h 13 3.1 Documents  PAGEREF _Toc499001574 \h 13 3.1.1 Project Test Plan  PAGEREF _Toc499001575 \h 13 3.1.2 Master Test List  PAGEREF _Toc499001576 \h 14 3.1.3 Test Procedures  PAGEREF _Toc499001577 \h 15 3.1.4 Test Reports  PAGEREF _Toc499001578 \h 16 3.1.5 Summary Test Reports  PAGEREF _Toc499001579 \h 17 3.2 Software  PAGEREF _Toc499001580 \h 18 3.2.1 Test Harnesses  PAGEREF _Toc499001581 \h 18 3.2.2 Test Scripts  PAGEREF _Toc499001582 \h 19 3.2.3 Test Suites  PAGEREF _Toc499001583 \h 19 3.2.4 Test Cases  PAGEREF _Toc499001584 \h 21 3.2.5 Test Data  PAGEREF _Toc499001585 \h 22 4 Teams  PAGEREF _Toc499001586 \h 24 4.1 Management Team  PAGEREF _Toc499001587 \h 24 4.2 Requirements Team  PAGEREF _Toc499001588 \h 24 4.3 Software Development Team  PAGEREF _Toc499001589 \h 24 4.4 Integration Team  PAGEREF _Toc499001590 \h 25 4.5 System Test Team  PAGEREF _Toc499001591 \h 25 4.6 Security Team  PAGEREF _Toc499001592 \h 26 4.7 Usability Team  PAGEREF _Toc499001593 \h 26 5 Testing Tasks  PAGEREF _Toc499001594 \h 28 5.1 Test Planning  PAGEREF _Toc499001595 \h 28 5.2 Test Reuse  PAGEREF _Toc499001596 \h 28 5.3 Test Design  PAGEREF _Toc499001597 \h 29 5.4 Test Implementation  PAGEREF _Toc499001598 \h 29 5.5 Test Execution  PAGEREF _Toc499001599 \h 29 5.6 Test Reporting  PAGEREF _Toc499001600 \h 29 5.7 Test Evaluation  PAGEREF _Toc499001601 \h 30 6 Test Tools  PAGEREF _Toc499001602 \h 31 7 Organizational Performance Testing  PAGEREF _Toc499001603 \h 32 8 Model Testing  PAGEREF _Toc499001604 \h 33 9 Unit Testing  PAGEREF _Toc499001605 \h 35 10 Commercial Component Testing  PAGEREF _Toc499001606 \h 38 11 Integration Testing  PAGEREF _Toc499001607 \h 40 11.1 Commercial Component Integration Testing  PAGEREF _Toc499001608 \h 40 11.2 Software Integration Testing  PAGEREF _Toc499001609 \h 41 11.3 System Integration Testing  PAGEREF _Toc499001610 \h 42 12 System Testing  PAGEREF _Toc499001611 \h 44 12.1 Functional Testing  PAGEREF _Toc499001612 \h 45 12.2 Performance Testing  PAGEREF _Toc499001613 \h 47 12.3 Load Testing  PAGEREF _Toc499001614 \h 49 12.4 Stress Testing  PAGEREF _Toc499001615 \h 51 12.5 Robustness Testing  PAGEREF _Toc499001616 \h 54 12.6 Configuration Testing  PAGEREF _Toc499001617 \h 56 12.7 Contention Testing  PAGEREF _Toc499001618 \h 57 12.8 Availability Testing  PAGEREF _Toc499001619 \h 58 12.9 Portability Testing  PAGEREF _Toc499001620 \h 59 12.10 Security Testing  PAGEREF _Toc499001621 \h 60 12.11 Usability Testing  PAGEREF _Toc499001622 \h 63 13 Launch Testing  PAGEREF _Toc499001623 \h 66 13.1 Alpha Testing  PAGEREF _Toc499001624 \h 66 13.2 Beta Testing  PAGEREF _Toc499001625 \h 68 13.3 Acceptance Testing  PAGEREF _Toc499001626 \h 70 Appendices  PAGEREF _Toc499001627 \h 73 A. Testing Techniques  PAGEREF _Toc499001628 \h 73 A.1 Assertions and Exceptions  PAGEREF _Toc499001629 \h 73 A.2 Equivalence Set Testing  PAGEREF _Toc499001630 \h 74 A.3 Boundary Value Testing  PAGEREF _Toc499001631 \h 74 A.4 State Based Testing  PAGEREF _Toc499001632 \h 76 A.5 Structured Testing  PAGEREF _Toc499001633 \h 76 A.6 Error Guessing  PAGEREF _Toc499001634 \h 77 A.7 Abstract Class via Concrete Test Subclass  PAGEREF _Toc499001635 \h 77 A.8 Interface via Concrete Class  PAGEREF _Toc499001636 \h 77 A.9 Embedded Test Suite  PAGEREF _Toc499001637 \h 78 A.10 Parallel Test Driver Hierarchy  PAGEREF _Toc499001638 \h 78 A.11 Test Subclasses  PAGEREF _Toc499001639 \h 78 A.12 Peer Testing  PAGEREF _Toc499001640 \h 78 A.13 Test First  PAGEREF _Toc499001641 \h 79 A.14 Bottom Up Testing  PAGEREF _Toc499001642 \h 79 A.15 Hierarchical Incremental Testing  PAGEREF _Toc499001643 \h 79 A.16 Polymorphism Testing  PAGEREF _Toc499001644 \h 80 A.17 Use Case Based Testing  PAGEREF _Toc499001645 \h 80 A.18 Record and Playback  PAGEREF _Toc499001646 \h 80 A.19 Automated Testing  PAGEREF _Toc499001647 \h 81 A.20 Regression Testing  PAGEREF _Toc499001648 \h 81 B. Open Issues  PAGEREF _Toc499001649 \h 81 C. Major Things To Be Done  PAGEREF _Toc499001650 \h 81 Introduction This mandatory deliverable document formally describes the plans for performing all testing on the project. Project Test Plan Goals The goals of this plan are to enable: Stakeholders (e.g., clients, managers) to understand how the application will be tested. The management team to properly schedule and staff the testing activity. Various test teams to understand their testing responsibilities. Project Test Plan Objectives The objectives of this plan are to document the: Project test philosophy in terms of essential concepts, goals, and objectives. Test work products to be produced. Teams involved with testing and their responsibilities. Testing tasks to be performed. Kinds of tests to be performed on the project. Intended Audience This plan has the following the intended audience: Client Team Management Team Requirements Team Software Development Team System Test Team Security Team Usability Team References This plan has the following the references: Project Documents: Environments Design Document, which documents the design of the test environments OPF Conventions: Project Test Plan Content and Format Standard, which specifies the content and format of this test plan Project Test Plan Template, which provides a starting point for this test plan Project Test Plan Inspection Checklist, which includes questions to be answered during the inspection of this test plan Reference Books: Robert Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison-Wesley, Reading, Massachusetts, 2000, pages 1191. Abstract: The most complete and correct reference book available. Industry Publications regarding Testing Techniques: Assertions and Exceptions. Interactive Software Engineering, Building Bug-free OO Software: An Introduction to Design by Contract"! on the web page http://www.eiffel.com/doc/manuals/technology/contract/index.htm, 1996, pages 1-10. Abstract: This paper introduces the main concepts (contracts, assertions, exception handling) and uses of Design by Contract including specification, analysis, documentation, testing, debugging, and quality assurance. State-Based Testing. Robert V. Binder, State-Based Testing, Object Magazine, Vol. 5, No. 4, SIGS Publications Inc., New York, New York, May 1995, pages 75-78. Abstract: This article documents the use of transition trees to generate specification-based test cases for class state models. Structured Testing. Thomas J. McCabe, Lori A. Dreyer, Albert J. Dunn, and Arthur H. Watson, Testing an Object-Oriented Application, The Journal of the Quality Assurance Institute, Vol. 8, No. 4, October 1994, pages 21-27. Abstract: This article discusses the application of structured testing rules to object-oriented software, the impact of an object-oriented programming language on control flow, the integration testing of an object-oriented application, safe and unsafe operations and classes, object-oriented metrics, and an approach to testing object-oriented application based on safety. Hierarchical Incremental Testing (HIT). John D. McGregor, Parallel Architecture for Component Testing, Journal of Object-Oriented Programming, Vol. 10, No. 2, SIGS Publications Inc., New York, New York, March-April 1997, pages 10-14. Abstract: This article describes the parallel architecture for component testing (PACT) consisting of test driver classes that mirror the structure of the classes they test. It also briefly describes 10 patterns in a generative pattern language that justifies PACT and provides guidance in how it should be used. Polymorphism Testing. R. Brownlie, J. Prowse, and M. S. Phadke, Robust Testing of ATT PMX/Starmail using OATS, ATT Technical Journal, May-June 1992, pages 41-47. Use Case Based Testing. Robert V. Binder, Use cases, Threads, and Relations: The FREE Approach for System Testing, Object Magazine, Vol. 5, No. 9, SIGS Publications Inc., New York, New York, February 1996, pages 73-81. Abstract: This article discusses the system-test component of the Flattened Regular Expression (FREE) testing method, which is based on using use cases to develop the system test plan. Binder discusses the following levels of testing: (0) ad hoc, (1) functional compliance, (2) reliability optimization, and (3) integrity verification. Functional compliance is based on use case coverage. Reliability optimization bases the number of test cases per use case on the operational profile. Integrity verification is based on thread coverage. Project Test Plan Overview This PTP has the following the organization: Introduction, which introduces this PTP to its readers. Test Philosophy, which defines key testing concepts and the goals and objectives of testing. Test Work Products, which documents the work products to be produced during the testing activity. Test Teams, which documents the teams involved in testing and their responsibilities. Test Tasks, which documents the testing tasks to be performed by the test teams. Test Tools, which documents the tools that will be used to perform testing. Model Testing, which documents the project plans regarding the testing of models. Unit Testing, which documents the project plans regarding the testing of individual units of software. Package Testing, which documents the project plans regarding the testing of commercial off the shelf (COTS) packages. Integration Testing, which documents the project plans for performing each of the following kinds of integration tests: Package integration testing Software integration testing System integration testing System Testing, which documents the project plans for performing each of the following kinds of system tests: Functional testing Performance testing Load testing Stress testing Robustness testing Contention testing Availability testing Portability testing Security testing Usability testing Launch Testing, which documents the project plans for performing each of the following kinds of tests during the Launch Phase: Alpha testing Beta testing Acceptance testing Appendices, which document testing techniques, major open issues, and items to be done. This PTP does not contain the following information that is documented elsewhere: Description of the test environments, which is documented in the environments design document. Listing of test suites and associated test cases, which are documented in the master test list. Descriptions of how to run individual test suites and test cases, which are documented in the test procedures. Test Philosophy This section of the project test plan documents the essential concepts of testing as well as the high-level goals and associated detailed objectives of testing on the project. Essential Concepts In order to understand testing, one must understand its essential concepts: A failure is the occurrence of an inconsistency between an executable work products actual behavior and its expected behavior. Clients naturally expect their delivered work products to behave as expected, and a failure has occurred when one does not. A work product can fail in many ways; for example, a software component can fail to correctly: Return the expected value. Throw the expected exception. Be left in the appropriate state. Send the correct messages with the correct parameters to the correct collaborators in the correct order. Handle the exceptions thrown to it. Complete execution within the required time. An oracle is the source (e.g., the requirements, design, or an authoritative domain expert) of information that specifies the expected (i.e., correct) behavior of an executable work product. A defect (a.k.a., fault or bug) is the underlying problem in the work product that causes the failure. A defect will typically not cause a failure unless it is executed under the appropriate circumstances. An error is the mistake that a human makes that results in the existence of a defect in the work product. Thus, a human error causes a defect, which in turn may cause one or more failures. A test is the controlled execution of a work product in the attempt to cause it to fail. A good test is one that causes one or more failures that in turn can be localized to identify one or more previously unknown underlying defects. Testing is the activity consisting of all tasks specifically involving tests. A regression test is the repetition of a test after the work product has been iterated to determine if any defects have been inadvertently introduced (i.e., if the work product has regressed). The iterative and incremental nature of the object-oriented development cycle greatly increases the frequency of regression testing and therefore increases the need to automate regression testing. Test completion criteria are documented criteria used to determine the adequacy of testing. A specific kind of testing can be stopped when the associated test completion criteria are met. Because the number of potential test cases is typically indefinitely large, test completion criteria are almost never comprehensive. Defect severity is a measurement of the importance of a defect in terms of the impact of its associated failures. Defect severity is used to determine the priority of defect fixing and whether or not the system is ready to launch. Severity one defect: A severity one defect causes catastrophic failure of the system or one of its essential components. A severity one defect prevents effective exception handling, preventing further system responses to at least one user. Severity two defect: A severity two defect causes the system to violate a business rule, a primary use case path, or a quality requirement affecting users. An example would be a defect that causes incorrect results to be returned to a user in response to a query. Severity three defect: A severity three defect is causes the system to violate a secondary use case path or causes an inconvenience to the users. An example would be data returned to a user that is correct but incorrectly formatted on the webpage. Observations Regarding Testing The philosophy of testing is based on the following observations: Testing and Quality: Although testing can show that the existence of defects, testing cannot prove that the application is defect free. You cannot test in quality at the end. Quality must be built in to the application from the very beginning. Testing is thus no substitute for good engineering of work products in the first place. Quality is not solely the responsibility of the testers. Everyone involved in the project is responsible for the quality of the application. Testing in the development cycle: Testing is an incremental activity, with additional test suites of test cases added over time until the test completion criteria are met. Additional regression tests are also added to cause failures as they are identified and fixed. Testing is an iterative activity, which test work products being iterated and improved over time and defects in the test work products are identified and fixed. Testing is a parallel activity in that it occurs concurrently with all other major project activities. Testing starts very early in the development process and continues throughout the development cycle. Early testing prevents defects from occurring by ensuring the testability of requirements. Early testing also prevents defects from lasting until late in the project by detecting them via failures as they are introduced via human error. Testing is a time-boxed activity, in that testing occurs in the context of scheduled builds and milestones. Destructive Testing: Testing is destructive rather than constructive. A good test causes failures. A great test causes failures that can be used to identify lots of defects. A good tester likes to break things. Few developers are psychologically motivated to use testing to break their own work products. Independent testing is typically more productive than developer testing for large software components and applications. Defects identified early in the development cycle are much easier, and less costly, to fix than those that are not found until after the application has been launched. Finding defects early also minimizes the risk of delivering an inadequate application. Testing work products are just as likely to initially contain defects as are deliverable work products. Tests can also have defects causing: False negative results - the test fails to report failures that occur. False positive results - the test reports failures that have not occurred. A good test is one that has a high probability of causing failures due to as yet undiscovered defects. A successful test is one that causes one or more failures due to one or more as yet undiscovered defects and that provides enough information to localize these defects. Object-orientation changes testing: Unit testing of object-oriented software is class testing, whereas unit testing of procedural software is function testing. Thus, object-oriented unit testing corresponds to the lowest level of traditional integration testing. Dynamic binding and complex inheritance create new kinds of defects due to unanticipated bindings and misinterpretations. Testability has decreased. Testability is a combination of controllability (the ability to set up test preconditions and control test stimuli) and observability (the ability to observe the actual test results so that they can be compared with the expected test results). Abstraction, inheritance, polymorphism, and exceptions decrease controllability, while encapsulation and information hiding decrease observability. Techniques have evolved to make object-oriented software more testable. The test oracles have changed. Test cases are now based on use cases, class responsibilities, class assertions, and the analysis of polymorphism and state models. User interface testing has changed due to the advent of modern GUIs and webpages. Techniques have changed. Structured testing and McCabes cyclometric complexity are less important due to the small sizes and simplicity of class methods. State-based testing is more important due to object state. Black box testing is more important due to information hiding and encapsulation. Testing Goals The testing activity has the following top-level goals: To provide input regarding the quality of the application. To reduce development risk. To provide input regarding the readiness of the application for launch. To provide input to improve the development process. To provide input to improve the effectiveness of staff training. Testing Objectives To support its goals, the testing activity has the following objectives: To cause failures in the executable work products under test so that any underlying defects can be identified and removed. To help identify defects that would be more costly to identify using other verification and validation techniques. To enable defect analysis so that the human errors causing these defects can be avoided in the future (e.g., via process iteration and staff training). To maximize the productivity of the testing teams (e.g., reuse of test plans, test harnesses, and test suites of test cases). Although initiated by testing, the following are not part of the testing activity: Refactoring Defect removal Defect analysis Process improvement Staff training Test Work Products This section of the project test plan documents the following test work products that are produced during the testing activity: Documents: Project Test Plan Master Test List Test Procedures Test Reports Summary Test Reports Software: Test Harnesses Test Scripts Test Suites Test Cases Test Data Although the Statement of Work (SOW) is not a testing document, it does summarize the scope of the testing effort and the required resources. Documents Project Test Plan Definition: The project test plan is the mandatory deliverable document that formally describes the plans for performing all testing on a project. The goals of the project test plan include: Enable stakeholders (e.g., clients and managers) to understand how the application will be tested. Enable the management team to properly schedule and staff the testing activity. Enable various test teams to understand their testing responsibilities. The contents of a project test plan include: Front matter (title page, revision page, table of contents, table of figures) Introduction (objectives, intended audience, references) Project test philosophy. Test work products to be produced. Teams involved with testing and their responsibilities. Testing tasks to be performed on the project. Kinds of tests to be performed on the project. Appendices Risks include: Failure to produce a project test plan: Increases the risk of inadequate testing and consequent failures in the production system. Decreases productivity, as stakeholders must determine their testing responsibilities. Inputs include: Project Statement of Work (SOW) Vision Statement System Requirements Specification Project Test Plan Content and Format Template Project Test Plan Inspection Checklist Evaluation criteria include: Is the project test plan complete, covering all required information and kinds of testing? Has inappropriate content been removed during tailoring? Is the project test plan being kept current? Guidelines include: Tailor the template, removing inappropriate information and adding missing information. Master Test List Definition: The master test list the mandatory deliverable document that lists all system and launch test suites of test cases. The goals of the test list include: Enable stakeholders (e.g., clients and managers) to understand how the application will be tested. Summarize all system test suites. The contents of the master test list include: Front matter (title page, revision page, table of contents, table of figures) Introduction (objectives, intended audience, references) For each kind of system and launch testing: The kind of testing. A list of associated test suites. For each test suite: The name of the test suite. The objective/scope of the test suite. Trace back to requirements. A list of associated test cases. Risks include: Failure to create a master test list makes it difficult to: Know if adequate system and launch testing is occurring. Perform regression testing. Identify the system tests to be reused during launch. Adding this information to the project test plan makes it much less reusable and require significantly more iteration. Inputs include: Project Test Plan System Requirements Specification Master Test List Content and Format Standard Master Test List Inspection Checklist Evaluation criteria include: Is the master test list complete, covering all types of testing, all test suites, and all test cases? Has inappropriate content been removed during tailoring? Is the master test list being kept current as new test suites and test cases are added (or removed)? Guidelines include: This document can be kept in the form of a database. Do not confuse the master test list with: Project Test Plan, which does not list individual test suites and test cases. Test Procedures, which only document a single test suite of test cases. Test Procedures Definition: Test procedures are informal documents that state how to manually execute a test suite of test cases. The goals of a test procedure include: Enable testers to manually execute a test suite of test cases. The contents of a test procedure include: How to set up the test preconditions (e.g., how to place objects, software components, databases in their correct pretest states). How to perform the test steps (e.g., stimulate the item under test by sending it messages, interact via webpages). How to determine and document the actual test postconditions (e.g., returned value, object state, database state, exceptions thrown or handled, and messages sent). How to compare the actual test postconditions with the oracles expected test outputs. How to report the results of the associated test. To whom to report the test results. Risks include: Failure to use test procedures makes it difficult for others than the test developer to perform manual testing and increases the difficulty of regression testing. Inputs include: Project Test Plan Master Test List Test Procedure Content and Format Standard Evaluation criteria include: Is the test procedure complete, covering all required information (e.g., test steps)? Has inappropriate content been removed during tailoring? Is the test procedure being kept current? Guidelines include: Because test procedures are primarily intended to document how human testers perform tests to support manual regression testing, test procedures can be deleted once the associated tests have been automated using test scripts. Do not confuse the test plan (which covers the overall testing approach at a strategic level) with the test procedure (which covers test plans at a local, tactical level). Test Reports Definition: Test reports are documents that formally document the results of executing one or more test suites. The goals of test reports include: Notify stakeholders of the results of a specific set of tests. The contents of a test report include: Coversheet (project, date, tester) Introduction: Test suites that were executed and their objectives Test environment Failures including: Test suite and test case. Requirements trace (for system tests) Inconsistent expected and actual behavior observed. Developer notified (for integration and system tests). Report any test suites or test cases that could not be performed: Test suite and test case. Exceptional behavior observed. Risks include: Failure to use test reports makes it difficult for developers to prioritize and perform defect fixing. Inputs include: Project Test Plan Test Report Content and Format Standard Evaluation criteria include: Is the test report complete, correct, and consistent with the standard? Guidelines include: Model test reports tend to be very informal. Unit test reports tend to be automatically generated by the test tool or the test suite. The only test reports that are formally documented are those that are used to communicate between teams (e.g., those that must be delivered to the development team by a system test team). System test reports may be delivered to the client. Summary Test Reports Definition: Test Summary Reports are periodic (e.g., weekly) reports to management documenting the current status of project testing. The goals of summary test reports include: Notify management of the current status of project testing (e.g., by seeing the percentage of planned tests that exist and by seeing the percentage of existing tests that pass). Enable management to determine: The effectiveness of defect removal If the application is ready for launch. The contents of a summary test report include: Coversheet (project, date) Introduction Planned tests (the total numbers of test suites and test cases that are currently planned to be produced) Existing tests (the total numbers of test suites and test cases that have been implemented and are being executed organized by type) Test summary (the total numbers and percentages of existing test cases that currently passed, failed including defect severity, and failed to execute). Trend studies: Chart of existing test cases (number and percentage, both total and organized by type) as a function of time (build) Chart of test cases passed (number and percentage, both total and organized by type) as a function of time (build) Risks include: Failure to use summary test reports prevents management and the client from: Understanding the current status of project testing. Knowing if the application is ready for launch. Inputs include: Project Test Plan Test Reports Test Report Content and Format Standard Evaluation criteria include: Is the test summary report complete and up to date? Is the test plan at the correct management level of abstraction? Are the tests that failed summarized by defect severity so that management can understand the relative importance of the defects found? Guidelines include: Generate this report regularly (e.g., weekly), summarizing the results of regression testing as well as initial testing. The test summary report can be automatically generated from a defect database. Software Test Harnesses Definition: A test harness is a software tool (e.g., JUnit) that automates the testing process by running test suites of test cases. The goals of a test harness include: Automate test execution. Provide a user interface to control test execution. To support these goals, the objectives of a test harness include: Execute test scripts. Generate associated test reports. Risks include: Failure to use test harnesses makes regression testing more labor intensive and less likely to be performed. Inputs include: Project Test Plan Requirements Specification System Architecture Document Design documentation (e.g., Javadoc) Evaluation criteria include: Are there any defects in the test harness? Is the test harness easy and intuitive to use? Is the test harness consistent with the test suites and test cases? Guidelines include: If the quality of the test harness is not at least as good as the quality of the item under test, then it will be difficult to know if the defect causing the failure is in the item under test or in the test harness. Test Scripts Definition: A test script is a software program (typically written in a procedural scripting language) that executes a test suite of test cases. The goals of a single test suite include: Automate the execution of test cases. Support regression testing To support these goals, the objectives of a test script are to: Execute each test case in the test suite. Report the results of the test suite. Risks include: The risks of not developing test scripts include the difficulty of automating regression testing. Test scripts could contain defects and may make it difficult to evaluate the results of executing the test script. Inputs include: Project Test Plan Requirements Specification System Architecture Document Design documentation (e.g., Javadoc) Evaluation criteria include: Did the test script execute without failure? Guidelines include: If the quality of the test script is not at least as good as the quality of the item under test, then it will be difficult to know if the defect causing the failure is in the item under test or in the test script. Test Suites Definition: A test suite is a cohesive collection of related test cases. The goals of a single test suite include: Perform a cohesive set of tests (e.g., the testing of a use case or a class). Cause failures that uncover underlying defects so that they can be identified and removed. Organize a cohesive amount of testing (e.g., one test suite per use case, one test suite per class). Help improve the quality of the item under test. Help developers understand the behavior of the item under test. Help developers improve the quality of the specifications (e.g., use case specification, class responsibilities) of the item under test. To support these goals, the objectives of a single test suite include: Document the purpose of the test suite (i.e., the part of the item under test being tested, the type of failures to be elicited). This could be to test an entire use case or class. Document the producer of the test suite. Provide a requirements trace (if a system or launch test suite). Execute its component test cases. Risks include: Failure to produce test suites increases the probability that the item under test will contain defects that will make the application fail to meet its requirements or miss its launch date: Lack of unit test suites increases the probability that integration and system testing will take significantly longer than expected. Lack of integration test suites increases the probability that system testing will take significantly longer than expected. Failure to automate test suites makes regression testing more expensive and less likely to occur. It is often difficult to identify the optimal (minimal) set of test cases that provide adequate test coverage. Failure to use appropriate testing techniques makes the identification of the component test cases difficult. Inputs include: Project Test Plan Requirements Specification System Architecture Document Design documentation (e.g., Javadoc) Evaluation criteria include: Is the test suite complete, capturing all necessary information? Is the test suite correct? Is the collection of test cases minimal (without redundancy)? Do the test cases provide adequate test coverage? Guidelines include: Use a combination of testing techniques to ensure adequate test coverage. An infinite number of test cases are often required to exhaustively test any nontrivial work product (e.g., class, software component, or application). Because exhaustive testing typically is impossible or at least impractical, the test suite should contain a minimal set of test cases that provides adequate coverage is critical. Test suites will be used at all levels of testing (e.g., model testing, unit testing, integration testing, and system testing). To support regression testing, test suites will be automated whenever practical. Separate the testing of server APIs and Human Interfaces into separate test suites. This enables the localization of defects to either the human interfaces or the underlying application layer. Test suites need not document how to perform the test unless they are automated. When performed manually, this information is documented in the associated test procedure. Test suites do not document the results of the tests, which are documented in the associated test report. Test Cases Definition: A test case is a work product that captures a single test on an executable work product. The goals of a single test case include: Perform a single test (e.g., a single test of a use case path or class method). Cause failures that uncover underlying defects so that they can be identified and removed. Help improve the quality of the item under test. Help developers understand the behavior of the item under test. Help developers improve the quality of the specifications (e.g., use case path and class responsibilities) of the item under test. To support these goals, the objectives of a single test case include: Document the purpose of the test case (i.e., the part of the item under test being tested, the type of failures to be elicited) Document the producer of the test case Prepare the item under test for testing (i.e., place the item under test, the test stimuli, and the collaborators of the item under test into their correct pretest states, and provide the necessary test data). Stimulate the item (e.g., send it test messages, raise test exceptions). Observe how the item responds (e.g., values returned, exceptions raised, changes in state, and messages sent) to the test stimuli. Compare the actual responses (i.e., postconditions) to the expected responses to identify failures that imply the existence of defects in the item under test. Report the results of the associated test. Risks include: Failure to produce test cases increases the probability that the item under test will contain defects that will make the application fail to meet its requirements. Failure to automate test cases makes regression testing more expensive and less likely to occur. Inputs include: Project Test Plan Requirements Specification System Architecture Document Design documentation (e.g., Javadoc, class assertions) Software (e.g., method signatures, assertions, branching and looping logic) Evaluation criteria include: Is the test case complete, capturing all necessary information? Is the test case correct? Guidelines include: Test cases will be used at all levels of testing (e.g., model testing, unit testing, integration testing, and system testing). To support regression testing, test cases will be automated whenever practical. The oracle can be incorrect, and the test case developer can make mistakes. Thus, test cases need to be evaluated for defects. If the quality of the test cases is not at least as good as the quality of the item under test, then it will be difficult to know if the defect causing the failure is in the item under test or in the test case. Test cases need not document how to perform the test unless they are automated. When performed manually, this information is documented in the associated test procedure. Test cases do not document the results of the tests, which are documented in the associated test report. Test Data Definition: Test data is a work product that captures the cohesive information (e.g., data or objects) that exists in the form of a file or database that tests need to execute successfully. The goals of test data include: Enable the performance of a test case Help developers understand the behavior of the item under test. To support these goals, the objectives of test data include: Provide the necessary content of files used during testing Provide the necessary content of database used during testing Risks include: Failure to produce test data means that many tests will not execute successfully. The oracle can be incorrect, and the test data developer can make mistakes. Thus, test data needs to be evaluated for defects. Inputs include: Project Test Plan Requirements Specification System Architecture Document Design documentation (e.g., Javadoc) Evaluation criteria include: Is the test data complete? Is the test data correct? Guidelines include: Use a combination of testing techniques to ensure adequate test coverage. An infinite amount of test cases is often required to exhaustively test any nontrivial work product (e.g., class, software component, or application). Because exhaustive testing typically is impossible or at least impractical, the test data should be the minimum amount that provides adequate coverage is critical. Test data will be used at all levels of testing (e.g., model testing, unit testing, integration testing, and system testing). To support regression testing, test data will be automated rather than entered manual whenever practical. Teams This section of the project test plan documents the responsibilities of the following teams that perform the tasks of the testing activity: Management Team Requirements Team Software Development Team Integration Team System Test Team Security Team Usability Team Management Team Responsibilities: The management team has the following responsibilities: Use the project test plan to: Staff the test teams. Procure the test tools. Ensure that the tasks of the testing activity are being performed: Ensure that the Project Test Plan is produced, maintained, and followed. Ensure that the Master Test List is produced and maintained. Ensure that Summary Test Reports are regularly produced and distributed. Use the test summary report to schedule the testing activity. Composition: The management team has the following composition: TBD Requirements Team The requirements team has the following responsibilities: Perform requirements engineering. Perform model testing on the use case model and the domain object model. Composition: The requirements team has the following composition: TBD Software Development Team The software development team has the following responsibilities: Design the software components. Perform design testing on the design object model. Develop or acquire the software components. Perform unit testing on the software components. Integrate the software components. Perform software integration on the integrated software components. Generate unit and software integration: Test procedures Test suites of test cases Test reports Correct the defects found as a result of testing. Composition: The software development team has the following composition: TBD Integration Team The integration team has the following responsibilities: Integrate the software components. Perform regression testing using the unit test suites produced by the software development team. Integrate the system components. Perform regression testing using a subset of the system functional test suites produced by the system test team. Generate: Test procedures Test suites of test cases Test reports Composition: The integration team has the following composition: Integrator System Test Team The system test team has the following responsibilities: Perform system testing including: Functional testing. Performance testing. Load testing. Stress testing. Robustness testing. Portability testing. Perform: Test planning Test reuse Test design Test implementation Test execution Test reporting Review and provide input to the: System test plan Master test list Summary test reports Generate system-level: Test procedures Test suites of test cases Test reports Composition: The system test team has the following composition: System tester Security Team The security team has the following responsibilities: Perform security testing including: Test planning Test reuse Test design Test implementation Test execution Test reporting Review and provide input to the: System test plan Master test list Summary test reports Generate security: Test procedures Test suites of test cases Test reports Composition: The security team has the following composition: Security Engineer Usability Team The usability team has the following responsibilities: Perform usability testing. Review and provide input to the: System test plan Master test list Summary test reports Generate usability: Test procedures Test suites of test cases Test reports Composition: The usability team has the following composition: TBD Testing Tasks This section of the project test plan documents the following general tasks comprising the testing activity: Test planning Test reuse Test design Test implementation Test execution Test reporting Test evaluation Test Planning Test planning is the task of planning the testing activity and the kinds of testing that will take place on a project. Test planning includes the following subtasks: Determine the project test goals and objectives. Identify the test work products to be produced. Identify the test teams to perform the other testing tasks (e.g., produce these test work products). Identify the general testing tasks to be performed. Determine the different kinds of testing to be performed. For each kind of testing: Define the tests Set the objectives of the tests Identify the preconditions for the tests Set the completion criteria for the tests Assign the test responsibilities to the testing teams Determine the associated testing tasks and testing techniques Determine the test tools to be used Determine the timing of the tests in terms of phases and major milestones Document the test plans in the project test plan document. Test Reuse Test reuse is the task of reusing test work products on the project. Test reuse includes the following subtasks: Identify reusable test work products (e.g., test suites of cases). Modify reusable test work products (if necessary) for use on the project. Reuse the test work products. Test Design Test design is the task of designing test suites of test cases. Test design includes the following subtasks: Identify the test suites. Determine the objectives of each test suite. Develop the test procedures (for manual tests) Determine the test cases necessary to meet the objectives (e.g., completion criteria) of the test suite. For each test case, determine and document in the associated test procedure the: Test preconditions (e.g., object state, software component state, and database contents). Test steps. Test postconditions (e.g., returned value, object state, database state, exceptions thrown or handled, and messages sent). Test schedule (for system tests) Required resources (for system tests) Test Implementation Test implementation is the task of actually programming test suites of test cases and creating associated test data. Test implementation includes the following subtasks: Code the test scripts. Code the test suites of test cases. Create the test data (e.g., content). Debug the test scripts, test suites, test cases, and test data. Test Execution Test execution is the task of running the test suites of test cases. Where practical, test execution should be automated in order to promote test productivity and enable regression testing. Test execution includes the following subtasks: Execute the test scripts, which execute the associated test suites of the test cases. Test Reporting Test reporting is the task of reporting the results of test execution to the relevant stakeholders. Where practical, test reports should be automatically generated and distributed by the test tools. Test reporting includes the following subtasks: Document the test results in the test report. Distribute the test results to the relevant stakeholders. Summarize the test results in the test summary report. Test Evaluation Test evaluation is the task of evaluating the test work products and the performance of the testing tasks. Test evaluation includes the following subtasks: Perform informal peer level evaluation. Perform formal evaluation via inspection Perform quality control inspection. Test Tools This section of the project test plan documents the testing tools used to perform testing: Record and Playback Tools Webpage Link Evaluation Tools Code Coverage Tools Test Data Generators Defect Reporting and Tracking Tools Organizational Performance Testing Definition Organizational performance testing is the testing of TBD. Objectives The objectives of organizational performance testing are to: TBD Preconditions The preconditions of organization performance testing are: TBD Completion criteria Organizational performance testing is complete when: TBD Responsibilities The TBD team performs organizational performance testing. Tasks and associated techniques TBD Tools The following kinds of tools are used to perform organizational performance testing: TBD Timing TBD Model Testing Definition Model testing is the testing of a model to identify defects before the model is implemented. Model testing involves the execution of the model using the wetware computer (i.e., human brain), either on a whiteboard or using an upperCASE tool. Objectives The objectives of model testing are: To ensure that use case models and object models are consistent (e.g., that class diagrams are consistent with relevant sequence diagrams). To save effort and time by identifying and correcting defects before the model is implemented. Preconditions The preconditions of model testing are: Use case model testing: Use case paths have been specified. Domain model class diagrams exist. Design object model testing: Use case paths have been specified. Design model class diagrams exist. State transition diagrams exist Completion criteria Model testing is complete when: Use case model testing: Every normal path through a use case has been checked against the relevant domain model class diagrams (e.g., the necessary domain model types exist, and they are connected by associations that allow them to collaborate to provide the behavior of the use case path). At least one exceptional path (if any) through each use case has also been checked against the relevant domain model class diagrams). Design object model testing: Every normal path through a use case has been checked against the relevant design model class diagrams (e.g., the necessary design model classes or software components exist, and they are connected by associations that allow them to collaborate to provide the behavior of the use case path). At least one exceptional path (if any) through each use case has also been checked against the relevant design model class diagrams). Class interfaces have been checked against the associated state transition diagrams. Responsibilities The requirements team performs model testing of the use case model and domain object model. The design team performs model testing of the design object model. Tasks and associated techniques Model testing includes the following tasks: Test reuse Test design Test execution Tools Model testing involves use of the following tools: Whiteboards Timing by Phase Conceive Model testing is only relevant if use case models and domain object models are produced to model the business. Architect Model testing of the use case model, domain object model, and the design object model is started. Build Model testing of the use case model, domain object model, and the design object model is completed. Value Not applicable Unit Testing Definition Unit testing is the testing of an individual unit of software, typically by its developer or a peer programmer. Objectives The objectives of unit testing are to: Enable the identification of unit-level defects by causing corresponding failures to occur. Help identify defects that are not easily identified during other forms of testing. Enable the developer to confidently iterate and refactor the software unit, knowing that any defect will be quickly discovered during regression testing. Enable the developer to determine if the unit: Is complete. Fulfills its responsibilities and does not violate its assertions (the oracles for unit testing). Works as designed. Is ready to be submitted for integration as part of a code build. Preconditions The preconditions of unit testing are: The unit responsibilities have been determined. The units assertions (i.e., class invariant, method preconditions and postconditions) have been specified. The unit interface has been designed. Completion criteria Unit testing is complete when: A complete test suite of test cases exists for every: Public software unit. Domain class and interface. Object-oriented classes. Unit testing is complete if test cases exist for: Blackbox testing. Every public operation of the classes under test that is not a pure getter or setter has been tested. This includes retesting inherited operations as well as overridden operations. This includes testing the return value, the outbound exceptions to be raised, the state of the object under test, the messages to be sent, and the inbound exceptions to be handled. Every constructor has been tested. This includes parameterized classes. Every responsibility has been tested. Every assertion has been tested. State based testing: Every state transition from every logical state of the classes under test. Every state of every message parameter. Every state of every collaborator. Every state of every exception that can be thrown to the classes under test. Structural testing: Every basis path through the classes under test has been tested. Every attribute has been accessed (read and write if not constant) Procedural functions. Unit testing is complete if: A test suite containing test cases exists for: Every state of every parameter. Every basis path through the function has been tested. HTML webpages. Unit testing is complete if: Tag testing (e.g., using WebLint) is performed to determine if the HTML is syntactically correct. A test suite containing test cases exists for: Every link (to determine if the link is broken). Every parameter state of every input parameter. Every client side code fragment (e.g., buttons that only affect the client and do not communicate with the server). These test suites execute properly. No failures are reported. Responsibilities The software development team performs unit testing: The class developer may test his/her own class. A peer of the class developer (pair programming) may test the class. Tasks and associated techniques Unit testing involves the following tasks and techniques: Test Design: Assertions and exceptions Boundary-value testing State based testing (e.g., transition tree) Structured testing Error guessing Decision tables Abstract class via concrete test subclass Interface via concrete class Dependency-based testing Parallel test driver hierarchy Embedded test suite Test subclasses Peer testing Test Implementation: Test first Peer testing Test Execution: Automated testing Regression testing Peer testing Test Reporting: Automatic report generation (by tool) Tools Unit testing involves use of the following tools: Junit WebLint Timing by Phase Conceive Not Applicable Architect Started. Build Completed. Value Not applicable Commercial Component Testing Definition Commercial component testing is the testing of commercial off the shelf (COTS) packages. Objectives The objectives of commercial component testing are to support the package selection task by: Identify defects in the commercial component that can cause the application to fail to meet its requirements. Identify limitations (e.g., performance, security, scalability, and operational availability) in a package. Preconditions The preconditions of commercial component testing are: The commercial component has been obtained (either bought or an evaluation copy) and installed on the chosen test environment. The software development team is adequately staffed. Completion criteria Commercial component testing is complete when: Blackbox functional testing has been completed. Blackbox testing of quality (e.g., performance, security, scalability, and operational availability) requirements. Tasks and Responsibilities Commercial component testing involves the software development team performing the following testing tasks: Test Design Test Implementation Test Execution Test Reporting Test Evaluation Tools The following kinds of tools are used to perform commercial component testing: Test harness Timing by Phase Conceive Possibly performed as part of vendor and product evaluation during the production of the technology strategy. Architect Probably performed as part of vendor and product evaluation. Build Not applicable unless a vendor's product proves unworkable, and vendor and product evaluation must be repeated. Value Not applicable Integration Testing Definition Integration testing is the testing of a partially integrated application to identify defects involving the interaction of collaborating components. This section of the project test plan documents the following kinds of integration testing: Commercial Component Integration Testing Software Integration Testing System Integration Testing The objectives of integration testing are to: Determine if components will work properly together. Identify defects that are not easily identified during unit testing. Commercial Component Integration Testing Definition Commercial component integration testing is the integration testing of multiple commercial-off-the-shelf (COTS) packages to determine if they are not interoperable (i.e., if they contain interface defects). Objectives The objectives of commercial component integration testing are to: Determine if the commercial components will work properly together. Identify any defects in how the packages integrate that will require the development of additional glue code. Preconditions The preconditions of commercial component integration testing are: The commercial components have been obtained (either bought or an evaluation copy) and installed on the chosen test environment. The software development team is adequately staffed. Completion criteria Commercial component integration testing is complete when: Blackbox functional testing that exercises the major interfaces between the components has been completed. Tasks and Responsibilities Commercial component testing involves the integration team performing the following testing tasks: Test Design Test Implementation Test Execution Test Reporting Test Evaluation Tools The following kinds of tools are used to commercial component package integration testing: Test harness Timing by Phase Conceive Possibly performed as part of vendor and product evaluation during the production of the technology strategy. Architect Probably performed as part of vendor and product evaluation. Build Not applicable unless a vendor's product proves unworkable, and vendor and product evaluation must be repeated. Value Not applicable Software Integration Testing Definition Software integration testing is the incremental testing of two or more integrated software components to produce failures caused by interface defects. Objectives Software integration testing has the following objectives: To cause failures involving the interactions of the integrated software components when running on a single platform. To report these failures to the development team so that the underlying defects can be identified and fixed. To help the development team to stabilize the software so that it can be successfully distributed prior to system testing. To minimize the number of low-level defects that will prevent effective system testing. Preconditions The preconditions of system integration testing are: Unit testing has occurred. All software components have been integrated. Completion criteria Software integration testing is complete when: A test suite of test cases exists for each interface between software components. All software integration test suites successfully execute (i.e., the tests completely execute and the actual test results match the expected test results). Responsibilities The integration team performs software integration testing. Tasks and associated techniques Software integration testing involves the following tasks and techniques: Test Design: Hierarchical Incremental Testing (HIT) Polymorphism Testing Test Execution: Regression Testing Tools TBD Timing TBD System Integration Testing Definition System integration testing is the testing of integrated system components. Specifically, software components are distributed onto multiple platforms (e.g., client, web server, application server, and database server) to produce failures caused by system integration defects (i.e., defects involving distribution and back-office integration). Objectives The objective of system integration testing is to: Cause failures involving the distribution of integrated system components. Cause failures involving integration of the system with other applications (e.g., legacy back-office systems). Report these failures to the development team so that they can be fixed. Help the development team to ensure that the software has been successfully distributed to improve the effectiveness of system and launch testing. Minimize the number of low-level defects that will prevent effective system and launch testing. Preconditions The preconditions of system integration testing are: All distributed software components have passed software integration testing. All system components have been integrated. Completion criteria System integration testing is complete when: All system components have been integrated (e.g., software components have been distributed to their hardware components). A system integration test suite of test cases exists for each interface between software components on different hardware components. All system integration test suites successfully execute (i.e., the tests completely execute and the actual test results match the expected test results). Responsibilities The integration team performs system integration testing. Tasks and associated techniques Distribute the software components on to the hardware components. Integrate the system with other external systems. Execute the system integration test suites. Report the results of the system integration test suites. Tools TBD Timing TBD System Testing Definition System testing is the testing of an integrated, blackbox application against its requirements during the construction phase. System testing is thus a requirements validation technique. Objectives The objectives of system testing are to: Validate the application (i.e., determine if it fulfills its requirements). Identify defects that are not efficiently identified during unit and integration testing. Determine the extent to which the application is ready for launch. Provide project status metrics (e.g., percentage of use case paths successfully tested). This section of the project test plan documents the following kinds of system testing: Functional Testing, which is the testing of the application against its operational requirements. Performance Testing, which is the testing of the application against its performance requirements. Load Testing, which is the testing of the application that attempts to cause failures involving how the performance of a system varies under normal conditions of utilization. Stress Testing, which is the testing of the application that attempts to cause failures involving how the system behaves under extreme but valid conditions (e.g., extreme utilization, insufficient memory inadequate hardware, and dependency on over-utilized shared resources). Robustness Testing, which is the testing of the application that attempts to cause failures involving how the system behaves under invalid conditions (e.g., unavailability of dependent applications, hardware failure, and invalid input such as entry of more than the maximum amount of data in a field). Configuration Testing, which is the testing of the application under different hardware and software configurations to determine an optimal system configuration. Contention Testing, which is the testing of the application that attempts to cause failures involving concurrency. Availability Testing, which is the testing of the application against its operational availability requirements. Portability Testing, which is the testing of the application against its portability requirements. Security Testing, which is the testing of the application against its security requirements. Usability Testing, which is the testing of the application against its usability requirements. Functional Testing Definition Functional testing is the testing of an integrated application against its operational (i.e., functional) requirements. Objectives The objectives of functional testing are to: Partially validate the system (i.e., to determine if it fulfills its operational requirements). Cause failures concerning the operational requirements that help identify defects that are not efficiently found during unit and integration testing. Report these failures to the development teams so that the associated defects can be fixed. Help determine the extent to which the system is ready for launch. Help provide project status metrics (e.g., percentage of use case paths successfully tested). Provide input to the defect trend analysis effort. Preconditions The preconditions of functional testing are: The operational requirements to be tested have been specified and fully implemented. The relevant software components have passed unit testing. Software integration has occurred (i.e., functional testing can begin prior to the distribution of the software components onto the hardware components). The relevant system components have passed system integration testing. The system test team is adequately staffed. The test environment is ready. Completion Criteria Functional testing is complete when: At least one test case exists for each normal use case path. Test suites for every scheduled use case execute successfully. Teams and Tasks Functional testing involves the following producers performing the following testing tasks: System Test Team performs: Test Planning Test Reuse Test Design: Use case based testing Test Implementation: Record and Playback Test Execution: Record and playback Test Evaluation Test Reporting Test Management Environments Functional testing is performed on the test environment. Tools The following tools are used during functional testing: Test Harness Use Case Tool Record and Playback Tool Phases Functional testing involves the following tasks being performed during the following phases: Strategy: No functional testing. Initiation: Test Planning Test Reuse Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Construction: Test Reuse Test Design Test Implementation New and regression Test Execution Test Evaluation Test Reporting Test Management Transition: Regression Test Execution Test Reporting Test Management Maintenance: Regression Test Execution Test Reporting Test Management Performance Testing Definition Performance testing is the testing of the system that attempts to cause failures to meet performance requirements under normal operating circumstances in order to identify inefficiencies and bottlenecks. Objectives The objectives of performance testing are to: Partially validate the system (i.e., to determine if it fulfills its performance requirements). Cause failures relating to performance requirements: Capacity (the minimum number of objects the system can handle). Latency (the maximum time to complete a system operation). Response time (the maximum system response times). Throughput (the maximum average system operation time). Report these failures to the development teams so that the associated defects can be fixed. Provide information that will assist in the optimization of performance (e.g., by helping identify performance bottlenecks). Reduce hardware costs by providing information allowing systems engineers to: Identify the minimum hardware necessary to meet the performance requirements. Tune the application for maximum performance by identifying the optimal system configuration (by repeating the test using different configurations). Help determine the extent to which the system is ready for launch. Provide input to the defect trend analysis effort. Preconditions The preconditions of performance testing are: The performance requirements to be tested have been specified. The relevant software components have been implemented and have passed unit testing. Software integration has occurred. Performance testing can begin prior to the distribution of the software components onto the hardware components to identify gross performance defects. However, performance testing must be repeated after system integration. The relevant system components have passed system integration testing. The system test team is adequately staffed. The test environment is ready. Completion Criteria Performance testing is complete when: A test suite of test cases for every performance requirement exists. The test suites for every scheduled performance requirement execute successfully Teams and Tasks Performance testing involves the following producers performing the following testing tasks using the following techniques: System Test Team performs: Test Planning Test Reuse Test Design: Use case based testing Test Implementation: Record and Playback Test Execution: Record and playback Test Evaluation Test Reporting Test Management Environments Performance testing is performed on the test environment. Tools The following tools are used during performance testing: Test Harness Use Case Tool Record and Playback Tool Phases Performance testing involves the following tasks being performed during the following phases: Strategy: No performance testing except potentially some performance testing of COTS software components during vendor and product evaluation. Initiation: Test Planning Test Reuse Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Construction: Test Reuse Test Design Test Implementation New and regression Test Execution Test Evaluation Test Reporting Test Management Transition: Regression Test Execution Test Reporting Test Management Maintenance: Regression Test Execution Test Reporting Load Testing Definition Load testing is the testing of an application that attempts to cause failures involving how its performance varies under normal conditions of utilization (e.g., as the load increases and becomes heavy). Objectives The objectives of load testing are to: Partially validate the application (i.e., to determine if it fulfills its scalability requirements): Scalability requirements (e.g., the number of users increases) Distribution and load-balancing mechanisms Cause failures concerning the load requirements that help identify defects that are not efficiently found during unit and integration testing. Report these failures to the development teams so that the associated defects can be fixed. Determine if the application will support typical production load conditions. Identify the point at which the load becomes so great that the application fails to meet performance requirements. Help determine the extent to which the application is ready for launch. Provide input to the defect trend analysis effort. Preconditions The preconditions of load testing are: A test suite of test cases for every scalability requirement exists. The scalability requirements to be tested have been specified. The relevant software components have passed unit testing. Software integration has occurred (i.e., load testing can begin prior to the distribution of the software components onto the hardware components). The relevant system components have passed system integration testing. The system test team is adequately staffed. The test environment is ready. Completion Criteria Load testing is complete when: At least one load test suite exists for each scalability requirement. The test suites for every scheduled scalability requirement execute successfully. Tasks Load testing involves the system test team performing the following testing tasks using the following techniques: Test Planning Test Reuse Test Design: Use Case Based Testing Workload analysis to determine the typical production workloads. Test Implementation: Develop test scripts simulating real-life workloads. Test Execution: Regression Testing Profiling. Test Evaluation Test Reporting Test Management Environments Performance testing is performed on the test environment. Tools The following tools are used during performance testing: Test Harness Use Case Tool Record and Playback Tool Phases Performance testing involves the following tasks being performed during the following phases: Strategy: No performance testing except potentially some performance testing of COTS software components during vendor and product evaluation. Initiation: Test Planning Test Reuse Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Construction: Test Reuse Test Design Test Implementation New and regression Test Execution Test Evaluation Test Reporting Test Management Transition: Regression Test Execution Test Reporting Test Management Maintenance: Regression Test Execution Test Reporting Test Management Stress Testing Definition Stress testing is the testing of the application that attempts to cause failures involving how its performance varies under extreme but valid conditions (e.g., extreme utilization, insufficient memory inadequate hardware, and dependency on over-utilized shared resources). Objectives The objectives of stress testing are to: Partially validate the application (i.e., to determine if it fulfills its scalability requirements). Determine how an application degrades and eventually fails, as conditions become extreme. For example, stress testing could involve an extreme number of simultaneous users, extreme numbers of transactions, queries that return the entire contents of a database, queries with an extreme number of restrictions, or an entry at the maximum amount of data in a field. Report these failures to the development teams so that the associated defects can be fixed. Determine if the application will support "worst case" production load conditions. Provide data that will assist systems engineers in making intelligent decisions regarding future scaling needs. Help determine the extent to which the application is ready for launch. Provide input to the defect trend analysis effort. Preconditions Stress test execution can begin when the following preconditions hold: The scalability requirements to be tested have been specified. The relevant software components have passed unit testing. Software integration has occurred (i.e., load testing can begin prior to the distribution of the software components onto the hardware components). The relevant system components have passed system integration testing. The system test team is adequately staffed. The test environment is ready. Completion Criteria Stress testing is complete when: A test suite of test cases for every exceptional requirement exists. Each stress test suite executes successfully. Tasks Stress testing involves the system test team performing the following testing tasks using the following techniques: Test Planning Test Reuse Test Design: Use Case Based Testing Workload analysis to determine the maximum production workloads. Test Implementation: Develop test scripts simulating real-life workloads. Test Execution: Regression Testing Profiling. Test Evaluation Test Reporting Test Management Environments Stress testing is performed on the test environment. Tools The following tools are used during stress testing: Test Harness Use Case Tool Record and Playback Tool Phases Stress testing involves the following tasks being performed during the following phases: Strategy: No stress testing except potentially some stress testing of COTS software components during vendor and product evaluation. Initiation: Test Planning Test Reuse Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Construction: Test Reuse Test Design Test Implementation New and regression Test Execution Test Evaluation Test Reporting Test Management Transition: Regression Test Execution Test Reporting Test Management Maintenance: Regression Test Execution Test Reporting Test Management Robustness Testing Definition Robustness testing is the testing of an application that attempts to cause failures involving how it behaves under invalid conditions (e.g., unavailability of dependent applications, hardware failure, and invalid input such as the entry of more than the maximum amount of data in a field). Objectives The objectives of robustness testing are to: Partially validate the application (i.e., to determine if it fulfills its robustness requirements): Operational requirements specified by exceptional use case paths. Operational availability requirements. Reliability requirements. Robustness requirements. Determine how well the application handles failures and performs rollover to help ensure that it will not fail catastrophically when the unexpected occurs. Cause failures concerning the exceptional operational requirements and robustness requirements that help identify defects that may not have been found during unit and integration testing. Report these failures to the development teams so that the associated defects can be fixed. Help determine the extent to which the application is ready for launch. Provide input to the defect trend analysis effort. Preconditions The preconditions of robustness testing are: The exceptional operational requirements and the robustness requirements to be tested have been specified. The relevant software components have passed unit testing. Software integration has occurred (i.e., robustness testing can begin prior to the distribution of the software components onto the hardware components). The relevant system components have passed system integration testing. The system test team is adequately staffed. The test environment is ready. Completion Criteria Robustness testing is complete when a test suite of test cases for every important, probable system failure mode exists: At least one test case exists for each exceptional use case path. At least one test case exists for each hardware failure. At least one test case exists for each COTS component failure. Responsibilities The system test team performs robustness testing. Tasks and associated techniques Perform exception testing: Develop one or more test cases for each exceptional path through a use case. Perform hardware failover testing: Disable commercial power supply to determine if Uninterruptible Power Supply (UPS) takes over until the generator supplies power. Disable primary UPS to determine if the secondary takes over. Disable primary Power Distribution Unit (PDU) to determine if the secondary takes over. Disable a primary network to determine if the secondary takes over. Remove workstation(s) from network to determine if the others take over. Remove primary router from network to determine if the secondary takes over. Remove primary firewall from network to determine if the secondary takes over. Remove primary load balancer from network to determine if the secondary takes over. Remove one switch from network to determine if the others take over. Remove web server(s) from network to determine if load balancing and failover occurs. Remove application server(s) from network to determine if load balancing and failover occurs. Remove database server(s) from network to determine if load balancing and failover occurs. Remove disk/tape library from network to determine if load balancing and failover occurs. Perform Internet Service Provider failover testing: Remove each ISP individually from the network to determine if the secondary takes over. Perform package failure testing: Force each package to fail. Force each database to fail. Tools TBD Timing PhaseConceiveArchitectBuildValue Testing TaskPlanningStartedCompletedMaintainedMaintainedReuseN/AStartedCompletedMaintainedDesignN/AStartedCompletedMaintainedImplementationN/AN/ACompletedMaintainedExecutionN/AN/ACompletedMaintainedReportingN/AN/ACompletedMaintainedEvaluationN/AN/ACompletedMaintainedManagementPerformedPerformedPerformedPerformed Configuration Testing Definition Configuration testing is the testing of different variations of the application against the configurability requirements. Objectives The objectives of configuration testing are to: Cause failures involving configurability requirements: Functional Variants Internationalization (e.g., language, currency, tax and tariff, etc.) Personalization Determine the effect of adding or modifying hardware resources such as: Memory Disk resources Processors Load balancers Determine an optimal system configuration Preconditions The preconditions of contention testing are: Variants exist. Completion criteria Configuration testing is complete when: At least one configuration test suite exists for each configurability requirement. The test suites for every scheduled configurability requirement execute successfully on the appropriate configuration. Responsibilities The system test team performs configuration testing. Tasks and associated techniques TBD Tools TBD Timing Planning for configuration testing occurs during the Conceive Phase. Reuse, design, implementation, execution, evaluation, reporting, and management of configuration testing occur during the Build phase. Regression configuration testing occurs during the Build and Launch phases. Contention Testing Definition Contention testing is the testing of an integrated application that attempts to cause failures involving actual or simulated concurrency. Objectives The objectives of contention testing are to: Cause failures involving concurrency (e.g., excessive locking, deadlock, livelock, starvation, race conditions, priority inversion, and lack of thread safety in shared software components or data). Report these failures to the development teams so that the associated defects can be fixed. Preconditions The preconditions of contention testing are: The relevant software components have passed unit testing. Software integration has occurred (i.e., contention testing can begin prior to the distribution of the software components onto the hardware components). The relevant system components have passed system integration testing. The system test team is adequately staffed. The functional test environment is ready. Completion criteria Contention testing is complete when: Test cases exist for TBD. These test cases execute without eliciting system failure. Responsibilities The system test team performs contention testing. Tasks and associated techniques TBD Tools TBD Timing Planning for contention testing occurs during the Conceive Phase. Reuse, design, implementation, execution, evaluation, reporting, and management of availability testing occur during the Build phase. Regression contention testing occurs during the Build and Launch phases. Availability Testing Some systems require high availability, which only allows a miniscule amount of downtime. Other systems require continuous availability, which doesnt allow for any scheduled downtime; the system must be available for users even while its hardware is replaced or its software is upgraded. Definition Availability testing is the testing of an integrated application against its operational availability requirements. Objectives The objectives of availability testing are to: Determine: If the application meets its operational availability requirements. For example, availability testing can cause failures due to memory leaks or database indexing defects. How stable the application is. Whether downtime is required for maintenance purposes. Report this information to the development teams so that any associated defects can be fixed. Provide information that will assist in the optimization of operational availability. Preconditions The preconditions of robustness testing are: Operational availability requirements have been specified. System integration has occurred. Completion criteria Availability testing is complete when: Test cases exist for each operational availability requirement. These test cases execute without eliciting system failure. Responsibilities The system test team performs availability testing. Tasks and associated techniques Place hardware temporarily offline. Upgrade a software component. Perform availability testing for several days. Tools TBD Timing Planning for availability testing occurs during the Conceive Phase. Reuse, design, implementation, execution, evaluation, reporting, and management of availability testing occur after functional testing during the Build phase. Regression availability testing occurs during the Build and Launch phases. Portability Testing Definition Portability testing is the testing of an integrated [partial] system against its portability requirements. Objectives The objectives of portability testing are to: Cause failures involving portability requirements. Determine if the system can be ported to each of its required environments: Hardware ram and disk space Hardware processor and processor speed Monitor resolution Operating system make and version Browser make and version Determine if the look and feel of the webpages is similar and functional in the various browser types and their versions. Report these failures to the development teams so that the associated defects can be fixed. Preconditions The preconditions of portability testing are: Portability requirements have been specified. System integration has occurred. Completion criteria Portability testing is complete when: The system has been ported to each required environment. Functional testing has been repeated on each required environment. Responsibilities The system test team performs portability testing. Tasks and associated techniques Port the software to each environment specified in the requirements specification. Rerun the functional tests. Rerun the usability tests involving webpage look and feel. Tools TBD Timing Planning for portability testing occurs during the Conceive Phase. Reuse, design, implementation, execution, evaluation, reporting, and management of portability testing occur after functional testing during the Build phase. Regression portability testing occurs during the Build and Launch phases. Security Testing Definition Security testing is the testing of a [partially] integrated system to determine if it contains any defects that negatively impact its security. Security tests may test software or completed systems. Security tests may be either automated using a security tool or performed manually (e.g., tests of physical security). Objectives The objectives of security testing are to: Cause failures involving security: Requirements: Identification Authentication Authorization Content Protection Integrity Intrusion Detection Nonrepudiation Privacy System Maintenance Techniques: Encryption and Decryption Firewalls Personnel Security Passwords Digital Signatures Personal background checks Physical Security: Locked doors Cameras Scopes: Internet Clients Intranet Clients Networks Data center with servers Report these failures to the development teams so that the associated defects can be fixed. For example, security testing will detect the following security failures: The system fails to identify and authenticate a user. The system allows a user to perform an unauthorized function. The system fails to protect its content against unauthorized usage. The system allows the integrity of data or messages to be violated. The system allows undetected intrusion. The system fails to ensure privacy by using an inadequate encryption technique. Preconditions The preconditions of security testing are: Security requirements have been specified. Security mechanisms have been implemented. Production environment has been developed (physical security and hardware security testing). Software integration has occurred (software security). Completion criteria Security testing is complete prior to launch when: Physical security is determined to be effective. Major software components and their code have been reviewed for security vulnerabilities, and fixes have been made. The hardware architecture has been reviewed for security vulnerabilities, and fixes have been made. Ports have been scanned, fixes have been made, and a final scan reveals no security vulnerabilities. Network traffic has been analyzed, fixes have been made, and a final analysis reveals no security vulnerabilities. Responsibilities The security team performs security testing. Tasks and associated techniques Test physical security. Perform security review of major software components and their code. Perform security review of hardware architecture (production environment) for hardware placement, network addressing and segment, and application distribution. Scan ports to identify host-level and network-level vulnerabilities. Analyze traffic to identify IP information about potential attackers. Fix security holes. Apply batch scripts to perform registry fixes. Tools The following kinds of tools will be used to perform security testing: Port scanners (e.g., ISS Internet Security Scanner and ISS DB Scanner) are used to identify open (listening) ports to identify vulnerabilities, which are either host-level (OS-specific) or network-level (routers, firewalls, and service/shares on networked machines). Traffic analyzers (e.g., Etherpeek) are used to capture packets moving in and out of specific network cards to identify IP information about potential attackers. Timing The first round of security testing occurs after the hardware components have been networked, and the hardware architecture has been reviewed for security considerations. The second round of security testing occurs after both the software components and the content have been distributed on the hardware components. The third round of security testing as part of the acceptance testing prior to launch. Security testing should be performed on a periodic basis thereafter. Usability Testing Definition Usability testing is the testing of a [partially] integrated system to determine if it contains any usability defects. Usability testing is done with representative users either in a usability lab or performed in the setting of use. Objectives The objectives of usability testing are to: Identify aspects of the systems user interface that: Fail to meet its quantitative or qualitative usability requirements: User orientation and navigation within the site Information consistency and presentation Appropriate use of language and metaphors Overall site functionality and streamlined processes Visual design feedback Should be iterated to make them more usable. Determine how well the user interfaces enable the different categories of users to perform their business tasks. Report these failures to the development teams so that the associated defects can be fixed. Preconditions The preconditions of usability testing are: Usability requirements have been specified. A working wireframe, prototype, or site exists to be tested. Users are available. Completion criteria Usability testing is complete when: The usability team has tested the user interface: All links navigate to the proper webpages. All user inputs produce proper results. Webpages are consistent with branding guidelines. Screens open and close properly. A group of representative users from each type of user have been identified and have evaluated the user interfaces. The results of the evaluation have been analyzed. Associated test reports with recommendations have been presented back to the development team. Responsibilities The usability team performs usability testing. A select group of users use the application under controlled circumstances. Tasks and associated techniques The usability team develops a test procedure for usability testing that will detail the participants and the techniques. The usability team determines the feasible number and types of users to test the user interface. The usability team prepares the environment, equipment, and user interfaces (e.g., wireframe prototypes). The usability team recruits the users to participate in usability testing, if possible from the actual target user communities. For each user type, the usability team identifies the subset of use case paths (e.g., the most critical, the most frequently exercised, or important exceptional paths) to test. The usability team develops the corresponding test cases. The usability team dry runs the test cases. The usability team observes the users executing the usability tests. The usability team collects test results and usability data. The usability team holds a debriefing session with the users. The usability team analyzes the usability test results. The usability team prepares a test report including recommendations. The usability team delivers the test report to the development team so that they can fix the defects. Tools The following kinds of tools are used to perform usability testing: A protocol of tasks and preference questions that the participants will run through. Equipment including, but not limited to: Computer and monitor Video camera(s) Video mixer Tape recorder(s) Screen capturing tool (e.g., Lotus ScreenCam) User action capturing tool (e.g., Winrunner) Scan converter [TBD????] Timing During a typical 6-8 week Architect phase, usability testing should be done twice, complemented by other evaluation methods. During a shorter Architect phase (less than 4 weeks), usability testing should be done on the wireframe or prototype when it is relatively complete; however, time should be left for changes to be implemented. During a Build phase, usability testing can also be done on a full prototype after the design effort is relatively complete. Launch Testing Definition Launch testing is the testing of the completed system in the production environment during the launch phase. Objectives The objectives of launch testing are to: Determine the extent to which the system is ready for deployment to the user community. Determine if the system is acceptable to the customer. This section of the project test plan documents the following kinds of launch testing: Alpha Testing, which is the initial development organization internal dry run of the acceptance tests of the application in the production environment. Beta Testing, which is the testing of the application in the production environment by a few key users prior to acceptance testing and the release of the system to the entire user community. Acceptance Testing, which is the formal customer-observed testing to determine if the system is acceptable to its customer. Alpha Testing Definition Alpha testing is the initial development organization internal dry run of the acceptance tests of the application in the production environment. Objectives The objectives of alpha testing are to: Cause failures that only tend to occur in the production environment. Report these failures to the development teams so that the associated defects can be fixed. Determine if the application is ready for beta testing, acceptance testing, and launch. To provide input to the defect trend analysis effort. Preconditions Execution of alpha testing can begin when the following preconditions hold: The system has passed all system tests. The transition phase has begun. The system test team is adequately staffed. The production environment is ready. The system has been ported to the production environment. Completion Criteria Alpha testing is complete when: An initial version of the acceptance test suites exists. The customer representative has approved these acceptance test suites. The acceptance tests execute on the production environment. Acceptance testing does not discover any: Severity one defects. Severity two defects that do not have adequate work arounds. Teams and Tasks Alpha testing involves the following producers performing the following testing tasks: System Test Team performs: Test Planning: Determine alpha testing completion criteria. Update the alpha testing subsection of System Test Plan. Test Design: Select an adequate subset of the system test suites of test cases (both functional and quality) to be repeated on the production environment during alpha testing. Test Implementation: Fix any defects in the test suites found during the evaluation. Test Execution: Execute the alpha test suites on the production environment. Test Evaluation: Inspect the initial alpha test suites of test cases. Test Reporting: Report failures that occurred during testing to the development teams so that the associated defects can be fixed. Test Management: Place alpha test work products under configuration control. Environments Alpha testing is performed on the production environment. Tools The following kinds of tools are used to perform alpha testing: System test harness Use case tool Record and playback tool Phases Alpha testing involves the following tasks being performed during the following phases: Strategy: No alpha testing. Initiation: Test Planning Construction: No acceptance testing. Transition: Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Maintenance: No alpha testing. Beta Testing Definition Beta testing is the testing of the application in the production environment by a few key users prior to acceptance testing and its release to the entire user community. Objectives The objectives of beta testing are to: Cause failures that only tend to occur during actual usage by the user community rather than during formal testing. Report these failures to the development teams so that the associated defects can be fixed. Obtain additional user community feedback beyond that received during usability testing. Help determine the extent to which the system is ready for: Acceptance testing. Launch. Provide input to the defect trend analysis effort. Preconditions Beta test execution can begin when the following preconditions hold: The system has passed all system tests. The transition phase has begun. The system has passed alpha testing. The production environment is ready. The system has been ported to the production environment. The selected group of users is ready. Completion criteria Beta testing is complete when: The time period scheduled for beta testing ends. The users have reported any failures observed to the development organization. These failures have been passed on to the development teams. Teams and Tasks Alpha testing involves the following producers performing the following testing tasks: System Test Team: Test Planning - Update beta testing subsection of System Test Plan. Test Evaluation - Evaluate effectiveness and adequacy of beta testing. Customer Organization: Test Implementation - Select beta test user group. Test Reporting - Pass on reported failures to developer organization. User Organizations: Test Execution - Use system under normal conditions of operation. Test Reporting - Report failures to customer organization. Environments Beta testing is performed on the production environment, onsite at a select group of users. Tools The following kinds of tools are used to perform beta testing: None Phases Beta testing involves the following tasks being performed during the following phases: Strategy: No beta testing. Initiation: Test Planning Construction: No beta testing. Transition: Test Implementation Test Execution Test Evaluation Test Reporting Maintenance: No beta testing. Acceptance Testing Definition Acceptance testing is the final testing of a system in the production environment to determine if it is acceptable to its customer. Objectives The objectives of acceptance testing are to: Determine whether the application satisfies its acceptance criteria. Enable the customer to determine whether to accept the application. Determine if the application is ready for deployment to the full user community. Report any failures to the development teams so that the associated defects can be fixed. Preconditions Acceptance test execution can begin when the following preconditions hold: All planned system testing is complete. The launch phase is started. Alpha testing is complete. Beta testing is complete. The production environment is ready. The system has been ported to the production environment. The customer is ready to witness the acceptance tests. Completion criteria Acceptance testing is complete when: Test scripts of test cases exist that test the system against its acceptance criteria. Acceptance test suites execute successfully. Acceptance testing does not discover any severity one defects. Acceptance testing does not discover any severity two defects that do not have adequate work arounds. Teams and Tasks Alpha testing involves the following producers performing the following testing tasks: Acceptance testing involves the following producers performing the following testing tasks: System Test Team: Test Planning: Update the acceptance testing subsection of System Test Plan. Test Reuse: Reuse acceptance criteria. Reuse various system test procedures, scripts, and suites as acceptance test procedures, scripts, and suites. Test Design: Define new acceptance criteria for acceptance testing. Design new acceptance test procedures, scripts, suites of test cases, and test data (if necessary). Test Implementation: Implement new test scripts, suites of test cases, and test data (if necessary). Test Execution: Execute acceptance test procedures, scripts, and suites of test cases. Test Evaluation: Evaluate acceptance criteria for acceptance testing. Evaluate acceptance test procedures, scripts, suites of test cases, and test data. Evaluate acceptance test results. Test Reporting: Report failures prioritized by severity to the development team. Test Management Place acceptance testing work products under configuration control. Customer Representative: Test Design: Define acceptance criteria. Test Execution: Observe execution of acceptance testing. Test Evaluation: Evaluate and approve acceptance criteria. Evaluate acceptance test results. Test Reporting: Accept or reject the system based on the acceptance test results. Environments Acceptance testing is performed on the production environment at the customer site. Tools The following kinds of tools are used to perform acceptance testing: System test harness Use case tool Record and playback tool Phases Acceptance testing involves the following tasks being performed during the following phases: Strategy: No acceptance testing. Initiation: Test Planning Test Reuse of acceptance criteria Test Design of acceptance criteria Construction: No acceptance testing. Transition: Test Reuse Test Design Test Implementation Test Execution Test Evaluation Test Reporting Test Management Maintenance: No acceptance testing. Appendices A. Testing Techniques This appendix briefly summarizes the different testing techniques that will be used on the project. A.1 Assertions and Exceptions Discussion: How do we know the expected behavior of an object? The requirements of the application are at much too high of a level of abstraction. A single requirement is implemented by many collaborating classes, and a single class collaborates with others to help implement many requirements. The class comment is still at too high of a level of abstraction, is potentially vague, and is not testable. Even the class responsibilities are not specific enough to be testable. Class method signatures are even more specific, but the operation name, parameter names, and parameter types only give the blackbox interface and syntax of the method, not its intended meaning. Looking at the method body only provides the as coded semantics, which may include defects, but not the intended semantics. Assertions are Boolean expressions that define the expected behavior of an object: Class invariants defined the valid states of a class. Method preconditions define the conditions that must be true prior to the execution of the associated method if the method is to execute successfully. Method postconditions define the conditions that most be true after the successful execution of the associated method. Exceptions are special objects that report the occurrence of error conditions. Assertions can provide an appropriate, testable oracle, defining the expected behavior of a class. They also define when exceptions should be raised (i.e., when assertions are violated), so that classes containing assertions are self-testing and significantly more robust than classes that do not. Assertions and exceptions bypasses encapsulation, thereby making classes more testable by making them more observable. Definition: Assertions and exceptions testing is a black-box unit testing technique for embedding tests within classes. Test Completion Criteria: The associated test completion criteria are to: For every public domain class: Every class invariant is coded. Every method precondition is coded. Every method postcondition is coded. Steps: During assertions and exceptions testing, testers perform the following steps: TBD. Limitations: TBD A.2 Equivalence Set Testing Discussion: Huge, practically infinite number of choices of test values. Exhaustive testing is impossible. Partition state space into regions of possible test values, each point in which should produce qualitatively equivalent behavior. Definition: Equivalence set (a.k.a., equivalence class) testing is a black-box unit testing technique for minimizing the number of test cases by creating a single test case for each equivalence set of type values involved in an interaction. Test Completion Criteria: The associated test completion criteria are to: For each method, create one test case for each equivalence set of relevant type values. Steps: During boundary value testing, testers perform the following steps: Identify all object and data types involved in an interaction. Identify all that have state models. Using the state models and any assertions involving the interaction, decompose the complete state space of an interaction (including input parameters, output values, attributes, etc.) into a set of disjoint regions in a multidimensional space. Each axis of the space represents a single object type involved in the interaction. All combinations of relevant data type values should produce the same qualitative behavior of the object under test. Select a point within the equivalence set of relevant type values. Create a test case from these values Limitations: In order to minimize the number of test cases, this testing technique assumes that all combinations of relevant type values within an equivalence set are equally likely to cause failure. However, this assumption is based on the assumption that the implementation of the class conforms to its specification in that the equivalence sets identified from the specification are actually equivalent as implemented. If the implementation does not match the design as specified (which it will not if a defect exists), then some combinations of values in the equivalence set will produce a different actual behavior than expected by the oracle. Nevertheless, this technique still will elicit failures due to many defects. A.3 Boundary Value Testing Discussion: Boundary testing assumes that defects are more likely to occur where data or objects are discontinuous rather than where data or objects are far from such boundaries. Traditional boundary value testing typically involved either boundaries in one dimensional numerical data types such as integers, floating point numbers, or real numbers or else the end points of enumeration types. With such traditional numerical data types, the negative and positive numbers would be bounded by the following three boundary values: the most negative number, 0, and the most positive number. Boundary value testing would create a test case for each of the following values: The minimum negative number (a boundary value) A negative number near the minimum negative number A typical negative number A slightly negative number near zero Zero (a boundary value) A slightly positive number A typical positive number A positive number near the maximum The maximum positive number (a boundary value) Boundary value testing is greatly impacted by the use of object-orientation. Traditional simple numerical data types are rarely used, except to build more complex object abstractions. It is typically much more difficult to identify the boundary conditions of such objects that may have numerous attributes. Modern boundary value testing typically involves decomposing the complete state space of an interaction (including input parameters, output values, attributes, etc.) into a set of disjoint regions in a multidimensional space, each axis of which represents a single object type involved in the interaction. Instantiating an object that is either on or near a boundary between regions is often nontrivial. Often, unit testers base their decisions on the state models of the classes involved. Definition: Boundary value testing is a gray-box unit testing technique for identifying test cases designed for eliciting failures associated with boundary values. Test Completion Criteria: The associated test completion criteria are to: For each method in the class to be tested: Create a set of test cases for each discontinuity (if any) in each: Input parameter Output value Attribute Collaborator Output value received from a collaborator Exception thrown Exception handled Whereby each set of test cases includes one test case for each of the following conditions: Far from the boundary value on one side Near the boundary on one side On the boundary Near the boundary on the other side Far from the boundary on the other side Steps: During boundary value testing, the tester performs the following steps (in the following decreasing order of importance): Identify all object and data types involved in an interaction. Identify all that have state models. Using the state models and any assertions involving the interaction, identify all relevant boundary values. To identify common failures caused by individual boundaries: Create a test case for each boundary value. Create two test cases near each boundary value, one on each side of the boundary. Create one test case near the middle of each volume bounded by the boundary values. Create a test case simultaneously using as many of the boundary values as is practical to cause rare failures caused by the interaction of multiple boundaries. Limitations: In order to minimize the number of test cases, this testing technique assumes that the boundary values identified from the specification and design are the only boundary values. However, this assumption is based on the assumption that the implementation of the class conforms to its specification and design. If the implementation does not match the design as specified (which it will not if a defect exists), then additional boundaries will exist due to the defects. Nevertheless, this technique still will elicit failures due to many defects. A.4 State Based Testing Discussion: The behavior of objects can be a function of their logical state. The same test stimuli (e.g., message and parameters or exception with attributes) can produce different test results depending on the state of the object under test, thereby increasing the difficulty of regression testing. For example, pushing an item onto a bounded stack behaves differently depending on whether or not the stack is full. Definition: State based testing is a black-box unit testing technique for identifying test cases that identify defects involving the implementation of the state models of classes with logical state. Test Completion Criteria: The associated test completion criteria are to: Test every class with logical states. Cause the corresponding objects under test to make every transition from every state. Steps: During state based testing, the tester performs the following steps: Create a state transition diagram for the class with logical states. Transform the state transition diagram into a transition tree. The root of the tree is the start state, and each branch is made up of a series of transitions between states that terminates when a transition returns the object under test to a previous state. Create a state-based test suite for the class under test, whereby each test case corresponds to a branch of the transition tree. Each test case starts by instantiating an object under test in the start state. The test case then sends the object under test a series of test stimuli (either messages with appropriate parameters or exceptions with appropriate attributes) that are intended to walk the object under test down the corresponding branch of the transition tree. The test oracle is the state transition diagram. Each test case compares each actual post-stimuli state of the object under test with the state that is predicted by the branch of the transition tree that generated the test case. A.5 Structured Testing Discussion: Structured testing is the traditional primary testing technique for procedural units of software and for logically complex methods of object-oriented software. This technique used to be more important when procedures contained large amounts of branching and looping; the advent of inheritance, polymorphism, and refactoring has limited its value when applied to the small, simple methods of object-oriented software. Definition: Structural testing is a white-box unit testing technique for identifying test cases that identify defects involving logic of a functional unit of software. Test Completion Criteria: The associated test completion criteria are: The number of test cases equals McCabes cyclometric complexity. Test every branch within the method. Steps: During structured testing, the tester performs the following steps: Determine the McCabes cyclometric complexity (i.e., the number of test cases) of the method/function. Develop a test case TBD. A.6 Error Guessing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.7 Abstract Class via Concrete Test Subclass Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.8 Interface via Concrete Class Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.9 Embedded Test Suite Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.10 Parallel Test Driver Hierarchy Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.11 Test Subclasses Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.12 Peer Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.13 Test First Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.14 Bottom Up Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.15 Hierarchical Incremental Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.16 Polymorphism Testing Discussion: Correctly programmed classes support polymorphic substitutability. In other words, the software should continue to function correctly if an instance of any class is replaced by an instance of any descendent of that class. Thus at run (test) time, one may be dealing with objects that are instances of expected classes or one may instead be dealing with instances of any of their subclasses. Polymorphic substitution can cause a combinatorial explosion of test cases. This is why it is often impractical to create a test case for every combination of classes that can be involved in an interaction (e.g., for every combination of message parameter class and descendent, object under test attribute class and descendent, object under test collaborator class and descendent). Definition: Polymorphism testing is a black-box software integration testing technique for identifying a practical set of test cases that determine if inheritance has been used properly to support polymorphic substitutability. Orthogonal Array Testing S (OATS) Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.17 Use Case Based Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.18 Record and Playback Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.19 Automated Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD A.20 Regression Testing Discussion: TBD Definition: TBD Test Completion Criteria: The associated test completion criteria are to: TBD Steps: During TBD testing, testers perform the following steps: TBD. Limitations: TBD B. Open Issues The following open issues must be resolved: What is the impact of wireless on testing? Can usability test planning be incorporated in this document will all other kinds of testing? How about kinds of testing that are not project (application) specific? C. Major Things To Be Done The following are major things that must be completed: Complete descriptions of testing work products Add composition of each team Complete timing of various kinds of testing Complete descriptions of testing techniques  Note that this is a traditional standard testing term and is not to be confused with Oracle, the company that makes relational databases.  Complete automation of testing is typically not practical for model and usability testing.  A software unit is the smallest software component that can be treated as a relatively independent software blackbox. For example, a software unit could be an object-oriented programming language class or interface, a procedural programming language function, or an HTML webpage.  Because these criteria typically overlap greatly, the total number of test cases will be reduced to avoid unnecessary redundant testing.  Simple accessors are so trivial that they are unlikely to contain defects, and any defects that they do contain will typically be elicited by testing other operations that use the accessors. Thus, they can be verified by visual inspection, and do not require explicit testing.  A link is broken if it does not map to the correct existing webpage or diagram.  After defects are located, severity one and severity two defects will be repaired. Severity three defects will be logged, and only repaired if team capacity and schedule are available.  This supports the creation of stress testing test cases and thus merges into stress testing.  Robustness testing thus overlaps functional testing.  Because these updates would be performed during a time of low traffic in a real-world environment, it is acceptable if the system handles a smaller load during the test. Nevertheless, all of the user requests have to be fulfilled.  In a sense, security testing is never complete because security testing should be repeated on a regular basis.  For example, the 20% that are used 80% of the time.  Where practical, reuse functional test cases.  It is also a technique for specifying the developers intended behavior of the class. Document ID: PTP Version: TBDProject Test Plan (PTP) Version Date: TBD Confidential( 2000 by Page  PAGE 7 >LN;<JKZ[uvwxy ()*+,<=WXYZ[{|jUjqUjUjwUjUj}UjU jU 5:;j5:;U5CJ5CJ 5B*CJ05CJ0?./0>LMN\j{|$$x$#$($./0>LMN\j{| $%&'(;z-\H{Q0 ^ - ^ 2 h  6 i  7 k Eu=-g$ #_ $%&'(;zl  *#(#$d$C$$l\p"Xz-\H{Q0 ^ - ^ 2 h  *'(BCDFGZ[uvwyz01KLMOPhijUjSUjUjYUjUj_UjUjeUjU jUjkUC  * + , . / = > X Y Z \ ] s t ' ( ) + , = > X Y Z \ ] f g j Uj5 Uj Uj; Uj UjA Uj UjG UjUjMU jUC   , - . 0 1 G H b c d f g } ~      0 1 2 4 5 H I c d e g h x y jUjUjUjUj#UjUj)Uj Uj/ U jUEh  6 i  7 k Eu=-gD} *     1 2 3 5 6 J K e f g i j  $%?@ACDTUopqstjUj|UjUjUjUjUj UjUjU jUjUC789;<_`z{|~  '()+,FGabcefyz#$>?jUj^UjUjdUjUjjUjUjpUjUjvU jUCD}*c6bQE;nYAq(@fIf A`Ƹ    @  c    d   d   i        2?@BC\]wxy{|  $%&()BC]^_abuv01245AB\j@UjUjFUjUjLUjUjRUjUjXU jUE}*c6bQE;nY * \]^`axy01KLMOPij$%?@ACDfgj"$Uj#Uj(#Uj"Uj."Uj!Uj4!Uj Uj: U jUjUC5679:MNhijlm~89STUWXuvj)Uj(Uj (Uj'Uj'Uj&Uj&Uj%Uj%Uj$U jUC !;<=?@PQklmop4M*!:!o""""####$F$r$z$% &Q&w&&&("(((((()7)))+,/,>,,,,,E-O-56"j5:;CJOJQJUmHj*Uj{*Uj)Uj)U jUMAq(@fIf A`-;J-;JU_&74Ƽ|rh^TJ@J    $      S  d      c   cT  b  s             JU_&74o"#%((+++/,,,E---4.$$ h h4o"#%((+++/,,,E---4..////ĶwmcYOE;    B      q      d    $  \  w   w      2  T    O-----4.@.../$///111144!7'7777788s9w9\:c:::1<I<t==^>q>O?b?Y@n@UUX!XXXYYc[h[L\R\]]]]`^j^^^__8a=abbc0c>dHdWeaeee5f=fhhWi]iiijjO?ȾxndZPF<  !}  +  *G  )  (  '>  &  %  $  $          E  #    5  "M4`44 6$6B6d666778q9\::1<t=^>O?Y@SArAAA  :H  9     8  7  6#  5^  4[   [K  4  3A  2/  1  0  /  3\  2  1ST{TUUUUVV$V3VFVVVVVWW&W0W?WLWXWcWmWWXXX${TUUUUVV$V3VFVVVVVWW&W0W?WLWXWcWzpf\RH>  =  <  ;  :  C  9  8  7  6  5  B    A  @  ?  >  =C  <Z  ;cWmWWXXXX:YYYZNZZZZZ)[X[c[r[[[L\\\xndaWMC@]  @  ?7  OFQ  N  M  L  K   J"  I[  H  G  Fo  E  D    jt  >XX:YYYZNZZZZZ)[X[c[r[[[L\\\|\\\\]"]}]]]]$\\|\\\\]"]}]]]]O^`^^_h___`@`l```xukaWMCr  B  A  ]  \:  [i  Z  YP   P  X  W,  V  U  T  S  R-  QM  P]O^`^^_h___`@`l``````a8aGaaaabbbbbbc$````a8aGaaaabbbbbbc:ccc>dRddxnkaWMJ@  h  gS  f  e  d)  cL  b^  an  `  I7  Hp  G  _  F  E  D;  CP  ^c:ccc>dRddddGeWeee1f\ffSggNhhhhWigiyiiii(j$dddGeWeee1f\ffSggNhhhhWigiyiiii(jajxukaWTJ@  v  u6a  tr  s  r7  qFk  p  o  n  m   l  k  j X   X  KB  Jl  i(jajjjk/lw$$tcuruuZvivvvw+w>wLwtwwwxxxyjysyyz.zurh^PDA>  m   m%    :  {          %U  Y  X  Y  W  V  u  >wLwtwwwxxxyjysyyz.zGz|zzzz{t{{{{{{|?|n|$.zGz|zzzz{t{{{{{{|?|n|||}}=~h~~~~Ż|yoa^[QGD    p   pQ  e       F  d        A  X      n|||}}=~h~~~~:I.A]{Ձ+V$~:I.A]{Ձ+Veփ_Ⱦ|rh^TJ@      z    #  N9   9C  W          $    .  X  eփ_]Dž:4Rx։/au$_]Dž:4Rx։/auǽ~tjg]SI?<J        '  E  a  t    a    ?  [  Z          u ܌J$N*j4ۑ{ĺ~tqg]SI?`    }    $  k  .  _       4Z   Z/        n       ܌J$N*j4ۑ{ETYi|${ETYi|:WŖDȾ|rh^TJ<     q  D            k          `    >    :WŖDÚ JÛb!?$Ú JÛb!?e˝Qϟ9?urh^TJB    N          :`  ~      =        U    ?e˝Qϟ9?ˠ۠)7FVԡ/x< $?ˠ۠)7FVԡ/x<|̣xndZWM?<]   ]n     5   r       ^  ],  \J                  <|̣7y}٤,X(BO˦Ϧ<$$̣7y}٤,X(BO˦Ϧĺ~tj`]SEB   V      a  `   _1  u        (  H  `   `  6  X  </9Icpͨ(<Q_oĺ~tj`VLB8  k    j  i  h  g$  f8  eZ  !   !    d*  c:  bD      7  Z  /9Icpͨ(<Q_oƩک*;Pg$$Ʃک*;Pgwߪ1Ucnzĺ~{qc`VLB8#  y.  x<  w`         v  u  t  %  s6  rG  qh  w  p  o  n  m  lgwߪ1Ucnzͫޫ'AN $$zͫޫ'AN 1BWkĺwtj`VLB8      "  C  ^  ?   ?  CP  j  z        ~  }    |  {  zNYqr]^Q^=>TUEN Zf /K``t4D">D!+grFd!$CJ6 j0J!U5_ 1BWk{aozȮ֮}ޯCw̰ݰ$k{aozȮ֮}ޯCw̰ǽukaWMC  -)  ,  +  *  )    (  '  &  %  $  #)  "  ]a  !      ̰ݰ&Pı3ny.x*Wĺwif\RH>  5  4  3*  2     1  0'  /   9  .        F  o      &Pı3ny.x*W@!Bh|'>b$@!Bh|'>bȷ׷ȸ-%SĺĶ}zpbzXND5  >o  =  <l   l  ;     :  9  8  7"   "          K  6bȷ׷ȸ-%SĺԺpKeмѼ:E'ĺԺpKeмѼ:EϽӽ 1Ż{xnif\WTJE'UY  I'  H'  G0';u'    F  E  D$  C>  B    A  @  ?   Ͻӽ 1l 8#1Yq'1l 8#1Yq<Pp¿{qg]XUK/  OO'c        N   .  F  Mn'|  Lg  K''  {'  J'' D<Pp6Z5F2=IY_)'6Z5F2=IY_))Ŀ}sndZPF>    Z  Yv  X  W'   V@'FV  Ub  Tm  S'  RY  Q'j  E  i    P     )45#R_$K{ !@v $$')45#R_$K{ !@v Ⱦzurh^TJ@    c4  J    b'  aE  `u  _'    a  n    ^7  ]  \  ['' vgTQ~Aĺ~tj`VLB8     B  o  e        )  d l           1   Y              J      vgTQ~Aq #4i' Aq #4i9F`w5UiƼxndZPF<k                 I  `  z    j''  iW  h'  g  f O     9F`w5Uiy+1ci'iy+1ciqĺvqg]SIA<'    s  r$  q?  p'OW  o]  n'    m         l  %  :  kG  W  iq^i4d '6EU'^i4d '6EU[@Ŀxnif\WMCA    '  &',<  ~K  }Z  |n  {z  z'u  y  x'  w  v'M  u  t'#|U[@0;+Tq4^i8C:H'0;+Tq4^i8C:H AU¸{qli_UPM'  M  '  R  'o'z      3N  k    '  g     AUy(5EXc$'y(5EXcƼ}ojgb_UKA  (    '{''   'X        '   '!  0  ?  S  _  'H  Uc$v#`#6<@GKgrȾ|wtol^YVQN'&}'t   t''  )  >  e  r  ''*    G'[    '@  Uc$v#`#6<@GKgrR$'R AwEwƼukaWROJGD' 'L  x      ' G'X  x     '4`    'Q    .      R AwEw$'H>Z K`4ɿvlbXNB='    c    7    L  z    =      Y    O    ''  H>Z K`4 U' U"wLT¸wroe[VSI  '@  }  '*  *  *H  *  *   *r '  ?      t    ' "wLT)4AXm $*')4AXm!Yft .ĺxnd_\RH>t      '   .  ;  s'y'            '  <   S  `  k  y  !Yft .9EYhx!-H'.9EYhx!-HWgtĺ~tj`VLB8-  =  L  g  s                       ,  ;  O  [  f  HWgt3h N  ' u  X     J  't3h N  ' u  X    ĺ~tj`VLB='        +      <  o          ''            J    5Io$8~tj`VLB .  >   S  h             ='M     '<  h        G  u   5Io$8HWgt $'8HWgt!(#1<H\k{Ⱦzpf\RH>               !  /  ;    *'1J  X  e  ''       !(#1<H\k{$0KZjw'$0KZjw[ĺ~tfa^YVLB)    ''             "  .  >  M  ]          $.ju,01_gr~BM < )):*D***G+Q+++f,y,--H3S3:::/:3:?:@:I:s:y:::::::;';F;Q;p;{;<=KKfZgZ^^__kkllrrjsvs)t;t}}}~~$~>~H~~~".BLmH j0J!U65_[q3{)i:I$'[q3{)i:I#ĺvlgdZPKH>0  '  C  b'v             a  '        0      #.;S /?L_  '#.;S /?L_ĺ{xndZURH>V    E'Le  s    ''  %  5  A  U  e             "    4CSbr#2Bĺ~tj`VLB8  "  =  I  Y  h  x                    %  1  <  J   4CSbr#2BOjy   O!"$'BOjy   O!"#l##&$Z$h$$$,%% &½~tolbXNDl    =  }  '  P      r    D  m'x'           "#l##&$Z$h$$$,%% &6&V&j&&&&'y''''''(D(T(h(' &6&V&j&&&&'y''''''(D(T(h(t((((((ȾxndZPFA>}'        !      4  v          )'/]    '  $  h(t(((((((),):)S)Z)):*F*T*_*k**********+'((),):)S)Z)):*F*T*_*k**********+ȾxndZPF<7  -K  ,W  +b  *p  $  )  (  '  &  %  $  #  "  #y  "'  !    B'H+(+7+G+S+n+}++++++++-!-N---.:.T.ĺ|wtj`VLB>  :Y  9  8  7*  'W'b'   i  6x  5  4  &  3  2  1  %  0  /  .+(+7+G+S+n+}++++++++-!-N---.:.T../0P0000$'T../0P0000)1d11G2t222!3c33334?4Z44Ƽxnd_\WMC  ;9  6'Y'  5  4W  3'  21  1y  0  /O  .  -'(  ,p  +  *  )$  (0)1d11G2t222!3c33334?4Z444L555&6o66 7_777X8'44L555&6o66 7_777X889B999999:ĺ~tj`VLB=:|'  K  J  96  Ij  8  H   G~  F  E  Dm  C  B   AR  @  ?  >,  =  <  7X889B999999:::::::#:):/:$$/$$l40 #` {$$ $':::::::#:):/:0:1:2:3:;:@:I:Q:[:f:q:r:s:y:}:::::::::::|xsnid_Z                              '  /  8  =  E  F  G  HI  O  U  _  h  ij  p  q'x"/:0:1:2:3:;:@:I:Q:[:f:q:$P$$l4r #   q:r:s:y:}:::::F[$$l4ֈ* # $[$$l4ֈ* #`:::::::::::::::::;;[$$l4ֈ* # $:::::::::::;;;;;;';+;/;9;D;E;F;Q;U;Y;c;n;o;p;{;;;;|wsnid_Z                 #  '  2  34  ?  I  M  Q  [  \]  h  r  v  z                ";;;;;';+;/;9;D;E;F;Q;U;Y;c;n;o;p;[$$l4ֈ* # $p;{;;;;;;;;;B<M<}<<<==g=$'[$$l4ֈ* # $;;;;;;B<M<}<<<==g=n=}====== >!>I>>?{qli_ZWMC  ?  >('<L  =y'  <  R  Q  P  O*  ;;  N  M  L  :''     g=n=}====== >!>I>>?$?Z?z?~?????[@@@@OAZAAMB$'?$?Z?z?~?????[@@@@OAZAAMBBBBCCD,DVDjDDȾzpf\RMJ'  JK  I  H,  Gg  F'  E  D''     Cu  B  A''  @'%'6MBBBBCCD,DVDjDDDDD(EHELEREVE]EE%FnFFGG%H0H_H$'DDDD(EHELEREVE]EE%FnFFGG%H0H_HjHI5IlII J.J[JȾukaWMHE'H  S  R  U  T  S  Q'a'l)   )&  P  O  N''  M'#V'g  L  K_HjHI5IlII J.J[JJJJJ2KmK~KKKKLFLLLPLWLL:MMMM'[JJJJJ2KmK~KKKKLFLLLPLWLL:MMMMNNIN~NNùzlgd_\RH  _  ^''     ]w  \  [''  Z  Y?  X'_'  W   VG'[|  U  TMNNIN~NNN O OBO[OO1P?PmPPPPP/QrQQQQ*RFRRRR'NN O OBO[OO1P?PmPPPPP/QrQQQQ*RFRRRRRĺ}zukaWROJ''  h  gR  f'r'  e3  dY'm  c  b'T  a  `  Z   Y  XC  W_  VRRRsSSSSU%UPUsUUUUUUUUUUVV.V8VKVUVhVV& $'RRsSSSSU%UPUsUUUUUUUUUUVV.Vĺ~tj`VLB M  !Y  \ l   t                         [  l<'G'ڞ  ڞ  kT  j  i.V8VKVUVhVVVVVVVVVVSWWWXVXXXĺ~tqg]SI?  r   qO  p  o  nj  m   -   ,   +   *  ]&  )&  (   '&  &&   %&  $ )  # 3  "VVVVVVVVVSWWWXVXXXY!YLYwYYY6ZJZ~ZZ#[[['& XY!YLYwYYY6ZJZ~ZZ#[[[_\p\\\\]]^F^^^ukaWMC>'  a    F    ~  }''u  |  {>  z  y  x'+b  w  v  u  t@'N  s[_\p\\\\]]^F^^^^_``7aabdbwbbkcvcccdMdvd '^^_``7aabdbwbbkcvcccdMdvddddeeŻukaWMC    _ 3  2 h  1   0   /   .0  ^f  ''3  3B    *    '    vddddeeee fLfffffg3g[ggg"hThhhh@i`ii:jj$' eee fLfffffg3g[ggg"hThhhh@i`ii:jjȾxsi_ZPF<  /    '  D  'U    Z  {  c  b  a  `2  V'j      '"~  j$kkl=llll5mzmmm*nnnnnnnoEo^oĺvlbXND:  j  i  h/  g;  fK  e`  d    "'(       I        2    d  j$kkl=llll5mzmmm*nnnnnnnoEo^oeoop0q@qKqq'^oeoop0q@qKqqqqDr{rrjs)ttttOuZuuu$v|vvv wŻ|wtj`VLGD'  %      ''/  /  j    Y    ''  U  &    'qqqDr{rrjs)ttttOuZuuu$v|vvv w4wTwwwwwxMxx$' w4wTwwwwwxMxxxxyRybyyyyzLzYzz{ƼxndZPF<L  o   5  n 7  4 e  3t  m  '5  lL  kw      6  V'j        =  xxxyRybyyyyzLzYzz{S{c{{{{{l|}|||}}G}[}i} '{S{c{{{{{l|}|||}}G}[}i}}}}} ~~>~J~ĺxspf\RH>   1  ?  tK  h  '      B'H'   :  s Q  9a  r   8  q   7  p 7  6i}}}}} ~~>~J~V~j~y~~~~~~~1%:Cw$'J~V~j~y~~~~~~~1%:Cwˁĺ~tj`VQND  ')  2  |G  {    ;    ''        z  y  x  w  v  uˁ8]т!p(m̄G[څCI'8]т!p(m̄G[څCƼxndZPFA>'    %  l        ~D  }V  '  K  |  '    4  Y  y  CI"BNbqÇw#$%Ⱦtrmrjfjdjjdbrrrrm   %   %           *  J  X  d    '  #') "BNbqÇ·R]ψd͉B`|'L“%>H{.;řә'1z{tyϝڝݟVaJTNX +0-8v&!ǾѾq# 6mr *afHM j0J!U5a0Di-XjyčO\_oȏRu 'uǐא5B_oՑKX 29ēғ$' >JUauؔ]{'5' )$ & F & F & F)Mtʝϝݟ'ˠ /1V/JNhک$ & F,N=hɮӮ -ϯ߯+|7- $-^v-S<?Ǿqi hm$ & F\aCH!&fk|$ & F&+kvU`{0;S]cmt)39CJb '?v{ +bg=B$doCNku{5d PUf{+0ASct)9J & F$'qv]b8=}$ & F_du>CTk{ 1IYj & F${ +ISYcj   &']^GH%hmH jUhh j j0J!U57Ju6m &]Gvw & Fw {ywrmh$$$$h$-$$l0" $8$$lFx" D($ o P$ !$$ !"#$%8$$lF@ "  $ / =!"#$%}DyK _Toc499001562}DyK _Toc499001563}DyK _Toc499001564}DyK _Toc499001565}DyK _Toc499001566}DyK _Toc499001567}DyK _Toc499001568}DyK _Toc499001569}DyK _Toc499001570}DyK _Toc499001571}DyK _Toc499001572}DyK _Toc499001573}DyK _Toc499001574}DyK _Toc499001575}DyK _Toc499001576}DyK _Toc499001577}DyK _Toc499001578}DyK _Toc499001579}DyK _Toc499001580}DyK _Toc499001581}DyK _Toc499001582}DyK _Toc499001583}DyK _Toc499001584}DyK _Toc499001585}DyK _Toc499001586}DyK _Toc499001587}DyK _Toc499001588}DyK _Toc499001589}DyK _Toc499001590}DyK _Toc499001591}DyK _Toc499001592}DyK _Toc499001593}DyK _Toc499001594}DyK _Toc499001595}DyK _Toc499001596}DyK _Toc499001597}DyK _Toc499001598}DyK _Toc499001599}DyK _Toc499001600}DyK _Toc499001601}DyK _Toc499001602}DyK _Toc499001603}DyK _Toc499001604}DyK _Toc499001605}DyK _Toc499001606}DyK _Toc499001607}DyK _Toc499001608}DyK _Toc499001609}DyK _Toc499001610}DyK _Toc499001611}DyK _Toc499001612}DyK _Toc499001613}DyK _Toc499001614}DyK _Toc499001615}DyK _Toc499001616}DyK _Toc499001617}DyK _Toc499001618}DyK _Toc499001619}DyK _Toc499001620}DyK _Toc499001621}DyK _Toc499001622}DyK _Toc499001623}DyK _Toc499001624}DyK _Toc499001625}DyK _Toc499001626}DyK _Toc499001627}DyK _Toc499001628}DyK _Toc499001629}DyK _Toc499001630}DyK _Toc499001631}DyK _Toc499001632}DyK _Toc499001633}DyK _Toc499001634}DyK _Toc499001635}DyK _Toc499001636}DyK _Toc499001637}DyK _Toc499001638}DyK _Toc499001639}DyK _Toc499001640}DyK _Toc499001641}DyK _Toc499001642}DyK _Toc499001643}DyK _Toc499001644}DyK _Toc499001645}DyK _Toc499001646}DyK _Toc499001647}DyK _Toc499001648}DyK _Toc499001649}DyK _Toc499001650, [B@B Normal$PPa$CJ_HmH sH tH J@J Heading 1$$ & Fd@&5CJ OJQJZ@Z Heading 2%$ & F|@& @5CJOJQJH@H Heading 3$ & F@&5CJOJQJHH Heading 4$ & F@&5CJOJQJLL Heading 5$ & F<@&5CJOJQJ@@ Heading 6 & F<@&5CJHH Heading 7 & F<@&5CJOJQJDD Heading 8 & F<@& 6OJQJJ J Heading 9 & F<@&56CJOJQJ<A@< Default Paragraph Font(U@( Hyperlink>*B*@@@TOC 1xx # 5;CJmHB@BTOC 2* *(# :CJmHB@BTOC 3& # 6CJmH** TOC 4 CJ** TOC 5 pCJ** TOC 6 LCJ** TOC 7 (CJ** TOC 8 CJ** TOC 9 CJ>O> Bullet 2 & F8( h6@6 Header$ !da$6 @6 Footer$ !da$8@8 Footnote Text $da$44 Rationale$h^ha$BOB Bullet 1  & F h(^&)@& Page Number6O6 Bullet 3 & F 8&@8 Footnote ReferenceH*8V@!8 FollowedHyperlink>*B* <>@< Title#$1$a$5CJ$OJQJ>OB> Tabletext$$1$dxCJD#D Table of Figures%H^`H,Ob, Bullet 4 &^*O* Label'$5CJPCPBody Text Indent($x CJOJQJ88 bulllll)$ & FCJ<O< Bullet1* & F  hCJ@Z@ Plain Text +$ CJOJQJb2u z(}FT1ff L6 ehQ      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQLLLLL666         eeeeeeeeh  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOP^Q!!!   !!! ! ! ! ! !!!!!!!!!!!!!!!!!!! !!!"!#!$!%!&!'!(!)!*!+!,!-!.!/!0!1!2!3!4!5!6!7!8!9!:!;!<!=!>!?!@!A!B!C!D!E!F!G!H!I!J!K!L!M!N!O!P!Q)Q&'.7CNPUC\c jq9w~=3mKlX;fd Bzw 6g#[&#.L59?EJePT\{cijkfr xD}6ֈY_VlN?19b  s  7 (#@Q78D!z $ m! "#g$%x&'()A*+,^-./012l34567$8$9F:;<Q=>L?^@A4B_CDE F#G H-I%JRKLMJN.OPtt ?\O-fo$L{%   %<gzh }J4.M4DSX]c(jo>wn|?<g b iURH  "h(+0X8/:q::;p;g=MB_HMRV[vdjqxi}u)-w %   "&(*,/1358:=@BDFHKMOQSVXZ\^`cehjmoqtvxz{|~4/3O?4K{TcW\\`dajnt.z~_u{?̣zk̰ĺ1) Ai.t 8[#B &(+T.4::;?D[JNR.VX^ej^o w{J~C%!#$')+-.024679;>?ACEGIJLNPRTUWY[]_abdfiklnprsuwy};JZvx )+<XZ{'CFZvy0LOh+.=Y\s (+=Y\f-0Gcf}14Hdgx  2 5 J f i    $ @ C T p s  8 ; _ { ~ ( + F b e y   # ? B \ x { %(B^au14A]`x0LOi$@Cf69Mil~8TWu <?Plo 4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4%4!` _Hlt499001651 _Toc499001562 _Toc434992861 _Toc499001563 _Toc499001564 _Toc434992862 _Toc499001565 _Toc434992863 _Toc499001566 _Toc434992864 _Toc499001567 _Toc460674817 _Toc499001568 _Toc499001569 _Toc499001570 _Toc499001571 _Toc499001572 _Toc499001573 _Toc499001574 _Toc499001575 _Toc499001576 _Toc499001577 _Toc499001578 _Toc499001579 _Toc499001580 _Toc499001581 _Toc499001582 _Toc499001583 _Toc499001584 _Toc499001585 _Toc499001586 _Toc499001587 _Toc499001588 _Toc499001589 _Toc499001590 _Toc499001591 _Toc499001592 _Toc499001593 _Toc499001594 _Toc499001595 _Toc499001596 _Toc499001597 _Toc499001598 _Toc499001599 _Toc499001600 _Toc499001601 _Toc499001602 _Toc499001603 _Toc499001604 _Toc499001605 _Toc499001606 _Toc499001607 _Toc499001608 _Toc499001609 _Toc499001610 _Toc499001611 _Toc499001612 _Toc499001613 _Toc499001614 _Toc499001615 _Toc499001616 _Toc499001617 _Toc499001618 _Toc499001619 _Toc499001620 _Toc499001621 _Toc499001622 _Toc499001623 _Toc499001624 _Toc499001625 _Toc499001626 _Toc460674821 _Toc499001627 _Toc499001628 _Toc499001629 _Toc499001630 _Toc499001631 _Toc499001632 _Toc499001633 _Toc499001634 _Toc499001635 _Toc499001636 _Toc499001637 _Toc499001638 _Toc499001639 _Toc499001640 _Toc499001641 _Toc499001642 _Toc499001643 _Toc499001644 _Toc499001645 _Toc499001646 _Toc499001647 _Toc499001648 _Toc499001649 _Toc499001650x((IJJU6&Q&..;]LMPRRX_f#lss%x\|;͛VCt%mOOKlXdzw5([&,6.;@ HDN\k,oOy6__jgqJ>%vS?  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEF_GHIJKLMNOPQRSTUVWXYZ[\]^?eeTTP&P&,#..;jLMPRRX_f7lst1xg|EŚܛfS3z\(]Zvz;H4m&A6@; AHTN\k9o[yHiРw^<&xX5$%(((())k,p,K LXY\\]])Ϟ۞,8LLTT\!\iinn&5{}?CDonald Firesmith@C:\windows\TEMP\AutoRecovery save of ProjectTestPlanTemplate.asdDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dotDonald Firesmith@C:\windows\TEMP\AutoRecovery save of ProjectTestPlanTemplate.asdDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dotDonald Firesmith@C:\windows\TEMP\AutoRecovery save of ProjectTestPlanTemplate.asdDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dotDonald Firesmith@C:\windows\TEMP\AutoRecovery save of ProjectTestPlanTemplate.asdDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dotDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dotDonald FiresmithaD:\Open\OPEN Webpages\Components\WorkProducts\TestSet\ProjectTestPlan\ProjectTestPlanTemplate.dot~8x.J#z(.Bp-R&xW <_q*0{MC* LvM$4|hLsm-g`sK'Sz *~@lD)vpT G OJmN|"+{ >19VK+ 1 v3$ qe` ~ BY"e   . N: nhR M9 FE 'f ql 2N O( > \^kJz'NJqM<RY|."VO&e /)f$t <_#R}H  l|SX7JboLT~/O98SSaYllGJjvZRD4M{FW dYZ8x<_XfUoTohT 1<q2lpu6C*W88s`agYKg86$ @n 6~IK[KLYzQK   1!O!pcSp"-"b#RI#[#TR%3#+$J6.$;$i=$f!f$d$Om$%vE%2J&Is&i'S'u=(lD( nX(;k( Z$(V$(C( {>)L)w~)^) <*H**g8**K*I D+N!\+ ,+Fh,V,V8,?, -Eq-te2%-B\U -Wa-j.e`.*f._q."."9.hY 0. //70m$0hg0w11;'232o[3/^3<_33O 4H26m6(7BZ7 B>7cK08.p9OP9DI9k99Mo9t:CE:Q] ;b&;Uj;ZG_<#<1<Z1<c=Z=<_3/=_?=aq=W=ma=Lx>R(>VA>fb?Ke?<_A&?;7>AFyAFA%B?BYB wBgCKD<_;QDzDwyEE4NFyF9G# )H01HDH T,ImFIxIyI6JJmJE\K"KwWKEKK0 MNmN/Nr4N<_QDOWGOkOOLVOGPRK!PtP J1PFPKQQ[U>R@RFRz}HRTS2ST TS*T YUtzU52U!VV(98Ww"k gWWrXj8eXV2Y<_IY'6ZgZh[E2\$5\c]X]"(]w B_$L_Jn_<_`(`aR#a<_Kbk8;yboJc+~c4{dnee9$fof.WfNfF,xgf0g.RH$gTAhehv~hvFi`1>iSTIiRjvPi aiUi6i3j;jT2wj]j#1jAjGjc|Gj!jkskkv|k %ln.l6:lv'^l40lClc5-l5R]m*mm-Mmcn-4)Cn$5nwaln|1A4nCEoFoooonjp.lp<_p[&qNdqꛬ=tqk8qNr]Nr*Sr"qr=zrt V9VW@ldWe`.WTXj]jTXcNDwXnjpXW=DY;ybYYBY4Z@ZLZXZdZpZ|ZZZZZZZzwZl|4[KL[+$[9~u`\aq=\O(]v|k]] -T^YYOw^-l_Lv_J_*mH`VA>`4NFa%RxtaU -afb?Am$Xn+{ n*Sr oW8on obxLpOm$pQq32xq%q9@rRI#r sO 4ls".s<*4tQtotGP`u@RuUA}(vi'v1v"qrTw;n~wNJxJmxK[xN Hy1<y*f.z3#tzRDz<R<{M${tzU| YUh|w B_|Mo90}`a}CE:}$g\~Y|~OP9$Is&`sK6iPehNrPu}|Q] ;V$(D%Bll u=(p2J&ԃb&;8t:oJc# )Hd[U>RȅyF,G k9vE%X1<t ̈؈(98WClr4NKDJn_8xZ= R#a/^3[QxV2YKe?xW .lpg8BZ7%-e N!\+lD(TE\Kxs܉b@-Mmk8qYZlz!{Ћ"9.YK45\iԺTAh0ne52UxID?vgc|GjX33>nX(l/NȾ|"$B>7Lx>ܿ wB8=tq\KQLNmFIW{`1!g8*Gjth[$5n,X]w~)j.@+N_wia|f!f$T+,+ C(h'^lXfU g|LVO ai4Nfm6mNHZ$(TSFP\ {>) *~Wp.WfV(_q.O<eXv~hzDP98~IUj;d(`#1jFE xQDO$t09GvFi=zrD(7-"H**X4{dR(>R}Hl1 A&?$gCe V,8e` \R @h 8^8`56CJ-R @h 8^8`OJQJo(@0S @8 ^`OJQJo(YDT @ OJQJo(PT @ OJQJo(T@ OJQJo(T @ OJQJo(@U @ OJQJo(Uv@ OJQJo(U @ OJQJo(0Vx@ OJQJo(V P@ OJQJo(V @ OJQJo( W `@ OJQJo(pWy@ OJQJo(W @ OJQJo(X @ OJQJo(`X@8 OJQJo(X @8 OJQJo(Y @8 OJQJo(PY @8 OJQJo(Y @8 OJQJo(Y @8 OJQJo(@@Z @h ^`OJQJo(@[@h ^`OJQJo([ @h ^`OJQJo(\ @h ^`OJQJo(l\ S@h ^`OJQJo(\ @h ^`OJQJo(4] @h ^`OJQJo(] @h ^`OJQJo(] @h ^`OJQJo(`^ @h ^`OJQJo(^ @h ^`OJQJo((_ @h ^`OJQJo(_ @h ^`OJQJo(_ @h ^`OJQJo(T` @h ^`OJQJo(` @h ^`OJQJo(a @h ^`OJQJo(a @h ^`OJQJo(a @h ^`OJQJo(Hb @h ^`OJQJo(b @h ^`OJQJo(c @h ^`OJQJo(tc @h ^`OJQJo(c @h ^`OJQJo(?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefgijklmnoqrstuvw~Root Entry F`g3@]lData u+1TableCWordDocument^SummaryInformation(hDocumentSummaryInformation8pCompObjjObjectPool@]l@]l  FMicrosoft Word Document MSWordDocWord.Document.89q