Prototype Usability Testing

Definition

Prototype usability testing is the integration testing of a user interface prototype against its usability requirements to determine if it contains any usability defects.

Objectives

The typical objectives of prototype usability testing are to:

• Partially validate a user interface prototype by determining if it fulfills its quantitative and qualitative usability requirements concerning ease of:
  • Installation by the environments team.
  • Usage by the user organization.
  • Operations by the operations organization.
• Identify usability defects involving the application's human interface such as:
  • Difficulty of orientation and navigation (e.g., navigation defects such as broken links and anchors within a website)
  • Efficiency of interactions (based on user task analysis)
  • Information consistency and presentation
  • Appropriate use of language and metaphors
  • Conformance to the:
    • Digital Brand Description Document
    • Website Design Guidelines
  • Programming defects (e.g., incorrectly functioning tab key, accelerator keys, and mouse actions)
• Determine if the application's human interfaces should be iterated to make them more usable.
• Determine how well the human interfaces enable the different categories of users to efficiently perform their business tasks.
• Report these failures to the development teams so that the associated defects can be fixed.
• Help determine the extent to which the application is ready for launch.
• Provide input to the defect trend analysis effort.

Preconditions

Prototype usability testing can begin when the following preconditions hold:

• The usability requirements to be tested have been specified and implemented.
• The user experience team is adequately staffed and trained in usability testing.
• The prototype under test exists.
• Test participants are available.

Completion Criteria

Prototype usability testing is typically complete when the following postconditions hold:

• The usability team has tested the user interface. For example:
  • All links and anchors navigate to the proper parts of the proper webpages.
  • All user inputs produce proper results.
  • Webpages are consistent with branding guidelines.
  • Screens open and close properly.
• A group of test participants representing users from each type of user have been identified and have evaluated the user interfaces.
• The results of the evaluation have been analyzed.
• Associated test reports with recommendations have been presented back to the development team.

Tasks

Prototype usability testing typically includes the user experience team performing the following testing tasks:

• Test Planning:
  • The user experience team plans prototype usability testing.
  • The user experience team enters plans for document usability testing in the project test plan by entering the following topics in the prototype usability testing subsection of the project test plan:
    • Goals
    • Objectives
    • Testing work products (see below).
    • Roles and responsibilities (e.g., test monitor and test participant including user profile and sources)
    • Tasks
    • Test environment including required equipment
    • Completion and success criteria
  • The user experience team summarizes its plans for prototype usability testing in the master test list.
• Test Reuse:
  • Where practical, the usability team reuses test work products.
• Test Design:
  • The user experience team characterizes and profiles the users of the prototype to identify the questions to be used to screen potential test participants.
  • The user experience team determines:
    • The appropriate number of each kind of test participant.
    • What to say to orient each test participant.
    • What to to ask each test participant prior to the test.
    • What observations to make during testing.
    • The test suite of test cases to develop.
    • The associated test procedure.
    • What to ask each test participant following the test.
    • What to pay each test participant.
• Test Implementation:
  • The user experience team develops the following work products:
    • Usability Test Screening Questionnaire
    • Usability Test Orientation Script
    • Usability Test Observations Form
    • Usability Pretest Questionnaire
    • Test Procedure
    • Test Suite
    • Test Case
    • Usability Posttest Questionnaire
  • The user experience team uses the usability test screening questionnaire to recruit the prototype usability test participants, if possible from the actual target user communities.
  • The usability team prepares the test environment, equipment, and prototype.
• Test Execution
  • The test monitor dry runs the test procedure and the associated test suite of test cases.
  • The test monitor greets the test participant.
  • The test monitor reads the usability test orientation script to the test participant.
  • The test monitor has the test participant sign a nondisclosure agreement and a consent to be recorded form.
  • The test monitor has the test participant fill out the usability pretest questionnaire.
  • The test monitor starts any recording devices.
  • The test participant tests the prototype by following the test procedure and performing the associated test suite of test cases.
  • The test monitor observes the test participants during the test and records his/her observations on the usability test observations form.
  • The test monitor has the test participant fill out the usability posttest questionnaire.
  • The test monitor debriefs the test participant and records his/her observations on the usability test observations form.
  • The test monitor thanks and compensates the test participant.
  • The test monitor analyzes the usability test results.
• Test Reporting:
  • The user experience team prepares a test report including recommendations.
  • The user experience team delivers the test report to the development team so that they can fix the defects.
  • The user experience team updates the current test summary report including recommendations.

Environments

Prototype usability testing is performed on the following environments using the following tools:

• Staging Environment
  • A protocol of tasks and preference questions that the participants will run through.
  • Equipment including, but not limited to:
    • Computer and monitor
    • Video camera(s)
    • Video mixer
    • Tape recorder(s)
    • Screen capturing tool (e.g., Lotus ScreenCam)
    • User action capturing tool (e.g., Winrunner)
    • Scan converter

Work Products

Prototype usability testing typically results in the production of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Usability Test Screening Questionnaire
  • Usability Test Orientation Script
  • Usability Test Observations Form
  • Usability Pretest Questionnaire
  • Test Procedure (a.k.a., test protocol, test script)
  • Usability Posttest Questionnaire
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Suite
  • Test Case

Phases

Prototype usability testing consists of the following tasks being performed during the following phases:

Guidelines

• Usability testing of user interfaces should begin early with relatively informal usability testing of paper prototypes.
• Usability testing should be done with representative users, either in a usability lab or in the production environment.
• During the construction phase, usability testing can be done on a full user interface prototype after the design effort is relatively complete.
• The following are suggested techniques for measuring usability:
  • Average time and number of steps required to install and configure the application.
  • Average time to complete a given user task.
  • Average percentage of tasks completed in a given time.
  • Average number or percentage of user errors.
  • Number of user interactions required to perform a specific user task.
  • Average number of times the user uses either inline help features and documentation.
  • Percentage of times that users expresse frustration or satisfaction.
  • Average time required to obtain proficiency using the application.
  • Success rate at finding the desired feature and using it optimally.