Document Usability Testing

Definition

Document usability testing is the small component testing of a document against its associated usability standards and its usability requirements to determine if it contains any usability defects.

Objectives

The typical objectives of document usability testing are to:

• Identify usability defects involving deliverable documentation to determine if it should be iterated to make it more usable.
• Determine how well the documentation enables the different categories of its users to efficiently perform their tasks.
• Report the observed defects to the development teams so that they can be fixed.
• Help determine the extent to which the application is ready for launch.
• Provide input to the defect trend analysis effort.

Preconditions

Document usability testing typically can begin when the following preconditions hold:

• The documentation standards exist.
• The usability requirements to be tested have been specified and implemented.
• The user experience team is adequately staffed and trained in document usability testing.
• The document under test exists and is ready for usability testing.
• An adequate number of appropriate test participants are available.

Completion Criteria

Document usability testing typically is complete when the following postconditions hold:

• The usability team has tested the deliverable documentation (i.e., either each document or a representative sample of documents).
• Each tested document has been tested by a group of test participants that are representative of the appropriate type(s) of users for that document.
• The results of the usability tests have been analyzed.
• Associated test reports with recommendations have been presented back to the development team.

Tasks

Document usability testing typically includes the user experience team performing the following testing tasks:

• Test Planning:
  • The user experience team plans document usability testing.
  • The user experience team enters plans for document usability testing in the project test plan by entering the following topics in the document usability testing subsection of the project test plan:
    • Goals
    • Objectives
    • Testing work products (see below).
    • Roles and responsibilities (e.g., test monitor and test participant including user profile and sources)
    • Tasks
    • Test environment including required equipment
    • Completion and success criteria
  • The user experience team summarizes its plans for document usability testing in the master test list.
• Test Reuse:
  • Where practical, the usability team reuses test work products.
• Test Design:
  • The user experience team characterizes and profiles the users of the documentation to identify the questions to be used to screen potential test participants.
  • The user experience team determines:
    • The appropriate number of each kind of test participant.
    • What to say to orient each test participant.
    • What to to ask each test participant prior to the test.
    • What observations to make during testing.
    • The test suite of test cases to develop.
    • The associated test procedure.
    • What to ask each test participant following the test.
    • What to pay each test participant.
• Test Implementation:
  • The user experience team develops the following work products:
    • Usability Test Screening Questionnaire
    • Usability Test Orientation Script
    • Usability Test Observations Form
    • Usability Pretest Questionnaire
    • Test Procedure
    • Test Suite
    • Test Case
    • Usability Posttest Questionnaire
  • The user experience team uses the usability test screening questionnaire to recruit the document usability test participants, if possible from the actual target user communities.
  • The usability team prepares the test environment, equipment, and documents.
• Test Execution
  • The test monitor dry runs the test procedure and the associated test suite of test cases.
  • The test monitor greets the test participant.
  • The test monitor reads the usability test orientation script to the test participant.
  • The test monitor has the test participant sign a nondisclosure agreement and a consent to be recorded form.
  • The test monitor has the test participant fill out the usability pretest questionnaire.
  • The test monitor starts any recording devices.
  • The test participant tests the document by following the test procedure and performing the associated test suite of test cases.
  • The test monitor observes the test participants during the test and records his/her observations on the usability test observations form.
  • The test monitor has the test participant fill out the usability posttest questionnaire.
  • The test monitor debriefs the test participant and records his/her observations on the usability test observations form.
  • The test monitor thanks and compensates the test participant.
  • The test monitor analyzes the usability test results.
• Test Reporting:
  • The user experience team prepares a test report including recommendations.
  • The user experience team delivers the test report to the development team so that they can fix the defects.
  • The user experience team updates the current test summary report including recommendations.

Environments

Document usability testing is typically performed on the following environments using the following tools:

• Usability Lab:
  • Equipment including, but not limited to:
    • Video camera(s)
    • Video mixer
    • Tape recorder(s)

Work Products

Document usability testing typically results in the production of the following work products from the test work product set:

• Documents:
  • Project Test Plan
  • Master Test List
  • Usability Test Screening Questionnaire
  • Usability Test Orientation Script
  • Usability Test Observations Form
  • Usability Pretest Questionnaire
  • Test Procedure (a.k.a., test protocol, test script)
  • Usability Posttest Questionnaire
  • Test Report
  • Test Summary Report
• Software and Data:
  • Test Suite
  • Test Case

Phases

Document usability testing typically consists of the following tasks being performed during the following phases:

Guidelines

• Usability testing tests the work product under test rather than the test participant. The test participant is always right.
• The test monitor should attempt to put each test participant at ease.
• The test monitor should impartially monitor the testing and not allow any vested interest to influence the test participant. The test monitor should avoid unintentionally influencing the test participant by way of voice or body language.
• The test monitor should treat each test participant as a unique individual, regardless of previous testing results.
• The test monitor should not rescue struggling test participants unless they are either totally stuck or becoming so frustrated as to endanger the continuation of the test.
• Document usability testing typically is only justified for critical deliverable documents.
• Test participants should be obtained from representative user groups.
• Document usability testing typically should be performed in a usability lab.
• The following are suggested techniques for measuring usability:
  • Average time to complete a given user task (e.g., locate a specific topic in the document or answer a specific question using the document).
  • Average percentage of tasks completed in a given time.
  • Average number or percentage of user errors.
  • Percentage of times that users express frustration or dissatisfaction.