Details of a usability study

Typical in-person study

A facilitator sits with an individual participant, who works through a set of tasks. Observers sit in a nearby room, watching and listening through a video connection. We take notes and have a debriefing session after each participant, and again at the end of the day.

Typical remote study

Each participant is at his or her own location, the facilitator is in another location, and the observers may be together or separate. We use software such as GoToMeeting to broadcast and record the sessions remotely, and Morae to broadcast locally.

Watch a brief video showing the observer’s view of a study.

An example of what observers see during a usability study

All the details

Planning the study: Developing schedules and tasks, recruiting participants

We’ll help you decide what to test, set up and run the study, and interpret the results with you.

Deciding what to test

What do you need to learn? Do you have paper prototypes or a finished product? Do you want to test all features or just new ones? Can participants use their own data or will we make it up?

Tasks and questionnaires

With your input and review, we’ll prepare a task list and a set of questions to ask before and after each session. The best tasks represent what participants do in a real work situation.

Recruiting participants

We recruit participants based on personas (user categories). We can invite your customers, work with a recruiting agency or even use Craigslist to find participants. This is one of the most important parts of a study, and it can take a couple of days or a couple of weeks.

Compensation

We give participants a gift for taking part in a study. It might be cash, a gift certificate or a copy of your software.

Location

We can run a study in different locations:

  • Using your office makes it easiest for your team to observe sessions
  • A usability lab is good if you don’t have space or don’t want participants in your office
  • Remote testing via the Internet lets us schedule participants who live or work anywhere
  • Going out to visit participants lets us learn about their work environments

Individual sessions: what each participant does

Each session has one participant and lasts between 45 and 90 minutes. Through careful observation and limited interaction with the participant, we see how well the product meets each customer’s needs.

Pre-test questionnaire

A few simple background questions give us details we use to customize the session. We give the compensation at this time to show that it doesn’t depend on particular results.

Tasks

We don’t force participants to work in any order, and we adjust the tasks to make the session as realistic as possible.

Post-test questionnaire

We end with questions about each feature the participant used. Comparison with responses from the pre-test questions can show whether the product matched the user’s conceptual model of the task. While many questions use a rating scale, we don’t do statistical analysis of the answers; the rating scale is a way to stimulate discussion.

Debriefing

Your team's observations are very important. We review notes and comment with the observers at the end of each session, and again at the end of the day.

Observers

Everyone on your product team should watch some or all of the sessions to learn first hand what your customers think. It can be a very humbling experience, but it’s always educational.

Observers take notes during a session. They can send messages to the facilitator with suggestions or questions.

Using Morae or GoToMeeting lets observers watch and listen from another room or another location. One observer takes notes that are synchronized with the audio and video, making analysis of the study much easier.

The usability report: the outcome of the study

The report is brief. It identifies problems and makes recommendations to fix them. We categorize and set priorities for the recommendations and meet to discuss them.

Usability reports cover these topics in addition to product-specific topics:

  • Navigation
  • Layout and visual design
  • Conformance with platform UI guidelines
  • Terminology
  • Usefulness of error messages
  • How well the product matches the user’s conceptual model of the task.

We can also create a short highlights video. This is particularly useful to show team members or managers who could not attend the study.