Krug, Steve. Rocket Surgery Made Easy. Berkeley, CA: New Riders Publishing, 2010. Print.
"Usability Testing:" Watching people try to use what you're creating/designing/building (or something you've already created/designed/built), with the intention of (a) making it easier for people to use or (b) proving that it is easy to use.
Quantitative test: Attempting to prove something through measurement.
Qualitative test: Attempting to gain insights to improve something.
These tests are of the qualitative kind.
I watched the demo he mentions in chapter 2: http://www.sensible.com/rocketsurgery/index.html
- Most tests last about an hour
- Have a simple site up like Google, something that won't distract the user while he introduces himself and explains the process.
- Use his script for the introduction.
- Bring up the site you want to test.
- Ask user to say what they're initial thoughts are, what the site is for, what it does, who it's for, what they would likely learn from the site. Don't click on anything.
- Ask user to do specific tasks. Read it aloud and give them a printed copy. Give them a scenario so they know "who they are supposed to be." Let them go through the process, clicking and all. Remind them to verbalize if needed.
- If they suggest a new feature, ask them how they would visual it working.
- Ask them to look around to see if there's anything else that might be relevant (if there is, and they missed it).
I noticed that while I watched the demo, and the woman was clearly going down the wrong path, I kept trying to tell her what to do in my head, even though I had nothing at stake. This could mean bad things for when I do a test for my own system. I'll have to watch that my frustration doesn't get displayed. It turned out her straying was useful in finding other issues. You have to let them reach their own conclusion, but if they get off topic, you can prompt them to revisit the issue once they reach the end of the thought process they were following.
Testing an existing site:
- Useful because you'll learn what you are currently doing wrong so you don't make the same mistakes as you redesign.
Testing other peoples sites (like other instructor evaluation sites):
- It's a free learning resource to see what works and what doesn't in the same genre.
Testing the sketch on a napkin:
- Hand someone, anyone the sketch and say "Can you tell me what to make of this? What do you think this is supposed to be?" Don't ask "Do you like this?" or "What do you think of this?" Ask them what they think it IS. Listen to what they say, and you can ask a few probing questions like "What do you think term 'x' means?"
- Lets you know if people "get it."
Testing wireframes:
- Wireframes are schematic diagrams of a page. Shows where different kinds of content will go, the relative prominence of things like headings, and the navigation devices like menus and search.
- Wireframes tests focus on interaction.
- Make up tasks, usually all related to navigation.
- Ask "How would you find..." or "What would you expect to see when you click on this link?"
- Tells if things are where people expect to find them, if the category names make sense, if the navigation is clear, etc.
Testing page design:
- Checks to see if the visual design has introduced any usability issues after we know the navigation, etc works via the wireframe tests.
- Start with home page, lead user through visual treatments of the pages and ask them to do a narrative of each one.
Testing working prototypes and beyond:
The People:
- It generally isn't important to test with people who are representative of your target audience. It's nice, but valuable data can be had from anyone.
- Three people per round is enough.
- He has a discussion on how to recruit users, screen them, & follow up.
- Tell users ahead of time what to expect, and schedule the time place. Send an email confirming the session along with reiterating the details and providing the non-disclosure agreement.
- After the meeting, send a thank you note.
- Have a backup "emergency" user in case one doesn't show.
- You cannot reuse the same people. One test per person unless it's a different application or site.
The Tasks:
- First, choose the tasks.
- The trick is making sure your list of important tasks are the things your users really want to do.
- Decide which ones to test in this month's round of testing.
- Typically a session lasts 50 minutes with 35 minutes of the participant actually doing tasks.
- There may be just one long task or as many as 10 small ones.
- If a given user is particularly fast, have "filler" tasks, like doing something on a competitor's site.
- To decide which tasks to try, ask yourself: What are your most critical tasks? What keeps you awake at night? What does you other user research suggest may not be easy to use?
- Second, expand the tasks into scenarios. Little stories that add context. It gives your user character, motivation, what they need to do, and a few details. "You are..." and "You need to..." and provides information they need to complete the task (like the username and password to the test account).
- Don't give any clues in the scenario. Avoid terms they will encounter on the site.
- You may want to place two restrictions: "Don't use search" and "Stay on this site"
- Pilot test the scenario to make sure they are clear, complete, and unambiguous. Use a co-worker, family, or friends. Anyone will do.
- Print out the scenarios in two formats: All on one page for yourself, and one per sheet for participants. Don't number the tasks.
Action:
- Use the checklists he provides so details don't get missed
- Facilitator: The person in the room with the participant. Part Tour Guide, part Therapist.
- Observer: Watch, learn, take notes. Write down the 3 most important usability problems they saw in that session. Suggest questions for the facilitator to ask the participant. Come to the debriefing session.
- The computer and screen recording software.
- A standard computer mouse, people shouldn't have to figure out how to move the pointer.
- Will likely need a microphone as the participant won't always face the computer.
- Follow his pre-test prep checklist (pg 68)
- Be aware of your biases and don't let them influence the participant in either direction.
- Participants should leave the room in no worse shape than when they entered.
- We may need to get permission from our Institutional Review Board to ensure we're meeting ethical standards.
- Make it a spectator sport. Seeing is believing. Watching as a group generate more ideas and discussion.
- Could use "Go To Meeting" for screensharing with observers.
The Debriefing:
- Prioritize the most important issues that can be fixed in the next month.
- Summarize the tests, issues, and decisions in a follow-up email.
Recommendations:
- KEEP IT SIMPLE!
- Do a round of testing one morning a month with three users. Debrief over lunch, spend the month fixing the issues, repeat next month.
- It needs to be a regularly scheduled thing or it won't happen.
- Tweak, don't redesign. Then verify.
- First impulse is to add something, consider taking something away.
Most common problems:
- Starting off on the wrong foot. The home page does a bad job of orienting the user.
- Failure to shout. People are moving too fast to notice subtle visual cues.
Summary:
- Keep it simple, this is quick and easy testing.
Remote Testing:
- Use screen sharing software and speakerphone for the participant (or VOIP)
- GoToMeeting is a good option for screen sharing.
- Could use "UserTesting.com" to test with just a user or two. Send them a task.
Maxims:
- A morning a month.
- Start earlier than you think makes sense.
- Recruit loosely and grade on a curve.
- Make it a spectator sport.
- Focus ruthlessly on a small number of the most important problems.
- When fixing problems, always do the least you can do.
TODO:
- Look into software that will record everything the user does on the site, like where they move their mouse and what they clicked on, etc. If it can record sound too, that'd be perfect. Don't need to record the actual user, so they don't need to worry about being identifiable.
- Set up a regularly scheduled test day once a month. Use co-workers, student help, and instructors, staff, or advisory board members who have volunteered. Take my laptop over to them with the test site up and recording software and just do the test. Do it on paper if needed and record with my iPhone.
- Look at and test other related sites that evaluation University instructors. Like RateMyProfessor. We can see what works and what doesn't. (follow process from chapters 5-9). Have each user do the same task on 2 to 3 different sites.
- Look into "remote testing" (chapter 14)
1/09/11 pg 1 - 13 didn't record time.
1/10/11 (10pgs, 13m = 1.3mpp) minutes per page
1/10/11 (11pgs, 15m = 1.4mpp)
1/11/11 (13pgs, 40m = 3.1mpp)
1/12/11 (26pgs, 16m = .6mpp)
1/13/11 (17pgs, 17m = 1mpp)
1/18/11 (45pgs, 36m = .8mpp)
Total: ~150m = 2h 30m
