Attempting to reach actual reality

Dec 17, 2015

Let’s do a usability test where we’ll take some participants and observe them with a product. We’ll take our findings and after a bit of analysis we’ll implement something. We’ll be able to base the development on solid research because we documented what the participants did and said. And that’s good.

Here are some things to bear in mind about usability tests:

Ironing out the creases

It’s good to practice. Practicing your test allows you to iron out any issues before they arise1. Imagine sitting down with your participant, you run through the first task annnd … they don’t understand what you’re asking! Oh no! All that time you spent crafting the perfect non-leading task and you have to rephrase what you’d like them to do. Will it be as effective as your original? Planning also means you’re not leaving everything to the last minute - printing stuff, getting equipment ready, sorting the room out and that.

Observation not conversation

As unnatural as it is, encouraging your participants to talk aloud2 about what they’re thinking gives great insight into why people find things easy or challenging. Also, try and avoid talking too much yourself. By the very nature of being a UXer, we want to help people achieve stuff so this is one that I personally really have to concentrate on. Avoid treating the session as a conversation and turn it into an observation3. Nielsen Norman Group has some good advice with their Talking With Participants article which covers three techniques (Echo, Boomerang and Columbo) that can be used in the test itself.

The whole truth?

So that’s all good. We’ve practiced and performed and now we can analyse our results.

Well hold on there Johnny-two-socks, the thing is sometimes participants might not be telling us the truth. Dun. Dun. Dunnn … There are several reasons why your results might be a little off the mark.

In UX Booth’s article Bridging the Gap Between Actual and Reported Behavior they have some insights into what could be going on in people’s heads.

1 - How would you like that answered?

Sometimes we answer questions in a way we believe the person who asked the questions would like them to be answered. Similarly, we can answer questions how we believe to be socially acceptable … do we really exercise as often as we say we do?

2 - I’ll do what you do

There have been cases of participants straying completely away from what’s clearly a correct answer just to fit in with peers around them:

“[In the Asch conformity experiments] Research subjects were shown several lines of different length, and asked to select the two of equal length. The subject didn’t know that everyone else answering the question was a confederate of the researchers. The planted participants would sometimes unanimously give an incorrect answer and, 37% of the time, the real research subject agreed with them.”

Kat Matfield, Bridging the Gap Between Actual and Reported Behavior

3 - Self Image

People also say what they want to be true and perhaps not what is actually true, viewing their capabilities through rose tinted specs. Not so much delusions of grandeur, more wishful thinking:

“ … our self-image doesn’t match up to reality. But that self-image still influences how we respond to questions about our intentions, rationale, or preferences”

Kat Matfield, Bridging the Gap Between Actual and Reported Behavior

4 - Where and when

People respond differently in different contexts. For example there are more suitable environments than an office room to evaluate the usability of a sports app. The results from one environment would be quite different to the other4.

It’s not just about how people subconsciously answer questions. The very fact that people are being observed encourages changes in behaviour. The Hawethorne effect demonstrates that if a participant is set a task to complete while being watched by someone they might change their approach to completing that task. For example during an observation the participant may read through instructions leading to a more successful task completion, whereas if they were alone at home they may simply ignore them and fail the same task. There is also the notion that if a participant is asked to achieve a task then they presume that task can be completed. Whereas if they were using a new product outside of the test environment they wouldn’t know that what they wanted to achieve was possible and would enter a journey within the product to find out5.

5 - Ultracrepidarianism

People may also give you an answer to a question even if they they have limited knowledge of the subject just because the question was asked of them. And, if asked why they did something, sometimes participants simply may not know so they make up a reason, or offer their opinion anyway6. This is known as Ultracrepidarianism.

So ...

Preparation will help things go smoothly and by applying some simple techniques during your test you can make the bits you have control over more successful. However the fact is, it’s inevitable that your participants will show some of the characteristics described above. The important thing is to be aware of them and to identify when they happen. You can’t control a participant's subconscious or how much value they place on social acceptance and if you do become aware of anything which might skew things, note it down and advise how it may affect the results.

Usability testing tips

Here are some top tips for running a usability test courtesy of usability.gov:

1 - Be polite and make participants feel comfortable

2 - Help participants understand that you’re after feedback on the design. It’s not them that is being tested

3 - Remain neutral (refer to the Talking With Participants article)

4 - Avoid helping and leading the participant

[5 - Encourage participants to think aloud]

6 - Take good notes - there should be two of you, one to ask questions and the other to record

7 - Measure both performance (success, time, errors) and subjective (self reported satisfaction and comfort) metrics