Moderated Usability Testing 101

While moderated usability testing is not the quickest or most simple format to undertake, it is certainly among the most valuable. And, with modern tools, it has become much more accessible to product teams than ever before. 

An introduction to Moderated Usability Testing

Usability testing means the application of test methods, which provide feedback on how easy a design is to use, for participants who represent actual users. Typically, participants are asked to complete tasks using the interface being tested, with both qualitative and quantitative information captured. 

In moderated usability testing, a facilitator or moderator conducts the test directly with the participant. The moderator, who, if you’re reading this, is probably you, acts as a neutral guide, working to a prepared script, but able to ask and answer new questions in response to the participant. In contrast, unmoderated usability testing is conducted with participants receiving written or recorded instructions and automatic feedback.

Remote and in-person options

Both unmoderated and moderated usability testing can be performed remotely with the help of online usability testing tools. Some of the possible benefits include participants becoming more comfortable, as well as true-to-life in their familiar surroundings. The wide uptake of work-from-home over recent times has supported remote testing, as more testers and participants now have good video call setups in their homes and offices. 

Moderated usability testing may also be undertaken in-person at any UX research lab, other professional locations, or in the participant’s home or workplace. Other in-person approaches include guerilla testing (sourcing participants at random in a public space) and observation (the moderator does not intentionally interact with the participant). However, guerilla and observation are unlikely to be the best approach if you’re aiming for a deep understanding of real users in their everyday state-of-mind.

Kinds of usability tests

When selecting a test method for moderated usability testing, it’s worth considering which will best serve your goals, either individually or combined. In this context, ‘test method’ refers to the tasks and instructions you will be giving the participants. There are many different methods, with usability tests often classified as follows:

Assessment studies are concerned with how well a design supports user tasks, and how satisfied users are with the design.

Comparative studies involve at least two options being tested, either as design alternatives or competitor comparison.

Explorative studies gather open-ended feedback, ideas, and opinions. This can appear similar to an assessment study, but is likely to be more conversational, may not involve external participants, may be undertaken at an earlier project stage, and may lead to more fundamental changes in the test subject.

Amongst the array of UX tests, there are many which are not compatible with moderated usability testing, from unmoderated 5-second tests, first click tests, acceptance tests, and user session recordings, to focus groups, surveys, eye tracking, and heatmaps.

The Benefits of Moderated Usability Testing

In many respects, Moderated Usability Testing is the gold standard for gaining user insights through testing. There are four overriding benefits to moderated usability testing.

1. Moderated usability testing yields high quality results

• Body language, subtle behaviors, and nuanced language are all recorded.
• Pain points should be very clear, and user/customer journeys can be seen close-up.
• Both qualitative and quantitative data can be gathered (though usually from a smaller pool).
• The moderator can form a rapport with the participant, eliciting the most honest feedback.

2. Moderated usability test participants are more engaged

• Participants can be asked questions reactively and be encouraged to articulate a train of thought.
• The risk of participants speeding through the test is minimized.
• The moderator/participant relationship keeps them both involved in the test.

3. Moderated usability testing is better for participants

• The moderator can help participants navigate the tasks if there is any confusion.
• The participant can be brought back to the topic if there are distractions or side-discussions.
• There can be privacy and security control to protect the participant (and the organization running the test).

4. Moderated usability tests are straight-forward to set up

• Moderated usability testing is more time-efficient than interviews and focus groups, and ideally the insights are more accurate.
• Modern tools like UXtweak make moderated usability testing quick to set up and seamless to run, with immediate results and lower costs.
• Moderated usability testing can be remote or in-person, or both.
• It is possible to test prototypes with limited functionality, because the moderator can guide the test.
• Moderated usability testing is suitable for everything from the most simple click tests and style perceptions, to the most complex flows and tasks.
• Moderated usability testing can be used with any usability test method where the user can be observed.

Try Moderated Usability Testing with UXtweak

The only UX research tool you need to visualize your customers’ frustration and better understand their issues

Register for free

When is the best time to use Moderated Usability Testing?

There is a commitment of effort, time, and money to organize a moderated usability test, and it’s generally true that the closer to completion a design is, the more accurate the test results will be to the experience of the released product. So, although some form of moderated usability testing can be run at any stage of product development or deployment, testing when a higher-fidelity working prototype is available is likely to yield the most accurate and valuable results.

That notwithstanding, the speed of prototype creation and testing supported by tools like Figma and UXtweak now puts moderated testing within easier reach more often than it has been in the past.

When might Moderated Usability Testing not be right?

There are a handful of scenarios where moderated usability testing is probably not the best approach.

These include:

• When there is only a very small amount of time to prepare and run a test.
• When the mere presence of a moderator may influence the results.
• When the moderator is unskilled or unsupported, which risks the validity of results and more broadly the organization’s brand reputation.
• When a large set of quantitative data is the desired result, yet the available resources would not provide this.
• For in-person testing, when there is no suitable space available, members need to travel excessively, or there is a very small monetary budget.

Preparing for a remote Moderated Usability Testing

Successful moderated usability testing sits on a foundation of thoughtful preparation. Here is a run-down of the key components for remote testing preparation.

Write a plan

It’s important to outline your study at the beginning with a carefully written plan. 

• Define your product users/personas and the key goals to be tested.
• Determine the profile and quantity of participants.
• Describe the testing method/s to be used.
• List who will perform each role. These include moderator, note taker, and verbal and non-verbal communication observer. They can all be the same person, but ideally there is one person for each role.
• List the time, money, tech tools, and spaces needed.
• What are the risks to the validity of outcomes and how will they be mitigated?
• Outline the nature of the outcomes to be produced and how they will be circulated.

Prepare the script and prototypes

Write the text which the moderator will read to the participant in the testing session, and do several trial-reads with people who can help to refine it. Prototypes can be as simple as wireframes, though normally it would be a clickable, shareable prototype in a tool like Figma. Testing platforms like UXtweak can be utilized to host and record interactive behavior.

Find relevant participants

Source from a pool which represents your users closely. Participant pools are available online, and we suggest recruiting directly with UXtweak Moderated Testing Tool. 

Set up a smooth testing environment

Conduct and record tests with a computer-based video call tool. Zoom tends to be the most universal, while Google Meet and Microsoft Teams are better if your participants are already using them. Ideally you will also have a method for capturing the interactions.

The most basic way is to have the participant share their screen in the video call; however, running your prototype through UXtweak gives you more accurate, deeper insights into their behavior and provides additional analytics after each test.

Running a Moderated Usability Test

Moderated usability testing sessions follow a sequence of events generally based on this structure.

1. Introduction to participants

• • Be clear with the participant that you are looking for honesty; they don’t need to be polite or worry about offending anyone.
  • Ask the participant to think out loud. This is important as it is likely to provide many rich, insightful moments. 
  • Ask the participant for their permission to record the session and fulfill any other legal requirements before proceeding.

2. Warm-up questions

• • These are general ‘small talk’ questions, which should be brief and non-invasive, such as “how has your day been so far?”
  • Try to put the participant at ease and build some personal rapport by responding to the things they share while listening more than talking.
  • The questions will help you gauge their state of mind and what’s going on for them right now.
  • Informally, also ask about the participant’s experience with the test subject, as most often they will have used the product previously.

3. The actual tasks

• • The participant is instructed and then performs the tasks which are being tested.
  • Don’t ask questions which might lead the participant to take certain actions or opinions.
  • Avoid answering questions, especially those which might tell the participant what to do.
  • Take notes about everything seen and implied, but don’t draw any conclusions yet.
  • Remember to take note of gestures and facial expressions.

4. Final questions and discussion

• • This is an opportunity for the participant to ask questions, as well as for both moderator and participant to discuss the test.
  • The discussion can go broad if necessary and comfortable, but keep in mind that there is a purpose to the session and the participant may have limited time available.
  • Further insights can still be gathered through this less formal chat.

5. Session review

• • After every few sessions, the testers can discuss their observations as a team.
  • Note where there are conflicting observations, however, do not revise these unless it becomes apparent that the tester has misunderstood something.

6. Report preparation and circulation

• • Most audiences for reports are time-poor, so include charts and visual aids/photos.
  • Avoid using any jargon.
  • Clearly state the conclusions and any suggested/preferred actions.
  • Consider how and when to present the report for the most appropriate impact.

Moderated Usability Testing script example

Not sure how to write a script for your test? Here is an example you can adapt to suit your needs. We suggest you edit it to make the language natural for you, as well as add pauses to encourage interaction.

And don’t just read out the questions – otherwise it might sound like a robotic monologue!

Intro

• Hi, I’m (moderator’s name), I’m a researcher at (organization name).
• We also have (other tester’s names) from (team names) on the call. They’re interested in this session, but for the most part, it will just be you and me talking.
• We’re trying out some new ideas we have for (product name), and we wanted to get your feedback on those. I imagine you’re pretty busy, so thank you again for helping us with this test.
• Before we get started, is it alright if I record this session? It will make it easier for me to focus on what’s happening. You won’t be named, the recording won’t be shared outside (organization name), and it will be deleted once we’ve completed the current testing program.
• That’s great, I’ve started the recording, thank you.

Warm up questions

• What does a regular day look like for you at the moment?
• Are there other people you work with a lot?
• How often do you use (product name)?
• How do you use (product name)?

Usability test instructions

• To start, I’ll briefly explain how we do the session so that we’re on the same page.
• As a participant, you’ll perform a set of tasks while I observe and take notes.
• It’s not a test of you, it’s a test of how good our product is to use. We want your most normal behavior, and we definitely want your most honest comments – don’t worry, you won’t hurt our feelings!
• You can ask questions during the test, but I’ll only give brief answers – there will be time a bit later to talk about what you thought and discuss questions more fully.
• We’re really interested in understanding how people think while they use our product. So, it will be super helpful if you can please think out loud: tell us in your own words what you’re thinking as you do the tasks. This can be easy to forget, so if it’s OK with you, I’ll remind you from time-to-time.
• For each part of the session, I’ll give you a scenario, and then I’d like you to do exactly what you would do normally in that situation.
• We’ll be testing out a prototype, which means it’s not totally functional like a finished product would be. There are a lot of things in the prototype which are inactive, but the main things we expect you’ll want to use are functional. You’ll only need a mouse, there is no need to use your keyboard.
• Do you have any questions before we start?

Scenario

A brief outline of a situation that the participant needs to envisage being in.

Task

A brief outline of a task the participant needs to perform.

Repeat the Scenario and Task instructions for each different user goal being tested.

Final questions and discussion

Ask any responsive questions here. You can go in-depth during this stage.

• Did anything feel different today?
• Did you notice anything in particular?
• Did you have any preference in the options shown?
• Do you have any questions?

Outro

• Would you be comfortable if we contacted you again from time to time if there are other things we’re looking for your feedback on?
• Thank-you for meeting with us today, it was really helpful.

Try Moderated Usability Testing with UXtweak

The only UX research tool you need to visualize your customers’ frustration and better understand their issues

Register for free

Moderated Usability Testing tips

Your skill as a moderator will grow as you conduct more tests.

Here are 10 tips to help you get off to a good start:

1. Ask participants to think out loud and accept as valid anything they verbalize.
2. Keep asking why to get to the root of an issue (the first response may not be the most helpful).
3. Observe everything fanatically.
4. Make the scenarios realistic for participants.
5. Make your instructions very clear – where things can be misunderstood, they usually are.
6. Practice the technology to be completely comfortable before any testing starts.
7. Sign in early so that you don’t miss the participant, and they feel welcomed to the session.
8. Recruit excess users, as there may be no-shows and extra respondents are not a problem.
9. Run a pilot test with any people at hand and repeat until all the bugs are ironed out.
10. Make building a good rapport with participants your priority.

Conclusion

When run well, moderated usability testing provides a high-quality mix of observed natural behavior and vocalized feedback, which is invaluable for product development. Moderated usability testing requires time to undertake, but conducting tests remotely is simple and low cost, while engaging participants in their own environment puts them at ease, giving you the best chance of capturing rich, accurate insights.

Register for your free account and try Moderated Usability Testing with UXtweak!