Usability studies: How many participants are enough?

Usability tests are necessary if we are to create valuable and successful products. But conducting such tests can become a headache if they are not done efficiently by recruiting an adequate number of usability testing participants. 

The 5-user rule: Cost-effective and optimal usability testing

According to the reputable Nielsen Norman Group, ‘testing with 5 people lets you find almost as many usability problems as you’d find using many more test participants.’

The logic behind their ‘5-user’ suggestion is that as you test more and more people, you uncover fewer new insights at a higher cost. After testing up to 5 people, the same usability issues would continue to be mentioned by additional participants with very little significant change. 

So it’s economical and optimal to test just enough participants who can give you sufficient insights at a low cost. Thus, the 5-user rule.

Improve your product’s UX with UXtweak

The only UX research tool you need to visualize your customers’ frustration and better understand their issues

Register for free

When to use more or less than 5 participants

There are instances, of course, where the 5-user rule is not applicable: In some situations, the type of product tested requires a larger number of participants to obtain significant test results. For example, e-commerce websites can be much more complex than software products, so testing as few as five participants may only uncover 35% of their usability problems. 

You may also need more or less than 5 participants for your usability study if you:

Have different target audience segments 

When a product has different user groups who behave differently, you’ll need to test participants that represent each group. 

For instance, a marketplace app would need participants for both the buyer and seller profiles. In this case, you don’t need to have five participants for each of the different user groups since there would be overlapping observations, and you could use as few as 3 or 4 participants for each group.

Need quantitative data 

In conducting quantitative usability studies, your results must be statistically significant to obtain accurate insights. For this, you will need at least 20 to 40 test participants. 

Since the focus is on measurable metrics and not qualitative findings, you will need many participants to get enough data which can accurately predict the behaviors of your overall target users. 

Use an Agile UX process

In an Agile UX approach, you run multiple usability tests as the product develops since it is an iterative process. In this development style, you conduct tests, use the insights obtained to make changes, then test the new version – and on and on it goes.

So rather than using five or more participants for each test, you can run multiple tests with as little as 3 participants each since there would be overlapping insights and discoveries. Using fewer participants also helps to save costs on your study.

Pros of using 5±2 participants in your usability study 

There have been different optimal numbers of participants suggested by researchers over time. Having a minimum of 3 participants ensures that you capture diversity in your user group, while having as many as 7 participants ensures that you uncover almost all the usability problems in the product. 

So the ideal number of participants lies in a baseline of 3 to 7 participants for the following reasons:

1. It maximizes the law of diminishing marginal returns

In the usability testing context, the law of diminishing marginal returns states that as you test participants, your study will reach a point where increasing the number of participants will lead to lesser insights obtained. This graph put forward by the Nielsen Norman Group explains it better. 

(NN/g)

Here,

• Testing the first participant gives fresh new insights, 
• Testing a third participant uncovers many problems the first two participants may have missed, 
• Testing the sixth participant still uncovers new problems the first five participants may have missed, 
• But by the 12th participant, there would be little to no new insights as most usability problems have already been discovered.

So for practical reasons, testing between 3 to 7 participants gives you that sweet spot where you obtain sufficient insights without falling into the diminishing marginal returns trap.

2. Testing 3 to 7 users is cost-efficient

When developing a product, one main reason usability testing faces a lot of pushback from stakeholders is the cost of conducting usability tests. Depending on the type of usability test, the number of participants recruited, as well as the product type among other factors, study costs can range from as little as $250 U.S dollars to $10K U.S dollars and even much more. 

From recruiting agency fees to software costs and procuring incentives for your participants like gift cards, there are a lot of expenses and hidden costs incurred in running a single usability test. And since costs rise as the number of participants increases, testing a few participants helps you minimize costs and maximize your budget. 

3. Testing 3 to 7 users is time-efficient

Usability tests often take a lot of time, from setup to test sessions to analyzing the results. Depending on the number of participants and the type of study conducted – moderated or unmoderated, it can take a day or many days for completion.

It is estimated that conducting usability tests could take as long as 11 to 48 hours for only 5 participants. So rather than spending a lot of time testing many participants for your study, test only a few participants that would give you more or less the same results at a lesser time. 

Does the 5-user rule hold for different usability tests?

Usability tests include focus groups, surveys, A/B tests, and many more, so the type of usability study carried out plays a huge role in determining the number of participants selected. For instance, usability tests such as card sorting and eye-tracking have different optimal numbers of participants.

(UXtweak)

Card Sorting

Since card sorting is a generative method whose goal is to find out how people organize and find their way around content, you would ideally need more than 5 participants for the usability test if you’re to get representative results, since people’s mental models widely vary.

The Nielsen Norman Group proposes testing with 15 participants – three times more than the suggested standard of 5 participants, while other researchers insist on testing 20 – 30 participants to get sufficient insights for the test. The difference is that 15 participants would give you a correlation of 0.90 between study results and the actual real-life results, while 20 participants would give you a correlation of 0.93. 

However, if you’re dealing with a large project and have a lot of risks involved, you could test as many as 30 participants.

(UXtweak)

Eye-tracking

The eye-tracking study can be used to get both qualitative and quantitative data, and as a result, the type of data you want affects the number of participants you need. For qualitative data, Pernice and Nielsen suggest using 5 participants, and these participants can then be tested for a long duration. 

However, if you’re looking to generate heat maps, they suggest recruiting and testing as many as 39 participants. Since heat maps are classified as quantitative tests, several factors can affect their credibility, such as eye problems or other things that could affect people’s gazing behavior.

Survey

Survey research is a quantitative research technique for gathering data from a group of respondents. Although a survey is not a usability testing method, it also serves as a tool for collecting feedback and user research.

In a survey, the number of respondents also matters. If you don’t recruit enough, you have a chance of getting an inaccurate result. If you recruit too many, you will spend extra money. According to NN group’s research, the following types of studies require an average of 40 respondents. 

Is 30 respondents enough for a survey?

Yes, 30 respondents is enough for a survey and will most of the time allow you to gather enough information for accurate statistics. However, according to research, the optimal number of participants for a survey is 40.

How many respondents does your study need?

Depending on the study question, the test’s design, and the level of confidence that researchers want to place in the results, a different number of participants may be required for usability studies. A general rule of thumb is that 3 to 7 participants are a commonly recommended sample size for usability testing.

However, it is also important to understand that a larger number of participants may be required depending on the research method and the type of data you’re looking to gather. While in usability tests where you’re gathering qualitative data about the user’s behavior 3-7 participants may be enough, it’s different for the quantitative research studies. In card sorting, for example, to get accurate statistics you may need around 30 to 40 respondents and sometimes even more.

Conclusion: The perfect number doesn’t exist 

For decades, we have had many debates in the research world, and still, there is yet to be a consensus on the number of participants needed for usability studies. Several factors such as the product type or size, the stage in the usability life cycle, the tasks conducted, the skill of the researcher, and even the personality of the participants all impact the appropriate number of participants necessary for your study.

But one thing all UX researchers agree on is that there is no one-size-fits-all number of participants for your usability study. So always take your own context into consideration.