When and how to run a preference testing?

That’s what preference testing is for. Don’t worry if you don’t know how to conduct one. We’ve put together this guide to help you out. Let’s get right into it! 

What is preference testing?

Preference testing is a research method for establishing a preference for a variety of options. Participants in the study are presented with 2-3 options and asked to choose which one they prefer. These tests are commonly used to determine which designs are the most aesthetically pleasing, understandable, or generate the most trust.

The Preference Test can be used to determine logo, icon, palette, design, mockup, content, copy, and text style preferences. This feedback contributes to the formation of a wider and larger image aimed at a deeper understanding of the optimum design for consumers. 

In a logo design, for example, you can present consumers with two different logo styles. The comments from users will not only tell you which designs they like but also why they choose them. The feedback from consumers will offer you useful information about your design, allowing you to make modifications until you find the correct option.

We also recommend you check out how preference testing works at UXtweak:

Advantages of preference testing

• A quick and effective way to get feedback from users
• Simulates what users do when purchasing products or services – choosing from a variety of options
• Simple and easy to understand, making them appropriate for even younger audiences

When should you conduct the preference testing?

Preference testing is most effective early in the design and prototyping stages before you spend too much time and effort on the development. It allows you to decide where to focus your resources, which choice is the most practical, and how you may improve it. 

When redesigning a product or website, preference testing can be used to evaluate the old and new versions of the design, as well as to compare your design to that of your competitors.

Quantitative feedback

The number of participants who selected each design will be displayed on the results page. This is a simple design comparison from which the statistical significance of the outcome may be calculated.

This is defined by statistical significance as the probability that the best design is picked and does not outperform the other designs by chance.

Qualitative feedback

Your qualitative input comes from interviewing participants about why they chose the design they did. You can divide their input into smaller groups to obtain a clearer idea of how many people had similar responses.

How to conduct a preference test?

Set your objectives

Firstly, make sure you understand what kind of data you want to collect from your users and why. Are you developing a new product or redesigning one that already exists? Do you have two or more different designs in mind?

Whatever your study goal is, write it at the top of your research board so that your test participants can see it.

It’s generally a good idea to provide participants with some background information when writing instructions. When comparing two different icons, for example, don’t just question, “Which icon do you prefer?” Include details like “Which icon do you prefer for incoming messages?” or “Which icon do you think is more reliable?” This level of specificity allows you to analyze your theory more precisely. 

However, avoid being too direct in your questions. People are often trying to be accommodating, so if they suspect you like one of the designs, it may influence them. They’re also less critical, even when told to be truthful, so it’s sometimes better to avoid asking direct questions to find out why they like one style over another.

Gather materials

Whether it’s logos, symbols, color palettes, or website designs, gather the items you wish to use for your preference test. In a preference test, two or three options are usually utilized. Remember that the more designs you include in a task, the more difficult it will be for your participants to choose and then justify their choice.

Choose a tool 

The Preference Testing tool from UXtweak has all of the features you’ll need to conduct thorough research. It’s simple to set up and use, and creating a UX preference test takes no more than 5 minutes.

Furthermore, the user interface is extremely user-friendly, making it an excellent choice even for complete beginners. We provide you with a free plan for small projects and subscribing to one plan you het access to all of our UX research tools. Here is where you may sign up and create your first preference test.

Here is how you can set up your preference test with UXtweak

1. Name your study and choose a language
2. Create tasks, where the respondents are supposed to express their preference for one of your designs
3. Design a variety of follow-up questions to collect qualitative data from your respondents based on your research objectives and the type and amount of elements you’re comparing.
4. Create your personalized messages to be included during the study, such as the welcome message, study instructions, thank you a message or you can use the default message
5. Add your unique branding and you’re all set!

Gather respondents

The respondents should reflect your customers, who will eventually be the ones to interact with the studied designs. The best approach is to speak directly to those who use – or are expected to use – the product or service and are aware of the context in which the designs are presented.

Don’t worry, you won’t have any difficulties with recruitment if you use UXtweak. You can send a link to your study via email or social media, or add a Recruiting Widget to your website to collect respondents right away. Alternatively, just use our User Panel to acquire responders for your UX study!

Analyze the results

The most basic piece of information you’ll acquire from a preference test is how many times each design was chosen as the preferred option. This data is presented in both absolute numbers and percentages.

It’s time for our favorite part: the analysis, once you’ve gathered all of the qualitative and quantitative input. We’ve put together a helpful guide to make the process easier for you!

Statistical importance of the data

Statistical significance is the likelihood that the best design is chosen and will not surpass the other design concepts by chance. The amount of significance will vary according to the sample size. Of course, the larger the sample size, the more meaningful the results. It will also depend on the degree of difference between the designs, as two identical designs will make distinguishing one version from another difficult.

To be remarkable, preference tests should be carried out with at least 20 users. The panel should not solely include people from your typical intended audience. However, it should also consider the specific circumstances and viewpoints that are useful to the design at hand. If you’re checking a design to attract potential leads, you could also try to draw in non-customers who fit your standard user’s profile in some way.

Limitations of preference testing

While preference testing is amazingly simple yet successful, it has its drawbacks:

• When you ask your users to pick a favorite design, for example, you don’t get any information on the size of the preference. It’s possible that they like both designs equally well, but one slightly more. Or they choose the one they loathe slightly less because they had to, but they still find it repulsive on its own
• Follow-up questions can help you learn more about why your users chose one choice over another, but individuals aren’t always adept at explaining why they prefer something
• Another difficulty is that some individuals mix up A/B testing and preference testing. In preference tests, different options are evaluated based solely on respondents’ internal preferences, while in A/B testing they are evaluated based on respondents’ performance and behaviour.

Merging preference tests with other techniques in hybrid research:

Because preference testing is constitutionally subjective and can only provide surface-level perspectives, it is best used in conjunction with other methods such as interviews and/or surveys to collect deeper insights and data that can help define the reasons behind participant decisions. It can also be used in tandem with A/B tests and behavioral analytics to gather more quantitative results.

So, what’s the point?

Regardless of its flaws, there is one justification to carry preference testing: allure. Attractiveness and usability are the 2 aspects of the UX coin, and both are required for a marketable product. 

Usability is concerned with the functionality of the user experience. It all boils down to whether or not users can accomplish numerous tasks with ease. Allure embraces all key variables of the user experience—the “something extra” or “wow element” that enhances the experience and can support the creation and maintenance of a strong association with the user.

Attractiveness is not just a pleasant extra thing to have in competitive markets; it is an essential component of a successful strategic plan. First, it benefits first impressions because a design with strong favorability frequently matches up with a more positive opinion of usability and overall reputation. This impression can give a product a serious advantage to prospective users in order to drive acceptance, and it can even persuade current users to excuse usability flaws.

Furthermore, high popularity is one of the characteristics that allow a company to set a higher price for its product. Customers will pay more for this factor of the experience because it strengthens their experience in a way that elicits an emotional response.

What is stopping you from trying preference testing?

When it comes to design research, preference tests are one the easiest ways to get feedback from your customers. And it is even easier with UXtweak. So, don’t wait around – register or log in to your UXtweak account and try your first preference test right away!