A Simple Guide to Cognitive Walkthrough

When user goals are known, but the flows and screens required to get there are still forming, going to external testing seems a disproportionate burden on resources, and relying only on individual internal opinions is risky. Cognitive walkthroughs give a reliable, proven structure to the review discussions which teams often already have as part of their inherent process, giving project outcomes greater validity and certainty.

Although the cognitive walkthrough method is tailored to internal participants, the essential concepts can also be adapted for larger, external group testing.

If you’re working in or learning about digital product development, especially if your project is new, complex, or requires interaction by a very broad range of users, then cognitive walkthroughs are here to help you reach the optimal product more efficiently.

Why you should conduct a Cognitive Walkthrough

Cognitive walkthroughs are a great way to bring some rigor to your discussions and evaluations of how well a new product or feature will help users. As a new product is being developed, to some extent, everyone in the team is concerned with creating the optimum way for users to achieve the goals which the product enables.

Countless meetings can be spent discussing options and setting the direction for how a product will serve users – a process which is heavily influenced by personalities, tastes, and opinions. While that may lead to some great results, it lacks the validity of a more formal, consistent, and hopefully objective process, which is exactly what cognitive walkthroughs bring.

Using cognitive walkthroughs as your early-stage testing method should give your project a more thoughtfully built foundation, while at the same time being quick and very low-cost to undertake. Your cognitive walkthroughs should hopefully uncover issues which would be increasingly costly to address as the project evolves.

Furthermore, as a method which uses rough prototypes, and often participants from within your own org, cognitive walkthroughs are a very efficient way to sense-check novel ideas and complex flows.

Many UX testing methods can be used at any stage throughout a product’s development and lifetime. However, as a product or feature matures, it naturally takes on visual style and layout, which means that later-stage testing will be influenced by UI presentation and not just focused on essential interactions.

Cognitive walkthrough is well-suited to very rough wireframe prototypes, because in a classic walkthrough, the testing group is internal, and therefore may be more forgiving of ‘unprofessional’ prototypes. Also, cognitive walkthroughs are conducted in a small group with a presenter, who can quickly explain things which may be missing from the prototype.

Cognitive walkthroughs take the participants out of their day-to-day working routine so that they can take a step back and evaluate the product they are working on. In this more reflective state of mind, new ideas can be uncovered, as well as aspects of the product which need improvement may be found, since they are often harder to see when working closely on project delivery tasks.

How are Cognitive Walkthroughs different from other usability tests?

At a glance, cognitive walkthroughs may appear similar to a prototype user test, or other usability tests. In some settings, they may, in fact, be very similar, but if you’re following the documented methods, there are a number of differences.

Heuristic Testing compared to Cognitive Walkthrough

Heuristic Evaluation (Nielsen and Molich, 1990; Nielsen 1994) is a method created to test how well an interface performs relative to a set of defined principals, which are accepted to be the basis of good user experience. 

Cognitive walkthroughs are aimed at evaluating new features, systems, and approaches, and also have a different structure. Heuristic Evaluation can provide feedback on how easy your product is to use, while cognitive walkthroughs are a formal review of how easy it is to learn. 

What can be evaluated with a Cognitive Walkthrough?

Cognitive walkthroughs provide insights to any aspect of a product which requires users to perform a new task. They are suitable at a whole-product, feature, or single-screen level.

Typically, the technique is used when: 

• The product, feature, or screen requires users to learn how to use it
• The UX addresses a problem in a way that is different to existing norms
• The user is required to complete complex workflows, or
• It is likely that the user is unfamiliar or completely new to this kind of flow

When to use Cognitive Walkthroughs in product development

As the outcomes can be high-impact and the prototypes can be rudimentary, cognitive walkthrough is well-matched to conceptual work phases. Furthermore, as a cognitive walkthrough is so easy to set up and administer, the technique can be used throughout product design and development, even on deployed products at a relatively high frequency.

Just be aware that the outcomes are shaped by the testing group, which is classically a limited, internal set.

Improve your product’s UX with UXtweak

The only UX research tool you need to visualize your customers’ frustration and better understand their issues

Register for free

Cognitive Walkthrough participant group options

In principle, a Cognitive Walkthrough is done by a small group of people who work in the product organization. This may be designers, product managers, engineers, IAs, department leads, in fact, anyone who may have a relevant perspective. Sticking with this group should make walkthroughs simple to organize and true to their original form and benefits. 

However, it is also possible to conduct a cognitive walkthrough with larger groups, as well as with testers from outside the product organization. As long as the cognitive walkthrough participant, question, and method structure are maintained, you should still gain excellent qualitative and quantitative results. 

This approach takes longer to organize and conduct, and is unlikely to be conducted quickly or often. Moreover, it’s important to source a pool of relevant users, which in the case of cognitive walkthrough means that they are new to the media being tested and are in the product target audience. 

Where did Cognitive Walkthrough come from?

For as long as people have been inventing new systems, they have tested how efficient those systems are for new users. Early software engineers created processes to review and test the sequence of actions that users need to take in order to interact with their projects, which is the origin of cognitive walkthrough for product development. 

In the 1990s, a group of researchers created a walkthrough methodology initially in reference to assessing ‘walk-up-and-use’ interfaces, such as ATMs. During the 90s, they evolved the method to be directly orientated to websites and coined the term ‘Cognitive Walkthrough’. Today, that same format is used in the evaluation of all kinds of digital products, with different practitioners adding their own refinements and variants.

The Cognitive Walkthrough method

Cognitive walkthrough is a formal method: it has a defined structure for whoever participates, what their roles are, how the session runs, and what questions are to be asked and discussed. The method is tried and tested and should yield meaningful results and confidence. 

However, as your experience with cognitive walkthrough grows, you can alter aspects of the format to suit your workflow and situation. Be careful, though, to maintain the integrity of the general structure to help ensure the quality of outcomes. 

The participants in a Cognitive Walkthrough

In its original form, a cognitive walkthrough comprises participants who represent the broader product team. 

This includes:

• Product experts who know all about this product 
• Domain experts who know all about this market
• One or more UX practitioners
• One or more Engineers

In the modern context, there is also room for product managers, product owners, UI designers, UX writers, information architects, UX researchers, and anyone else on a product squad. Certain organizations may also involve senior managers, founders, and even those in sales and marketing.

And, in actual practice, the participants are often only a group of product or UX designers!

The four questions always used in Cognitive Walkthroughs

The fundamental characteristic of a cognitive walkthrough is the questions which are to be discussed. Their original wording is still useful and relevant; however, there is a simplified and more universal version from Chris Kimmer which we prefer. 

You can use either, or you may find further options for all or any of the four questions that suit you better.

Cognitive Walkthrough planning

A small amount of planning for your cognitive walkthrough will help the session run well and maximize your chances of meaningful results. A good cognitive walkthrough agenda will cover the following information.

1. Overview

• Purpose – The purpose of the workshop, what is being evaluated, and why? 
• Audience – Who are the personas that the evaluators should empathize with?
• Requirements – List the prototypes, tools and spaces to be used
• Rigor – Are there any challenges to research rigor and objectivity, and how will they be overcome?
• Outcomes – Outline what artifacts will be created and their use.

2. People

List the specific people required to do the workshop, and what role each will play. It is acceptable for the same person to perform multiple roles. A cognitive walkthrough requires the following roles to be performed. 

• Facilitator – Runs the session.
• Presenter – Operates the prototype.
• Recorder – Takes notes and collates them after the session.
• Evaluators – Discuss the four questions.

3. Subjects

Make a list or table of each sequence or goal to be evaluated. Note if there are any aspects specific to each sequence, such as a different persona, scenario, or prototype.

Preparing a cognitive walkthrough prototype

Although we’re only touching on it lightly here, preparing the prototype or prototypes for a walkthrough workshop should be the main preparation activity. It may not be a painstaking task though; remember, these prototypes only need to be as finished as is necessary to demonstrate the concept.

Once you have completed the above agenda and your prototype, you’re ready to conduct the workshop. Typically, the agenda is circulated either before or at the beginning of the session, however, be mindful of the potential of this kind of information to influence the results. You should always strive to create an agenda, but if you feel it will skew the workshop outcomes, there is no need to circulate it.

How to conduct a Cognitive Walkthrough workshop

The final aspect of the cognitive walkthrough method is the workshop itself. The following sequence is recommended for a classic ‘internal’ workshop.

1. The Facilitator starts by talking through the goals, roles, personas, rules, and optionally,  the agenda.
2. The Presenter steps through the interface, stopping at each decision point.
3. The Evaluators discuss each of the four questions and agree to an action at each step.
4. The Presenter takes the agreed actions.
5. The Recorder notes success or failure, and any qualitative feedback.
6. The Recorder summarizes and circulates the outcomes after the workshop.

Ideally, your workshop will run closely to this format, but in practice, you should expect it to vary a lot from the format. The outcomes are still valid, and more so if you’ve been able to keep to the four questions.

Workshops with external participants will need to be adapted from this format to suit the specific setting.

Tips for conducting a Cognitive Walkthrough workshop

• Overemphasize personas to the evaluators – it’s important that they’re empathetic with your actual new users.
• Keep laptops closed – we need full attention!
• Do not try to justify the prototype – participants should talk about what they encounter, not why it is the way it is, with strong focus on user reaction. 
• When testing with internal participants it can be helpful to really limit the number of participants.
• Conversely, a larger group of external participants can create more meaningful results.
• Test microcopy options, not just button placement.
• For an external walkthrough, consider using a mixed-method approach, such as a survey or interview to add depth to observations.

Conclusion: The benefits of Cognitive Walkthroughs in collaborative work

Applying a cognitive walkthrough structure to the normal practice of stepping back from your work and discussing it with colleagues adds diligence and improved potential to impact quality at a comparatively low cost. 

From their genesis in the early-days of software programming to their fascinating development for walk-up-and-use systems, as well as their current use in assessing digital products, walkthroughs are a proven, valuable, and enduring tool. 

In order to streamline your research and empathize with users better, register for your free UXtweak account and give a try to our various testing tools!

Sources:

[Clayton Lewis, Peter Polson, Cathleen Wharton, John Reiman, Testing a Walkthrough Methodology for Theory-Based Design of Walk-Up-and-Use Interfaces, 1990 https://www.cs.helsinki.fi/u/thusu/opinnot/kaytarv/artikkelit/p235-lewis.pdf ]

[Clayton Lewis, Peter Polson, Cathleen Wharton, John Reiman, The cognitive walkthrough: A practitioner’s guide, 1994 https://www.colorado.edu/ics/sites/default/files/attached-files/93-07.pdf ]