One of the issues that comes up time and again when planning user testing (whether usability testing, user experience testing, online in-depth interviews (IDIs) for qualitative research, etc), is whether to conduct the tests on a moderated or unmoderated basis.
Just like in politics, there are some who are firmly aligned with one type of testing or the other, and then there are the majority who see benefits for each methodology depending upon the context and the goals, and who may even wish to use both methodologies for the same project.
Defining Moderated and Unmoderated Testing
Moderated Testing: A moderator asks questions, instructs and directs respondents in terms of activities, and ensures that the respondent is guided in such a manner that the goals of the usability, user experience or IDI study are accomplished.
User Experience Designer Kevin Hoxie and team moderating a usability test.
Unmoderated Testing: Typically (although not always) conducted in a remote session, i.e.:, not in a lab environment, but in a respondent’s home or office or other “real life” location, sometimes referred to by market researchers as “field testing”.
Respondents have an assigned set of activities and questions which they go through with no assistance, direction or guidance from a moderator.
So what are the perceived benefits and disadvantages of each?
Moderated User Testing
Typically moderated testing is thought of as carrying the danger of moderator bias. Just like in a focus group, even a trained moderator is thought to transmit information subconsciously to the respondent, through the tone and timbre of voice instructions, facial instruction (if viewable by respondent), etc.
This is viewed as true regardless of whether the moderated test is a remote test or a lab test, although a local or lab setting is clearly more of an “un-natural” setting than a respondent home or office, whether moderated or not.
Moderated usability, user experience and IDI testing is also thought of as inherently more expensive and time consuming and logistically challenging than unmoderated remote user testing.
Unmoderated User Testing
Unmoderated usability testing, user experience and IDI testing is viewed as unable to cope with the flexibility of conditional logic and branching questions (e.g.: “Are you married?”; if the answer is yes the moderator asks respondent to look for a particular item, and if the answer is no, a different item).
Unmoderated testing is also viewed as lacking in the ability to probe into unexpected results and insights.
Are the above perceptions true?
Moderated:
Recent research has shown that it is extremely difficult to avoid transmiting information and potential biasing factors from a moderator to a respondent, whether in a group focus-group setting or in a one-on-one in-depth interview setting, whether in a lab or local setting, or during a remote moderated user test.
In addition, un-natural testing environments such as usability labs and focus group settings clearly bias results.
An altogether different question is whether this is a good thing or a bad thing. Focus group experts maintain that a group think setting is positive in terms of co-creation of research insights.
And therefore the question should be rephrased as: Is the bias of the setting and moderation a benefit or not?
And if it is a disadvantage for a particular project, is it enough of a disadvantage so as to offset the disadvantages of unmoderated usability, user experience & IDI testing?
Unmoderated:
It is a fact that unmoderated usability & user experience testing cannot allow for automatic probing of unexpected results.
However, because unmoderated usability & user experience testing is inherently less costly and time consuming than moderated user testing, a series of iterative testing sessions and workflows (with each activity/question design building upon the results of the previous user test results) can allow for an equivalent result to the real time probing allowed by moderated user testing.
And in terms of conditional logic and branching questionnaires, there are many unmoderated remote user testing platforms that allow for this kind of “result dependent” asset and stimuli presentation, direction, and questionning.
What about combining moderated and unmoderated user testing?
It is possible to combine both methodologies in a number of ways:
- Conducting larger sample sets, and a large number of activity/workflow/asset-stimuli iterations with unmoderated AND moderated usability and user experience testing platforms. The moderated usability and user experience testing is used as a means to help design the project (ensure design of unmoderated user testing is correctly set up) and to deepen and enrich the learnings through probing the critical work flows and issues that arise from unmoderated user testing.
- Conducting moderated usability & user experience testing with an unmoderated user testing platform. In this manner, a set of unmoderated user tests can be set up, with assigned dates/times, and the moderator and observers can “join in” if they are available and listen to the respondent and direct them or question them as they engage in the usability or user experience test.
This way, there is a fallback if scheduling conflicts prevent a moderator and/or observer from attending. And a set of recording deliverables that include both moderated and unmoderated sessions for easy analysis.
Conclusion
The preferences of research professionals for one type of testing over another are rapidly evolving as new platforms allow for combined moderated and unmoderated usability, user experience and IDI testing.
Some unmoderated user testing platforms allow for conditional and branching logic. And some allow for combined moderated and unmoderated user testing.
But the most important aspect for a researcher is to design a study as the sum of various iterations:
Some are more appropriate for moderated testing and some are more easily achievable and with less bias through unmoderated testing.
However, the goal is the insights and market validation learned through all of the test iterations (including combined unmoderated and moderated user testing), which is unlikely to be achieved cost-effectively with a single methodology.
Originally published on September 11, 2017
Q&A
Unmoderated usability testing is a method where participants complete tasks on a product or website without real-time guidance or interaction from a facilitator. This approach is typically conducted remotely and allows users to provide feedback independently, offering insights into how they navigate and experience the interface in a natural setting.
In remote unmoderated usability testing, participants access the testing platform from their location, complete predefined tasks, and share their feedback through recorded videos, screen captures, or surveys. This method is cost-effective and enables a larger pool of participants across different locations.
Unmoderated user testing provides several advantages: • Cost-efficiency compared to moderated methods. • Scalability, allowing feedback from a diverse audience. • Natural user behavior since participants work in their own environment without a moderator’s presence.
Moderated usability testing involves a facilitator guiding participants through tasks and asking questions in real-time. Unmoderated testing, on the other hand, allows participants to complete tasks independently without live supervision. Each approach has its strengths: moderated testing offers deeper qualitative insights, while unmoderated testing is faster and more scalable.
Unmoderated testing is ideal for: • Quick feedback on usability issues. • Testing with a larger and more geographically diverse audience. • Scenarios where real-time interaction isn't critical. Moderated testing is better for: • Exploring complex user behavior or scenarios. • Gathering detailed qualitative insights through direct interaction. • Testing prototypes or early-stage designs where guidance may be needed.
An unmoderated usability study evaluates how users interact with a product or service by having them perform tasks without direct supervision. These studies often leverage remote platforms that capture user interactions, offering insights into usability, navigation, and user experience.
Unmoderated testing has some limitations, such as: Limited ability to probe deeper into user behavior during the session. Dependence on clear task instructions to avoid participant confusion. Potential lack of context for nuanced usability issues that a moderator could identify.
Several platforms support unmoderated user testing, including tools that provide task recording, survey capabilities, and data analytics. Popular choices include Userlytics, UserTesting, and Maze.
To maximize the effectiveness of an unmoderated test: • Clearly define tasks and objectives for participants. • Provide detailed instructions to minimize confusion. • Use tools that record user interactions and provide actionable analytics.
