Skip to content

Unmoderated vs Moderated Usability & User Experience Testing

 By Userlytics
 Sep 11, 2017
 2756 views
Home  »  Blog   »   Unmoderated vs Moderated Usability & User Experience Testing

One of the issues that comes up time and again when planning user testing (whether usability testing, user experience testing, online in-depth interviews (IDIs) for qualitative research, etc), is whether to conduct the tests on a moderated or unmoderated basis.

Just like in politics, there are some who are firmly aligned with one type of testing or the other, and then there are the majority who see benefits for each methodology depending upon the context and the goals, and who may even wish to use both methodologies for the same project.

First, a few definitions:

Moderated Testing: A moderator asks questions, instructs and directs respondents in terms of activities, and ensures that the respondent is guided in such a manner that the goals of the usability, user experience or IDI study are accomplished.

moderated testing. Unmoderated vs Moderated Usability & User Experience Testing.

User Experience Designer Kevin Hoxie and team moderating a usability test. Source: kevinhoxie.com

Unmoderated Testing: Typically (although not always) conducted in a remote session, i.e.:, not in a lab environment, but in a respondent’s home or office or other “real life” location, sometimes referred to by market researchers as “field testing”.

Video-in-Video

Respondents have an assigned set of activities and questions which they go through with no assistance, direction or guidance from a moderator.

So what are the perceived benefits and disadvantages of each?

Moderated User Testing

Typically moderated testing is thought of as carrying the danger of moderator bias. Just like in a focus group, even a trained moderator is thought to transmit information subconsciously to the respondent, through the tone and timbre of voice instructions, facial instruction (if viewable by respondent), etc.

This is viewed as true regardless of whether the moderated test is a remote test or a lab test, although a local or lab setting is clearly more of an “un-natural” setting than a respondent home or office, whether moderated or not.

Moderated usability, user experience and IDI testing is also thought of as inherently more expensive and time consuming and logistically challenging than unmoderated remote user testing.

Unmoderated User Testing

Unmoderated usability testing, user experience and IDI testing is viewed as unable to cope with the flexibility of conditional logic and branching questions (e.g.: “Are you married?”; if the answer is yes the moderator asks respondent to look for a particular item, and if the answer is no, a different item).

Unmoderated testing is also viewed as lacking in the ability to probe into unexpected results and insights.

Are the above perceptions true?

Moderated:

Recent research has shown that it is extremely difficult to avoid transmiting information and potential biasing factors from a moderator to a respondent, whether in a group focus-group setting or in a one-on-one in-depth interview setting, whether in a lab or local setting, or during a remote moderated user test.

blind justice. Unmoderated vs Moderated Usability & User Experience Testing.

In addition, un-natural testing environments such as usability labs and focus group settings clearly bias results.

An altogether different question is whether this is a good thing or a bad thing. Focus group experts maintain that a group think setting is positive in terms of co-creation of research insights.

And therefore the question should be rephrased as: Is the bias of the setting and moderation a benefit or not?

And if it is a disadvantage for a particular project, is it enough of a disadvantage so as to offset the disadvantages of unmoderated usability, user experience & IDI testing?

Unmoderated:

It is a fact that unmoderated usability & user experience testing cannot allow for automatic probing of unexpected results.

However, because unmoderated usability & user experience testing is inherently less costly and time consuming than moderated user testing, a series of iterative testing sessions and workflows (with each activity/question design building upon the results of the previous user test results) can allow for an equivalent result to the real time probing allowed by moderated user testing.

circle. Unmoderated vs Moderated Usability & User Experience Testing.

And in terms of conditional logic and branching questionnaires, there are many unmoderated remote user testing platforms that allow for this kind of “result dependent” asset and stimuli presentation, direction, and questionning.

logic

What about combining moderated and unmoderated user testing?

It is possible to combine both methodologies in a number of ways:

  1. Conducting larger sample sets, and a large number of activity/workflow/asset-stimuli iterations with unmoderated AND moderated usability and user experience testing platforms. The moderated usability and user experience testing is used as a means to help design the project (ensure design of unmoderated user testing is correctly set up) and to deepen and enrich the learnings through probing the critical work flows and issues that arise from unmoderated user testing.
  2. Conducting moderated usability & user experience testing with an unmoderated user testing platform. In this manner, a set of unmoderated user tests can be set up, with assigned dates/times, and the moderator and observers can “join in” if they are available and listen to the respondent and direct them or question them as they engage in the usability or user experience test.

This way, there is a fallback if scheduling conflicts prevent a moderator and/or observer from attending. And a set of recording deliverables that include both moderated and unmoderated sessions for easy analysis.

Conclusion

The preferences of research professionals for one type of testing over another are rapidly evolving as new platforms allow for combined moderated and unmoderated usability, user experience and IDI testing.

Some unmoderated user testing platforms allow for conditional and branching logic. And some allow for combined moderated and unmoderated user testing.

But the most important aspect for a researcher is to design a study as the sum of various iterations:

Some are more appropriate for moderated testing and some are more easily achievable and with less bias through unmoderated testing.

But the insights & validation learned through all of the test iterations (including combined unmoderated and moderated user testing) is the goal and is unlikely to be achieved in a cost effective manner with a single type of methodology.

circle. Unmoderated vs Moderated Usability & User Experience Testing.

Interested in UX Testing?

Data Visualizations


Latest Posts

Blog
September 30, 2024

UX Checklist for Websites: Crucial Steps Before Going Live

Optimize your site before launch with our detailed UX checklist for websites. Learn how UX research can make your website successful.
Read More
Webinar - ULX SWOT Analysis & UX Metrics
Webinar
October 10, 2024

Bridging the Gap: Leveraging Market Research Insights in UX Research

Join our webinar Leveraging Market Research Insights in UX Research and discover how it can drive to better product design.
Read More
Whitepaper
March 5, 2024

The State of UX in 2024

Discover 'The State Of UX In 2024' report: Key insights on UX research evolution, roles of product managers, and future trends.
Read More
Why CEOs Need To Care About UX – Before It’s Too Late
Podcast
September 10, 2024

Why CEOs Need To Care About UX – Before It’s Too Late

Why CEOs Need To Care About UX - Before It's Too Late. Interview with Userlytics' Founder & CEO Podcast
Read More

Didn’t find what you were searching for?

Ready to Elevate Your UX Game?