One of the issues that comes up time and again when planning user testing (whether usability testing, user experience testing, online in-depth interviews (IDIs) for qualitative research, etc), is whether to conduct the tests on a moderated or unmoderated basis.
Just like in politics, there are some who are firmly aligned with one type of testing or the other, and then there are the majority who see benefits for each methodology depending upon the context and the goals, and who may even wish to use both methodologies for the same project.
First, a few definitions:
Moderated Testing: A moderator asks questions, instructs and directs respondents in terms of activities, and ensures that the respondent is guided in such a manner that the goals of the usability, user experience or IDI study are accomplished.
User Experience Designer Kevin Hoxie and team moderating a usability test.
Unmoderated Testing: Typically (although not always) conducted in a remote session, i.e.:, not in a lab environment, but in a respondent’s home or office or other “real life” location, sometimes referred to by market researchers as “field testing”.
Respondents have an assigned set of activities and questions which they go through with no assistance, direction or guidance from a moderator.
So what are the perceived benefits and disadvantages of each?
Moderated User Testing
Typically moderated testing is thought of as carrying the danger of moderator bias. Just like in a focus group, even a trained moderator is thought to transmit information subconsciously to the respondent, through the tone and timbre of voice instructions, facial instruction (if viewable by respondent), etc.
This is viewed as true regardless of whether the moderated test is a remote test or a lab test, although a local or lab setting is clearly more of an “un-natural” setting than a respondent home or office, whether moderated or not.
Moderated usability, user experience and IDI testing is also thought of as inherently more expensive and time consuming and logistically challenging than unmoderated remote user testing.
Unmoderated User Testing
Unmoderated usability testing, user experience and IDI testing is viewed as unable to cope with the flexibility of conditional logic and branching questions (e.g.: “Are you married?”; if the answer is yes the moderator asks respondent to look for a particular item, and if the answer is no, a different item).
Unmoderated testing is also viewed as lacking in the ability to probe into unexpected results and insights.
Are the above perceptions true?
Moderated:
Recent research has shown that it is extremely difficult to avoid transmiting information and potential biasing factors from a moderator to a respondent, whether in a group focus-group setting or in a one-on-one in-depth interview setting, whether in a lab or local setting, or during a remote moderated user test.
In addition, un-natural testing environments such as usability labs and focus group settings clearly bias results.
An altogether different question is whether this is a good thing or a bad thing. Focus group experts maintain that a group think setting is positive in terms of co-creation of research insights.
And therefore the question should be rephrased as: Is the bias of the setting and moderation a benefit or not?
And if it is a disadvantage for a particular project, is it enough of a disadvantage so as to offset the disadvantages of unmoderated usability, user experience & IDI testing?
Unmoderated:
It is a fact that unmoderated usability & user experience testing cannot allow for automatic probing of unexpected results.
However, because unmoderated usability & user experience testing is inherently less costly and time consuming than moderated user testing, a series of iterative testing sessions and workflows (with each activity/question design building upon the results of the previous user test results) can allow for an equivalent result to the real time probing allowed by moderated user testing.
And in terms of conditional logic and branching questionnaires, there are many unmoderated remote user testing platforms that allow for this kind of “result dependent” asset and stimuli presentation, direction, and questionning.
What about combining moderated and unmoderated user testing?
It is possible to combine both methodologies in a number of ways:
- Conducting larger sample sets, and a large number of activity/workflow/asset-stimuli iterations with unmoderated AND moderated usability and user experience testing platforms. The moderated usability and user experience testing is used as a means to help design the project (ensure design of unmoderated user testing is correctly set up) and to deepen and enrich the learnings through probing the critical work flows and issues that arise from unmoderated user testing.
- Conducting moderated usability & user experience testing with an unmoderated user testing platform. In this manner, a set of unmoderated user tests can be set up, with assigned dates/times, and the moderator and observers can “join in” if they are available and listen to the respondent and direct them or question them as they engage in the usability or user experience test.
This way, there is a fallback if scheduling conflicts prevent a moderator and/or observer from attending. And a set of recording deliverables that include both moderated and unmoderated sessions for easy analysis.
Conclusion
The preferences of research professionals for one type of testing over another are rapidly evolving as new platforms allow for combined moderated and unmoderated usability, user experience and IDI testing.
Some unmoderated user testing platforms allow for conditional and branching logic. And some allow for combined moderated and unmoderated user testing.
But the most important aspect for a researcher is to design a study as the sum of various iterations:
Some are more appropriate for moderated testing and some are more easily achievable and with less bias through unmoderated testing.
But the insights & validation learned through all of the test iterations (including combined unmoderated and moderated user testing) is the goal and is unlikely to be achieved in a cost effective manner with a single type of methodology.
Originally published on September 11, 2017
