Quantitative User Testing vs Qualitative
Your funnel report has flagged the same drop for three sprints in a row: shoppers reach the shipping step of checkout and leave. Then your team watches a handful of recorded sessions, and participant after participant scrolls past the delivery estimate looking for a total cost that never appears. One source told you where the problem sits, and the other showed you why it happens, which is the practical difference between qualitative vs. quantitative UX research.
Most product decisions need both kinds of evidence, at different moments, and in different amounts. Below, we compare what each approach measures, how many participants each one needs, and which methods belong where. We finish with a study plan that runs them in sequence and a way to report the results so stakeholders can act on them.
TL;DR
- Qualitative UX research shows why users struggle by observing a few sessions directly; quantitative UX research measures how many and how much from a larger sample.
- Nielsen Norman Group (NN/g) suggests about 5 participants per round of qualitative usability testing and about 40 for most quantitative studies.
- Report small studies as counts with context, since a percentage from 5 or 10 participants carries a wide margin of error.
- The strongest plans run both in sequence: numbers to locate the problem, sessions to explain it, then numbers again to confirm the fix.
Compare Qualitative vs Quantitative UX Research at a Glance
The clearest way to tell the two apart is how the evidence reaches you. In NN/g’s guide to choosing UX research methods, qualitative studies collect behavior and attitudes firsthand, by watching or listening to participants, while quantitative studies collect them through an instrument such as a survey, a task metric, or an analytics tool.
| Qualitative UX research | Quantitative UX research | |
| Question it answers | Why are users struggling, and what would help? | How many succeed, how long it takes, how much better the new version performs |
| Data | Observed behavior, think-aloud comments, quotes, video clips | Task success, time on task, error counts, ratings, clicks |
| Typical sample | About 5 participants per round | About 40 participants for most studies |
| Study conditions | Flexible; the researcher can probe and adjust | Fixed tasks and conditions so results can be compared |
| Analysis | Themes and findings ranked by severity | Descriptive statistics and confidence intervals |
| Best stage | Any stage, especially while the design is still changing | A working product or prototype you want to benchmark |
| Main risk | Generalizing from a few users | Knowing the size of a problem without knowing its cause |
The same study script can feed either approach. What changes is the claim you can make at the end, so the choice starts with the decision you need to support.
What Qualitative UX Research Shows
Qualitative UX research explores users’ needs, experiences, and behavior through methods such as interviews, observation, and usability testing. The protocol stays flexible, so a researcher can ask a follow-up question the moment someone pauses, or reword a task that stops producing useful evidence. The output is a set of findings, each tied to the moments that produced it and ranked by how badly the problem blocks the task.
Qualitative testing is also where the quotes and clips come from that make a readout memorable. A short clip of one participant explaining that they expected free shipping until the final screen can account for a drop in conversion faster than a chart can.
Both moderated and unmoderated sessions produce qualitative data. In a moderated session, the researcher guides the participant through the tasks, observes, and asks follow-up questions about what they did and what they expected. In unmoderated usability testing, participants complete the study on their own devices while the session and their think-aloud commentary are recorded, and both formats run well as remote usability testing.
What Quantitative UX Research Measures
A quantitative study runs fixed tasks with enough participants that the numbers describe the wider audience. Behavioral metrics such as task success rate, time on task, and error rate record what participants did. Questionnaires such as the System Usability Scale (SUS) and the Single Ease Question (SEQ) record how easy the experience felt.
A SUS score runs from 0 to 100 but does not represent a percentage, and a score of about 68 is treated as average. Customer Satisfaction (CSAT) and Net Promoter Score (NPS) belong to the same attitudinal family, measuring satisfaction and loyalty, so they sit beside usability metrics rather than standing in for them.
Numbers also need a reference point. NN/g’s comparison of quant and qual usability testing uses a 60% completion rate to make the point: on its own, the figure cannot tell you whether the design is doing well or badly. Quantitative work is therefore usually comparative, setting this release against the last one or your flow against an agreed target.
On the Userlytics platform, quantitative testing covers tree tests, card sorts and surveys along with time-on-task and success or failure measures, with SUS, SEQ and NPS available as templated questions. For a closer look at survey design, analytics and benchmarking, see our guide to quantitative UX research methods.
Separate What Participants Say From What They Do
Qualitative vs. quantitative is one axis. NN/g’s research-methods landscape adds a second one that settles most debates about where a method belongs: attitudinal research captures what users say about their beliefs and preferences, and behavioral research captures what they actually do. The two often disagree, which is exactly why the distinction matters when you choose a method.
| Qualitative (observed or heard directly) | Quantitative (measured through an instrument) | |
| Behavioral: what participants do | Moderated usability testing; unmoderated think-aloud sessions; field observation | Task success and time on task at scale; tree testing; first-click testing; A/B testing on live traffic; product analytics |
| Attitudinal: what participants say | User interviews; open-ended survey questions; diary studies; focus groups | Closed survey questions; SUS and SEQ; NPS and CSAT |
Surveys show why the second axis helps. A survey is attitudinal by design, and it becomes qualitative or quantitative depending on its questions: rating scales produce numbers you can compare, while open text produces reasons you have to read.
Two methods change sides depending on how you run them. A card sort with a handful of participants explaining their groupings out loud is qualitative, and the same sort with a large sample analyzed for how often cards were grouped together is quantitative. Unmoderated usability testing works the same way, producing findings from a few think-aloud recordings and metrics from a large sample with fixed tasks.
A tree test is a useful contrast. Participants search a stripped-down, text-only hierarchy of your navigation, so the method is behavioral, and with a large enough sample it reports how many found each item and where the rest went instead.
Place your planned method on both axes before you write the screener. If the cell does not match the claim you need, change the method before you recruit, and browse our overview of UX research methods for alternatives.
Size the Sample to the Claim You Want to Make
Sample size follows from what you plan to say when the study closes. “The delivery cost is hard to find” needs a different base than “the new checkout raised completion.”
| Study type | NN/g guidance | What it can support | What it cannot support |
| Qualitative usability testing | About 5 participants per round; 3 to 4 per group when testing two groups | The main problems in a flow and why they happen | How common each problem is across your audience |
| Several small qualitative rounds | Three rounds of 5 instead of one round of 15 | Evidence that each fix worked before the next round | A benchmark score |
| Quantitative usability testing | About 40 participants for most studies | A success rate within about 15 percentage points of the true rate, at 95% confidence | Why participants failed |
| Lower-precision quantitative study | 15 participants, accepting a 20% margin of error at 90% confidence | A directional read when budget is tight | A verdict between two designs that perform closely |
| Card sorting | At least 15 for a qualitative sort; 30 to 50 for a quantitative one | Groupings participants expect, and with the larger sample, how consistent they are | Whether participants can find items in the finished menu |
The 5-participant guidance comes from a mathematical model by Jakob Nielsen and Thomas Landauer, which estimated that five participants uncover about 85% of a design’s usability problems, provided the design still has plenty of problems left to find. The same article suggests splitting a budget for 15 participants into three studies of 5, so each round can check the fixes from the round before.
The 40-participant figure comes from a calculation in NN/g’s summary of quantitative sample sizes. To estimate a success or conversion rate within 15 percentage points at 95% confidence, you need 39 participants, which NN/g rounds up to 40.
Larger samples take more recruiting, so many teams reserve quantitative rounds for decisions that genuinely need a number. On the Userlytics platform, those studies can recruit from a panel of 2M+ participants in 150+ countries, or from your own customers through BYOU (Bring Your Own users).
Report Small Studies as Counts With Their Limits
A percentage from a small study looks far more precise than it is. In NN/g’s article on why numbers from qualitative studies mislead, 7 of 10 participants complete a task, yet the success rate for the whole audience could fall anywhere between 39% and 90% at 95% confidence. Flexible protocols add more variation, since the researcher probes and adjusts differently from one session to the next.
Write the count and the base instead. For instance, “4 of 5 participants missed the delivery estimate” tells a stakeholder exactly what the team saw, and one more clause can name the limit: the round documents the problem observed in the sessions and leaves its frequency to a larger study.
Choose the UX Research Method That Fits the Decision
Start with the question the team has to answer, then pick the approach and the method. The table maps common product questions to the evidence that answers them.
| The question in front of the team | Lean | A UX research method that answers it |
| Why do users abandon onboarding at the same screen? | Qualitative | Moderated or unmoderated think-aloud sessions on that screen |
| How many participants can finish checkout without help? | Quantitative | Unmoderated usability testing with task success and time on task, about 40 participants |
| Did the redesign perform better than the version it replaced? | Quantitative | The same tasks and metrics run on both versions |
| Can users find “Returns” in the new menu structure? | Quantitative, then qualitative | A tree test at scale, then sessions on the paths that failed |
| What do buyers expect a pricing page to answer? | Qualitative | User interviews plus think-aloud reviews of the current page |
| How common is a motivation that came up in interviews? | Quantitative | A survey with closed questions built from the interview themes |
| Which of two concepts should move into design? | Both | Preference questions at scale plus sessions that capture the reasons |
| Where should the next research round focus? | Quantitative | Analytics or a benchmark study that shows the weakest tasks |
Match the Approach to Your Design Stage
The design stage narrows the choice further. NN/g separates formative research, which shapes a design while it is still changing, from summative research, which assesses a design once it is built.
- Discovery: interviews and field observation show what users are trying to get done before anything is drawn.
- Design iteration: short qualitative rounds on clickable prototypes catch the biggest problems while they are still cheap to fix.
- Pre-launch validation: a quantitative round on the near-final build sets the baseline you will compare against later.
- Post-launch tracking: repeated benchmarks and analytics show whether the experience holds up from release to release, with qualitative sessions whenever a metric moves.
Qualitative work stays useful at every stage. Its job shifts from shaping the design early on to explaining why a number moved later.
Not sure which study design fits your research question? Userlytics’ UX consultants can plan the study, moderate the sessions, analyze the data, and present recommendations to your stakeholders. Explore UX Consulting
Combine Qualitative and Quantitative UX Research in One Study Plan
Combining the two works best as a sequence, where each round answers the question the previous one raised. Three orders cover most situations, and a single study can also carry both kinds of data.
Start With the Numbers When You Know Where Users Struggle
Analytics or a benchmark study points to the step where completion falls. A qualitative round of about 5 participants on that step shows what they misunderstand or cannot find. Once the fix ships, the same metric tells you whether the change worked.
Start With Sessions When You Do Not Yet Know What to Measure
Interviews or exploratory sessions surface a reason or a workaround you had not expected. A survey with closed questions built from those themes then shows how widespread each one is. The roadmap can weigh them by frequency instead of by how vivid the quote was.
Collect Both Kinds of Data in the Same Study
One study can carry both, with one caution. NN/g’s comparison lists think-aloud under qualitative methodology and notes that quantitative studies usually run without it. Narrating each step slows participants down, so keep timed tasks silent and ask open questions once each task ends.
Moderated sessions can mix the two as well. A researcher can run a card sort inside a live session and then ask why the participant placed certain cards together, which adds the reasons behind each grouping.
For unmoderated studies, Userlytics’ AI Moderation feature adds follow-up questions to the activities you select. You write the objective in plain language; the AI evaluates each response against it and asks a follow-up in the study’s language, and each follow-up response is kept with the activity it came from. The feature brings probing to unmoderated research, while moderated sessions remain the place for open conversation.
See Both Approaches in a Three-Round Model Study
Here is how a team might plan a checkout redesign with both kinds of evidence. The plan is an illustrative model study and reports no results.
| Round | Approach | Setup | What it answers |
| 1. Baseline | Quantitative | About 40 participants, unmoderated, three checkout tasks on a staging site with test payment details; task success, time on task, and SEQ after each task | Which task performs worst, and by how much |
| 2. Diagnosis | Qualitative | Two rounds of about 5 participants with think-aloud, focused on the weakest task, with a fix tested between rounds | Why participants stall, and whether the first fix removes the problem |
| 3. Re-measure | Quantitative | About 40 new participants, identical tasks, wording, metrics, and staging setup | Whether the redesign moved the metric by more than the margin of error |
Keep rounds 1 and 3 identical in tasks, wording, and audience, since a change in any of them weakens the comparison. Round 2 is free to adapt from session to session, because its job is understanding.
See How a Real Study Counted Qualitative Evidence
A Userlytics x Sage study on AI visibility and B2B software buyers shows both kinds of evidence in one report.
- Study design: 60 unmoderated sessions with prospective finance-software buyers, across three markets.
- Counts reported with their base: Sage showed up in the AI assistant’s first answer in 41 of the 60 sessions and in 47 sessions overall. Its website confirmed or strengthened the AI-formed impression in 45 of 60 visits, or 75%.
- Observed reasons: The sessions showed where trust broke down, for example, when missing pricing led AI assistants to fill the gap with outside price estimates the buyers had no way to check.
- A measure the team can repeat: Userlytics assessed each journey with its six-dimension AI Readiness framework and gave Sage a scoring instrument so the research can be rerun over time.
The whitepaper presents its findings as directional, without claiming they represent all B2B software buyers. The counts say how often something happened in this sample, and the observed sessions explain why.
Report Both Kinds of Evidence So Stakeholders Act on Them
A study changes a decision only when the users making it can read the evidence quickly. These steps keep both kinds of findings readable in one report.
- Lead with the decision. Open with the question the study answered and what the team should do next.
- Include the sample size. Put the number of participants next to each figure, and keep percentages for samples large enough to carry them.
- Pair each number with a clip. A short video of one participant hitting the problem gives the metric a face, and the metric shows the clip reflects more than one session.
- Rank findings by severity. Look for recurring patterns, and also investigate severe issues encountered by individual participants before deciding what to change.
- State the limit. One sentence on what the study cannot show answers the objection before anyone raises it.
On the Userlytics platform, the Qual and Quant AI Reports follow the same split. Qual Reporting turns a completed study’s sessions into themes, key findings, strategic areas, and recommendations with supporting evidence and links to sources across sessions. Quant Reporting compiles the charts from a study’s Metrics tab into a shareable report, pairing each chart with an AI-generated summary.
The researcher reviews what the AI organized, checks the evidence behind it, and decides what to present to stakeholders. During analysis, AI Insights answers plain-language questions about a study, with every answer citing the sessions, timestamps, and annotations it is based on.
Avoid These Mistakes in Qualitative vs Quantitative UX Research
Most problems with mixed evidence come from asking a study for a claim it was never designed to support. Watch for these seven.
- Generalizing from a small sample. A five-person study can collect quantitative data, but estimates will carry considerable uncertainty. Report the counts and sample size, and avoid presenting them as reliable estimates for your wider audience.
- Reporting percentages from a handful of sessions. “60% of participants” from a sample of 5 implies a precision it cannot carry; write “3 of 5” and state the limit.
- Treating every survey as quantitative. Open-ended answers are qualitative data that have to be read and coded, and closed questions only measure the options you thought to offer.
- Swapping attitudinal metrics for behavioral ones. A rising NPS indicates a more favorable balance of promoters to detractors; however, it does not show whether users can complete a task.
- Reading SUS as a percentage. A SUS of 80 sits above the average of about 68, and it does not mean the product is 80% usable.
- Timing tasks while participants think aloud. Narration can slow users down, so collect time on task in silent tasks and gather the reasons afterward.
- Exclude unusual sessions without watching them. A participant who finishes very fast or fails every task may reveal a real problem, so review the recording first and exclude a session only when the evidence clearly shows the participant ignored the instructions or never engaged, noting why.
Your Next Step in Qualitative vs Quantitative UX Research
Choosing between qualitative and quantitative UX research gets easier once the claim is written down. Before your next planning meeting, pick one decision on the roadmap and write the sentence you would need to say to settle it.
If that sentence contains “how many,” “how much,” or “better than,” plan a quantitative round of about 40 participants. If it contains “why” or “what gets in the way,” start with a qualitative round of about 5, and book the follow-up round now so the first study’s open question already has a slot.
See how one study plan can combine both the participant feedback and usability metrics. Book a free demo.
FAQs
Quick answers to the questions teams ask most about qualitative vs quantitative UX research.
Userlytics
Since 2009, Userlytics has helped enterprises, government organizations, nonprofits, agencies, and startups understand their users and optimize their user experiences. Through our UX research platform, global participant panel, and educational resources, we make user research more accessible while sharing insights, expert perspectives, and conversations that help shape the UX community.
Schedule a Free Demo