A study structure where the same participants test all conditions. Every participant interacts with both Version A and Version B. Statistically powerful but requires counterbalancing to control order effects.
Definition: A study structure where the same participants test all conditions. Every participant interacts with both Version A and Version B. Statistically powerful but requires counterbalancing to control order effects.
A Within-Subjects Design (also called a repeated measures design) has the same group of participants test all conditions. Every participant interacts with both Prototype A and Prototype B.
This design is statistically powerful because each participant acts as their own baseline. Individual differences (like tech-savviness or patience) are controlled for, making it easier to detect a true difference between designs.
Benefit: Highly statistically powerful and requires fewer participants
Drawback: Risk of order effects—participants may perform differently on the second condition due to practice, learning, or fatigue
To mitigate order effects, you must use counterbalancing: vary the order of conditions so that half the participants test A then B, and the other half test B then A. This allows you to statistically account for any order effects in your analysis.
Within-subjects design is excellent when:
For comparing subtle design variations where learning would transfer heavily, consider between-subjects instead.
A study structure where different groups of participants test different conditions. Group 1 tests only Version A; Group 2 tests only Version B. Eliminates order effects but requires more participants.
A technique for controlling order effects in within-subjects designs by varying the sequence of conditions across participants. Half test A→B; half test B→A.
This term is referenced in the following articles:
The idea that you only need five users is one of the most famous, and most misunderstood, heuristics in UX research. Here is what the numbers actually mean and when they apply.
The structure of your study, who sees what, and in what order, determines what conclusions you can draw. Understanding the trade-offs between study designs is fundamental to research craft.