P-Value

A p-value tells you how likely your observed result would be if the null hypothesis were true—that is, if there were no real difference or effect. A small p-value means the data would be surprising under the assumption of no effect.

What P < 0.05 Actually Means

If p = 0.03, there is a 3% chance of seeing a result this extreme assuming the null hypothesis is true. It does not mean:

• There is a 97% chance the effect is real
• There is a 3% chance you are wrong
• The effect is large or practically meaningful

The 0.05 threshold is a convention, not a natural law. Fisher originally proposed it as a rough guide, not a binary decision rule.

Why P-Values Are Not Enough

• Large samples detect trivial effects: With enough participants, you can get p < 0.001 for a difference so small it has no practical impact
• Small samples miss real effects: A genuine improvement might produce p = 0.12 simply because you did not have enough data
• P-values say nothing about magnitude: They tell you whether an effect exists, not whether it matters

What to Do Instead

Report effect size alongside your p-value. A statistically significant result with a tiny effect size is not worth acting on. A non-significant result with a large effect size may warrant a larger study. Use confidence intervals to communicate the range of plausible values—far more informative than a single yes-or-no threshold.