The Logic of a Test
A test starts from the null hypothesis (nothing is going on: , ) and an alternative . Compute the test statistic . The p-value is the probability, IF were true, of a result at least this extreme. Small (below , usually ) is evidence against .
A test is a proof by contradiction with probability in place of certainty. Assume nothing is going on; ask how surprising the data would then be; if very surprising, doubt the assumption. What a -value does NOT measure is the probability that is true — it conditions on , it does not evaluate it. And a large -value never proves ; it only says the data are compatible with it.
A coin lands heads times in flips. : ; : . Under , SE and . Two-sided, both tails count: -value . At , reject : the coin looks biased.
A maker claims its batteries last hours on average; a consumer group suspects less. : (the claim, the "nothing unusual" value); : (what the group wants evidence for). The hypotheses are about the population mean, never about , and they are fixed before looking at the data.
The coin's -value is below — reject — but above — do not reject. The evidence is the same; the standard of proof differs. That is why reporting the -value itself is more informative than a bare "significant".
Reading as "a chance the coin is fair". The -value assumes the coin is fair and asks how often such data would occur; the probability that the coin IS fair is a different question that a test does not answer.
For the same coin, test : . What is the -value?
Show the answerHide the answer
Only the upper tail now counts as "at least as extreme": , half the two-sided value.
Errors and Power
A Type I error rejects a true (probability ). A Type II error fails to reject a false (probability ). Power is the probability of correctly rejecting; it grows with sample size and effect size, and shrinks when is made smaller.
The cutoff for rejecting is set by the null distribution alone, so that only of its area lies beyond. Whether the test then detects a real effect depends on where the TRUE distribution sits relative to that cutoff. Moving the cutoff outward (smaller ) protects against false alarms but misses more real effects; a larger sample narrows both curves and pulls them apart, raising power without costing .
Test : against : with , , . SE , and we reject when . If in fact , then . The power is about ; a real effect of points is missed about a third of the time.
Same test with : SE , cutoff . If , power . Four times the data turned a coin-flip-ish test into a near-certain one.
A screening test's is "the patient is healthy". A Type I error tells a healthy patient they may be ill (anxiety, more tests); a Type II error sends an ill patient home. Which error is worse decides how small should be — a statistical choice with real-world stakes.
Treating "fail to reject " as " is true". With power , a real -point effect goes undetected about of the time; a non-significant result from a small study is often just a study too small to see anything.
In the power example, is lowered from to (cutoff ). Does the power rise or fall, and roughly to what if ?
Show the answerHide the answer
It falls: the cutoff moves right, past the true mean. Power . Fewer false alarms cost many more misses.
Tests for Means and Two Samples
For a mean: with degrees of freedom. To compare two independent groups: . Paired data (before/after on the same subjects) uses a one-sample test on the differences.
Every test statistic has the same shape: (observed expected under ) divided by the standard error, i.e. how many SEs the data sit from the null. The two-sample SE adds variances, as independent quantities must. Pairing is powerful because each subject serves as their own control: differences cancel out everything that is constant within a person, leaving much less noise.
Group A: , , . Group B: , , . SE ; . With about degrees of freedom, the two-sided -value is about : a real difference at the level.
Six students take a test before and after a review session; the score changes are . Test : mean change . The differences have and , so with degrees of freedom: two-sided . Treating the before and after scores as two independent groups would bury this effect under the large differences BETWEEN students.
Treating "statistically significant" as "large" or "important" — with a huge sample, a trivial difference has a tiny -value. Report the effect size (here, points or points) alongside the test.
A sample of has , ; test : . Compute .
Show the answerHide the answer
SE , so — the sample mean is one SE above the null value, not significant (the two-sided -value is about ).
Why It Works
If is true, the test statistic follows a known distribution, so we can compute exactly how often results as extreme as ours would occur by chance alone. When that frequency is , one of two things is true: either holds and we witnessed a -in- fluke, or is false. The rarer the fluke would have to be, the less comfortable the first explanation becomes. Fixing in advance keeps us honest: it is a promise about how often we are willing to be fooled by chance, made before seeing the data.