Samples and Bias
A population is everyone you want to describe; a sample is the part you actually measure. A simple random sample gives every group of individuals the same chance of being chosen. Bias is a systematic tilt: voluntary response, convenience sampling, undercoverage, non-response, and leading questions all produce it.
Randomness is not sloppiness; it is the only known defense against the biases you cannot think of. A sample chosen by a person — even a careful one — reflects that person's reach and habits. A sample chosen by a random number generator reflects nothing but the population. Larger samples reduce random error; they do nothing at all about bias.
A website asks visitors to rate the site. Only people who feel strongly respond: voluntary response bias, which usually over-represents the unhappy. Surveying shoppers at a mall about online shopping misses everyone who never visits malls — undercoverage — and those are exactly the people most likely to shop online.
Choose of club members, numbered –, using the random digits . Read two digits at a time; keep numbers from to , skip the rest, skip repeats. keep; skip (no such member); keep; skip (already chosen); keep; skip; keep. The sample is members .
In 1936 the Literary Digest mailed about million ballots to people drawn from phone books and car registrations and got about million back, predicting a landslide for Landon. Roosevelt won easily. In the Depression, phone and car owners were wealthier than voters as a whole, and those who bothered to reply differed again. Millions of biased responses estimated the wrong thing very precisely.
Believing a bigger sample fixes bias. Doubling a convenience sample doubles the number of people from the same wrong place; the error that comes from WHO is sampled does not shrink with . Only a better sampling method removes it.
A radio host asks listeners to phone in whether they support a new tax. of callers oppose it. What kind of bias is this, and which way does it likely push?
Show the answerHide the answer
Voluntary response (plus undercoverage of non-listeners). People with strong feelings — typically opponents — are the ones who call, so the likely overstates opposition.
Sampling Designs
A stratified sample divides the population into groups (strata) and takes a random sample within each, so every group is represented. A cluster sample randomly picks whole groups (classrooms, blocks) and measures everyone in them. A systematic sample takes every th individual from a list after a random start.
Stratifying uses what you already know: if grades differ in their answers, sampling each grade separately removes the chance of an unlucky mix and makes estimates more precise. Clustering trades precision for convenience — surveying five whole classrooms is cheaper than finding sixty scattered students — and works best when each cluster is a miniature of the population. All three designs are still random; they only organize the randomness.
A school has ninth-graders, tenth-graders, eleventh-graders and twelfth-graders: students. For a sample of () that mirrors the grades, take of each: , , and , chosen at random within each grade.
From an alphabetical list of the same students, a sample of needs . Pick a random start between and — say — then take students . Quick and usually fine, unless the list has a pattern with period .
Confusing strata with clusters. Strata are groups that DIFFER from each other (sample within every one); clusters are groups that each LOOK LIKE the population (sample a few whole ones). Picking two whole grades at random would be a cluster sample, and could easily miss the seniors entirely.
A school randomly picks homerooms and surveys every student in them. Stratified or cluster?
Show the answerHide the answer
Cluster: whole groups are chosen at random and everyone in them is measured.
Experiments versus Observation
An observational study records what happens; an experiment assigns treatments. Only an experiment with random assignment can establish cause, because randomization balances every lurking variable across the groups, seen and unseen. A control group, blinding and replication make the comparison fair.
The logic is the same as sampling, turned inward: random assignment makes the treatment groups alike in everything except the treatment, so any difference too large to be chance must come from the treatment. Without randomization, the people who chose the treatment differ from those who did not, and the difference could be them rather than it.
To test whether a study app raises scores: randomly split volunteers into two groups of (for instance by shuffling their names); one group uses the app, the other does not; after a month compare mean scores. The random split is what licenses a causal conclusion; the in each group is replication, so chance differences average out.
Prior grades strongly affect test scores. To remove that noise, first sort the volunteers into blocks — A students, B students, and so on — and randomize to app or no app WITHIN each block. The comparison is then made grade-by-grade, so differences in prior grades cannot blur the effect of the app.
Testing a headache pill, the control group gets an identical-looking sugar pill (a placebo), because feeling treated can itself reduce pain. In a double-blind design neither patients nor the people rating their pain know who got which, so expectations cannot tilt the measurements either.
Concluding cause from an observational study ("students who use the app score higher") — the app users may simply be the more motivated students. Motivation is a lurking variable that random assignment would have split evenly.
Cities with more police have more crime. Does hiring police cause crime?
Show the answerHide the answer
No — a lurking variable: city size (and the underlying crime rate) drives both police hiring and crime counts. Only an experiment, or careful comparison of similar cities, could isolate the effect of police.
Why It Works
Suppose of volunteers are highly motivated, and nobody knows which. Randomly split them /: the number of motivated people in the app group has mean (half of ) and standard deviation . So the two groups almost always differ by only a handful of motivated people — and the same is true simultaneously for every other trait, measured or not. That is why a large difference in outcomes can be credited to the treatment: chance alone rarely produces a large imbalance, and probability tells us exactly how rarely.