In plain English
You only ever see a sample, but you want to talk about the whole population. Inference is the set of rules for how far that leap is safe.
The advanced view
Sampling distributions are the hinge: a statistic is itself random, and the CLT tells us the mean's distribution is approximately normal with standard error σ/√n. Confidence intervals invert a test; p-values report tail probability under the null and say nothing about effect size or the probability the null is true.
Inference draws conclusions about a population from a sample. Two frameworks: frequentist (classical testing and confidence intervals) and Bayesian (updating priors with data). Coursework and interviews are mostly frequentist; cutting-edge applications lean Bayesian. Hypothesis testing: state H0 (usually no effect) and H1, compute a test statistic, compare to a critical value or compute a p-value. The p-value is not the probability H0 is true — it is the probability of data at least as extreme, assuming H0.
SE = σ / √n 95% CI = x̄ ± 1.96 × SE t = (x̄ − μ0) / SE Power = 1 − P(Type II error)
Type I error (false positive) is rejecting a true H0, controlled by α. Type II error (false negative) is failing to reject a false H0. Low-power studies produce unreliable results, so always consider sample size and effect size. Key tests: t-test for means, chi-square for categorical independence or goodness of fit, F-test for variances and in ANOVA, and ANOVA itself for comparing more than two group means.
Essential vocabulary
- p-value
- Probability of results at least as extreme under H0. Evidence against H0, not proof.
- Significance level (α)
- Threshold for rejecting H0. 0.05 is a convention, not a law — weigh the cost of each error.
- Standard error
- Standard deviation of the sampling distribution, σ/√n. Precision of the estimate, not spread of the data.
- Degrees of freedom
- Independent pieces of information in the data. Sets the shape of t and chi-square.
- Non-parametric tests
- Mann–Whitney, Wilcoxon, Kruskal–Wallis. Use when normality fails or data are ordinal.
Why it works
The √n in the standard error is the whole story. Averaging n independent draws divides variance by n, so precision improves with the square root of effort — quadrupling the sample halves the error. That single fact explains why survey costs explode for the last point of precision and why small samples in a case deserve wide bands.
Common pitfalls
- ×Reading a p-value as the probability the hypothesis is true.
- ×Confusing statistical with practical significance in a large sample.
- ×Testing many hypotheses and reporting only the survivor.
- ×Quoting a mean without a standard error when n is small.
How it is used — a survey margin of error
Step 1 of 4
- 1n = 400 customers, 60% say they would repurchase.
Deeper
Deeper: p-values, power and multiple testing
A p-value is P(data this extreme | null true). It is not the probability the null is true, and it says nothing about effect size. Statistical significance with a huge sample can be commercially irrelevant; a large effect with a small sample can fail to reach significance while being the right decision.
Power (1 − β) is the probability of detecting a real effect. Underpowered tests waste time and produce exaggerated 'winners' when they do hit. And if you test twenty metrics at 5%, one false positive is expected by construction — pre-register the primary metric or correct for multiplicity.
Must know cold
- ✓z = (estimate − hypothesis) ÷ standard error; 1.96 is the 5% two-sided threshold.
- ✓CI ≈ estimate ± 1.96 × SE.
- ✓Type I = false positive, Type II = missed real effect.
- ✓Sample size scales with 1/effect² — halving the detectable effect needs 4× the data.
Exercises
Try each one on paper before revealing the worked solution.
Exercise 1
A/B test: 8,000 per arm, control conversion 5.0%, treatment 5.8%. Is the lift significant, and what is it worth on 3M annual visitors at 60 AOV?
Exercise 2
The team tested 12 metrics and reports the one with p = 0.03. What do you say?
A hypothesis test asks where the observed statistic falls on this curve if the null were true. The tails are the rejection region, their combined area is α, and the width of the whole curve shrinks with √n — which is why quadrupling the sample halves the standard error.