In plain English
Regression draws the best straight line through a cloud of points, and tells you how much of the pattern the line actually explains.
The advanced view
OLS minimises squared residuals, producing coefficients that are partial effects — the change in y for a unit change in x holding other regressors fixed. Unbiasedness needs exogeneity; usable standard errors need care with heteroskedasticity, autocorrelation and clustering. Endogeneity from omitted variables, simultaneity or measurement error breaks causal reading.
Linear regression is the most important statistical tool for applied work. OLS minimises the sum of squared residuals, and under the Gauss–Markov assumptions — linearity, exogeneity, homoscedasticity, no autocorrelation, no perfect multicollinearity — it is BLUE: best linear unbiased estimator.
y = β0 + β1x1 + β2x2 + … + ε R² = 1 − (SS_residual / SS_total) Adjusted R² = 1 − [(1−R²)(n−1) / (n−k−1)] t for βj = β̂j / SE(β̂j)
R² is the share of variance explained and always rises with more variables, so use adjusted R² for model comparison. But R² is not the point: the coefficients — signs, magnitudes, significance — answer the question. Diagnose heteroscedasticity (robust standard errors), multicollinearity (check VIF), autocorrelation (Durbin–Watson) and endogeneity (omitted variables, reverse causality, measurement error; fix with instruments or natural experiments). Logistic regression models binary outcomes through log-odds: ln(p/(1−p)) = β0 + β1x, so exponentiate coefficients to read odds ratios.
Finance connection
Empirical finance is regression applied to financial data. CAPM is a regression — regress excess stock returns on excess market returns, the slope is beta and the intercept is alpha. Fama–MacBeth regressions test factor models, and event studies use residuals to measure the impact of corporate announcements.
The line is chosen so that the sum of the squared vertical red segments — the residuals — is as small as possible. Squaring is why one far-away point moves the line more than ten near ones, and why the fitted line always passes through the point (x̄, ȳ).
Deeper
The geometry, the algebra and what the coefficients mean
In simple regression the slope has a closed form you should be able to write from memory: β̂₁ = Cov(x, y) / Var(x), and the intercept follows from β̂₀ = ȳ − β̂₁x̄. That single expression tells you three things at once. The slope is a covariance, so it inherits the sign of the co-movement. It is scaled by the variance of x, so a predictor that barely varies produces an unstable coefficient. And because the intercept is defined off the means, the fitted line must pass through the average point — which is why a regression can fit the middle of the data well and still be wrong everywhere else.
Multiple regression changes the interpretation, not the machinery. β̂ⱼ is the effect of a one-unit change in xⱼ holding the other regressors fixed — the partialled-out effect. Mechanically, it equals the simple regression of y on the part of xⱼ that the other regressors cannot explain. That is the whole intuition behind multicollinearity: if x₁ is nearly a linear combination of x₂, almost nothing is left after partialling out, the denominator collapses, and the standard error explodes even though the joint fit is fine.
Standard errors, not point estimates, decide whether you have learned anything. SE(β̂ⱼ) = σ / √(Var(xⱼ) · n · (1 − R²ⱼ)), where R²ⱼ is from regressing xⱼ on the other predictors. Read the four levers: less noise, more spread in the predictor, more observations, and less collinearity all sharpen the estimate. VIF is simply 1/(1 − R²ⱼ); a VIF of 10 means the standard error is √10 ≈ 3.2 times wider than it would be with an orthogonal predictor.
Interpretation depends on the functional form, and interviewers test this. Level–level: a one-unit change in x moves y by β units. Log–level: y changes by roughly 100·β percent. Level–log: a 1% change in x moves y by β/100 units. Log–log: β is an elasticity — the percentage change in y per percentage change in x. Elasticities are the form consultants actually use for pricing work, so default to logs when both variables are strictly positive and multiplicative in nature.
Dummies and interactions carry most of the business content. A dummy shifts the intercept: the coefficient is the difference in mean y between the group and the omitted baseline, so always name the baseline out loud. An interaction between a dummy and a continuous variable changes the slope, which is how you say "price sensitivity is higher in the discount channel". With k categories include k−1 dummies; including all k plus an intercept gives you perfect collinearity, the dummy-variable trap.
The parts worth memorising
β̂₁ = Cov(x, y) / Var(x) β̂₀ = ȳ − β̂₁x̄ ŷ = Xβ̂, β̂ = (X'X)⁻¹X'y SST = SSE + SSR R² = SSE/SST = 1 − SSR/SST SE(β̂ⱼ) = σ̂ / √(Var(xⱼ)·n·(1 − R²ⱼ)) VIF ⱼ = 1/(1 − R²ⱼ) t = β̂ⱼ / SE(β̂ⱼ) F = [(R²_full − R²_restr)/q] / [(1 − R²_full)/(n − k − 1)] log–log slope = elasticity; log–level slope ≈ % change in y
The full bar is the variance of y; the filled part is what the model explains. A high R² with a wrong-signed coefficient is worthless, and a 0.05 R² on daily returns can be extremely valuable. Adjusted R² penalises extra regressors so you can compare models of different size.
Each line is β̂ ± 1.96·SE. When the interval spans zero you cannot sign the effect, however large the point estimate looks. Wide intervals usually mean too little variation in that predictor or collinearity with another, not the absence of a real effect.
Worked example — reading a regression table end to end
Step 1 of 10
- 1Model: ln(units) = β0 + β1·ln(price) + β2·ad_spend + β3·discount_channel
Residuals against fitted values: a structureless band around zero. This is the single most informative diagnostic — plot it before you read a single p-value. Anything other than noise here means the model is misspecified, not that the data are awkward.
The spread of the residuals grows with the fitted value, typical of revenue, spend and firm-size data. β̂ stays unbiased but the standard errors are wrong, so the fix is heteroscedasticity-robust (White/Huber) standard errors, not dropping observations.
The true relationship bends and eventually turns down; the straight fit averages over that and reports a modest positive slope. Its residuals would show a clear arc. Fix it with logs or a quadratic term, and remember the turning point of β1x + β2x² sits at −β1/(2β2).
The dashed line is the fit on the cluster; the solid line is the fit once the far-right point joins. Extreme x-values carry leverage, and squared residuals do the rest. Always report whether your conclusion survives with the point excluded rather than deleting it silently.
Within both segments the relationship is negative, but the pooled fit slopes upward because the segments differ in level. This is omitted-variable bias in picture form: the bias equals the effect of the missing variable times its correlation with the included one, so you can sign it before you have any data.
x₁ and x₂ move together, so the data cannot separate their effects. The joint fit and the forecasts are fine; the individual coefficients are unstable and may flip sign between samples. Drop one, combine them, or say plainly that only the joint effect is identified.
A linear probability model runs off the top and bottom of the [0,1] range and assumes a constant marginal effect. The logistic curve is bounded, and its marginal effect is largest around the 50% point — the reason churn and default models are fitted in log-odds and read as odds ratios.
Deeper
Endogeneity, and the honest ways out
OLS is only causal when the regressor is uncorrelated with the error. Three things break that. Omitted variables: something drives both x and y and is left out. Simultaneity: y also drives x, as with price and quantity set together in a market. Measurement error in x, which drags the coefficient toward zero — attenuation bias. Naming which of the three threatens your regression is more impressive in an interview than any diagnostic statistic.
Instrumental variables are the classical fix. A valid instrument z must be relevant (correlated with x, first-stage F above roughly 10) and exogenous (uncorrelated with the error, affecting y only through x). Two-stage least squares fits x̂ from z, then y on x̂; the price is a much larger standard error, so a weak instrument is worse than no instrument.
Design-based approaches are what employers use now. Difference-in-differences compares the change in the treated group with the change in an untreated control, which removes both fixed group differences and common time shocks — check for parallel pre-trends. Fixed effects absorb everything constant within a firm or a year. Regression discontinuity exploits a threshold rule. Randomised experiments make exogeneity true by construction, which is why an A/B test beats any econometric correction when it is feasible.
Panel data adds structure and traps. Cluster your standard errors at the level where shocks are correlated, usually the firm; ignoring that inflates t-statistics dramatically. Random effects is more efficient but assumes the unobserved effect is uncorrelated with the regressors — fixed effects is the safer default in business data, and a Hausman test formalises the choice.
Worked example — signing omitted-variable bias before you regress
Step 1 of 9
- 1Regression: sales_growth =
Regression — must know cold
- ✓β̂₁ = Cov(x, y)/Var(x); the fitted line passes through (x̄, ȳ).
- ✓A multiple-regression coefficient is a partial effect: other regressors held fixed.
- ✓|t| > 2 is roughly 5% significance; report the confidence interval, not just the star.
- ✓Log–log slope is an elasticity; |elasticity| > 1 means a price rise loses revenue.
- ✓Heteroscedasticity breaks standard errors, not unbiasedness ⇒ robust SEs.
- ✓Omitted-variable bias sign = sign(effect of omitted) × sign(correlation with included).
- ✓VIF = 1/(1 − R²ⱼ); above ~10 the individual coefficient is not interpretable.
- ✓With k categories use k−1 dummies and name the baseline.
- ✓Binary outcome ⇒ logistic; exponentiate the coefficient to get an odds ratio.
- ✓Correlation is not causation; identification comes from design, not from more controls.
Common pitfalls
- ×Choosing a model on R² alone and ignoring signs, magnitudes and intervals.
- ×Reporting p-values without ever plotting residuals against fitted values.
- ×Adding controls until the coefficient of interest becomes significant.
- ×Extrapolating far outside the range of x that the data actually cover.
- ×Deleting inconvenient outliers instead of reporting sensitivity to them.
- ×Interpreting an interaction coefficient without also reporting the main effects.
- ×Treating a statistically significant but economically trivial effect as a finding.
Regression exercises
Try each one on paper before revealing the worked solution.
Exercise 1
A regression of ln(volume) on ln(price) gives β̂ = −1.4 with SE 0.25 on n = 180. Is demand elastic, and what happens to revenue if price rises 5%?
Exercise 2
Two nested models: R² = 0.48 with 3 regressors, R² = 0.52 with 6 regressors, n = 120. Do the three extra variables earn their place?
Exercise 3
A churn logistic regression gives coefficient +0.62 on 'had a service outage'. Baseline monthly churn is 3%. Translate it for a client.
Exercise 4
Messy prompt. A client says 'our regression proves discounts drive growth — the coefficient is huge and significant'. Structure your response.
Consulting connection
Almost every quantitative recommendation you will make rests on a regression someone else ran. Your value is not fitting another model; it is asking what the residuals look like, what is missing from the specification, and whether the design can support the causal claim in the client's slide title.
Why it works
Least squares works because the residual vector is orthogonal to the regressors: the fitted values are the projection of y onto the space spanned by X, and projection is by definition the closest point. That geometry is why adding a regressor can never lower R², and why collinearity inflates variance — the columns point in nearly the same direction.
Common pitfalls
- ×Reading a coefficient as causal without an identification story.
- ×Chasing R² — a high R² with an unstable coefficient is worthless for decisions.
- ×Extrapolating well outside the range of x seen in the data.
- ×Forgetting that log-log coefficients are elasticities, not units.
How it is used — a pricing elasticity
Step 1 of 4
- 1log(volume) = 8.2 − 1.4 × log(price), R² =
Deeper
Deeper: omitted variables, endogeneity and reading coefficients
A coefficient is the average change in y per unit of x, holding the other included variables constant. The phrase 'included' is the whole problem: an omitted variable correlated with both x and y biases the coefficient, and the direction of the bias is the product of the two correlations. Advertising spend appears to raise sales partly because both rise in strong quarters.
R² measures fit, not correctness, and it never falls when you add variables — use adjusted R² or out-of-sample error. For causal claims you need design (experiment, instrument, difference-in-differences, regression discontinuity), not more controls.
Must know cold
- ✓ŷ = a + bx; b = cov(x, y) ÷ var(x).
- ✓R² = share of variance explained; adjusted R² penalises extra variables.
- ✓t = coefficient ÷ standard error; |t| > 2 is the rough 5% rule.
- ✓Extrapolating outside the observed range of x is where models break.
Exercises
Try each one on paper before revealing the worked solution.
Exercise 1
Orders = 25 + 2.4 × spend (spend in $K/day), r = 0.75. Predict orders at 40, and state the share of variation explained.
Exercise 2
The model says each extra $1K of spend brings 2.4 orders at 60 profit each. Should the client double spend?