Panel data is the closest most applied economists get to a controlled experiment: the same firms, people or countries observed year after year. Pooled carelessly, as if the observations were strangers, it can get the sign of an effect wrong — not just the size. The fixed-versus-random-effects choice is where that care lives, and it is the panel question every econometrics tutor in London gets asked in exam season. Here is the whole decision, with a data set where pooling really does flip the sign.
1 · What panel data buys you
A panel follows N units over T periods, so the outcome is yᵢₜ — firm i in year t. Write the model as
yᵢₜ = β₀ + β₁xᵢₜ + αᵢ + uᵢₜ.
The new object is αᵢ: the unobserved individual effect. It collects everything about unit i that does not change over the sample — a plant’s management quality, a person’s ability, a country’s institutions. In a single cross-section it is pure omitted-variable territory.
The panel’s gift is repetition. Because αᵢ appears in every one of unit i‘s observations, it can be removed rather than measured. How you treat αᵢ is the entire fixed-versus-random distinction.
2 · Fixed effects — the within estimator
Fixed effects (FE) makes no assumption about αᵢ except that it is constant over time. Subtract each unit’s own averages from its observations — demeaning, the within transformation — and αᵢ vanishes: it equals its own average. OLS on the demeaned data uses only within-unit variation: how each firm’s outcome moves when its own regressor moves, with everything permanent about the firm differenced away.
The price is what else the transformation kills. Any regressor that never changes over the sample — a firm’s founding year, a country’s legal origin — is constant within units, so demeaning wipes it out too: FE cannot estimate time-invariant variables. And with little within-unit variation, FE estimates get noisy.
3 · Random effects — efficiency, at a price
Random effects (RE) treats αᵢ as part of the error term and assumes it is uncorrelated with the regressors. Under that assumption there is no bias to remove, and full demeaning throws information away. RE instead quasi-demeans — subtracting only a fraction θ of each unit’s mean, where θ rises toward 1 as the individual effects grow relative to the noise or as T grows. The result is more efficient than FE when the assumption holds, and it keeps time-invariant regressors because it never fully wipes out the unit means.
Everything hangs on that assumption. If αᵢ is correlated with xᵢₜ — abler workers get more training, better-governed countries trade more — RE inherits the omitted variable bias that FE was built to remove.
The practical arbiter is the Hausman test. It compares the FE and RE estimates: under the RE assumption both are consistent and should agree; if they differ significantly, the RE assumption is rejected and you report FE.
Worked example — one panel, two verdicts
Three factories report monthly training hours per worker (x) and defects per 1,000 units (y) for four periods each. To keep the arithmetic exact, the noise term is set to zero — every y below is exactly αᵢ − 2xᵢₜ.
| Firm A | Firm B | Firm C | |
|---|---|---|---|
| (x, y) | (1, 8) (2, 6) (3, 4) (4, 2) | (4, 12) (5, 10) (6, 8) (7, 6) | (7, 16) (8, 14) (9, 12) (10, 10) |
Step 1 — Pool everything and run OLS. Treating the 12 observations as one sample: the means are x̄ = 5.5 and ȳ = 9, with Σ(x − x̄)(y − ȳ) = 66 and Σ(x − x̄)² = 87. The pooled slope is 66/87 = +0.76. Conclusion: training raises defect rates.
Step 2 — Look inside any one firm. Firm A’s defects fall 8, 6, 4, 2 as training rises 1, 2, 3, 4 — a slope of exactly −2. So do Firm B’s and C’s: within every firm, training lowers defects.
Step 3 — The within estimator. Demean each firm’s data and pool: the demeaned sums are Σx̃ỹ = −30 and Σx̃² = 15, so the FE estimate is −30/15 = −2. That is the true β₁.
Step 4 — Why pooling lied. The fixed effects are αA = 10, αB = 20, αC = 30 — and the high-α firms are the heavy trainers, because struggling plants train more. The between-firm regression (on the three firm means) slopes +1.33, and pooled OLS blends between and within variation: the upward between effect swamps the downward within effect.
Step 5 — What RE would do here. Random effects assumes αᵢ is uncorrelated with training. Step 4 shows the opposite, so RE is inconsistent on this data, landing between the pooled and FE answers. A Hausman test would reject it.
Step 6 — Interpretation. Same 12 numbers, three slopes: +0.76 pooled, +1.33 between, −2 within. Only the within estimate answers the causal question “what happens to this firm’s defects if it trains more”. The panel’s whole value was the licence to throw the between-firm variation away.
Same twelve numbers, three different slopes — could you say which one answers the causal question? Choosing between fixed and random effects, then defending it with the Hausman test, is exactly the panel skill exams reward and careless pooling quietly fails. Building that judgement on your own problem sets is what a one-on-one econometrics tutor does with you. Book a trial session.
Practice
Q1. A panel has two units and two periods. Unit 1: (x, y) = (2, 10) then (4, 6). Unit 2: (6, 20) then (8, 16). Compute the pooled OLS slope and the within (FE) slope.
Q2. The RE quasi-demeaning fraction is θ = 1 − √(σ²ᵤ / (σ²ᵤ + T σ²ₐ)), where σ²ᵤ is the idiosyncratic error variance and σ²ₐ the variance of the individual effects. With σ²ᵤ = 1, σ²ₐ = 4 and T = 5, compute θ and interpret it.
Q3. For one coefficient, FE gives −2.0 (standard error 0.3) and RE gives −1.2 (standard error 0.2). Compute the one-parameter Hausman statistic H = (bFE − bRE)² / (SE²FE − SE²RE) and test at 5% (χ² critical value 3.84). Which estimator do you report?
Answers. Q1: pooled: x̄ = 5, ȳ = 13, slope = 32/20 = +1.6. Within: each unit’s differences give Δy/Δx = −4/2 = −2 — the sign flips, as in the worked example. Q2: θ = 1 − √(1/21) = 0.78. The individual effects are large relative to the noise, so RE subtracts 78% of each unit’s mean — most of the way to fixed effects (θ = 1 would be the within transformation; θ = 0, pooled OLS). Q3: H = (−0.8)² / (0.09 − 0.04) = 0.64/0.05 = 12.8 > 3.84: the estimates differ by more than sampling error allows, so reject RE and report fixed effects.
Key takeaways
- Panel data lets you remove, not measure, everything permanent about a unit — the unobserved effect αᵢ.
- Fixed effects demeans within units: consistent even when αᵢ is correlated with the regressors, but it cannot estimate time-invariant variables and leans entirely on within-unit variation.
- Random effects quasi-demeans: more efficient and keeps time-invariant regressors, but only valid if αᵢ is uncorrelated with the regressors.
- Pooled OLS blends between and within variation — with correlated effects it can reverse the sign, as +0.76 vs −2 above. The Hausman test is the standard referee between FE and RE.
Why London students choose our econometrics tutoring
- Course-matched notation: the within transformation, quasi-demeaning and the Hausman test are taught in your own course’s notation and software — the panel commands your problem sets actually use.
- Exam-question shapes, drilled: compute a within estimator by hand, state the RE assumption precisely, choose an estimator and defend it — the three forms panel questions take, practised on past papers.
- One-on-one and online: sessions work from your lecture notes and returned scripts, so the feedback lands where the marks are being lost.
FAQ
Q: What is the difference between fixed effects and random effects?
A: Both model an unobserved, time-constant individual effect. Fixed effects removes it by demeaning, so it tolerates correlation with the regressors. Random effects leaves it in the error term, assumes no such correlation, and gains efficiency when that holds.
Q: When should I use fixed effects?
A: Whenever the unobserved individual effect is plausibly correlated with your regressors — which in applied economics is most of the time. If the Hausman test rejects, fixed effects is the defensible choice.
Q: Why can’t fixed effects estimate time-invariant variables?
A: The within transformation subtracts each unit’s mean, and a variable that never changes equals its own mean — it demeans to zero, absorbed into αᵢ.
Q: What does the Hausman test actually test?
A: Whether the FE and RE estimates differ significantly. Under the RE assumption both are consistent, so a large gap is evidence against it — reject RE, report FE.
Q: Is pooled OLS ever fine on panel data?
A: Only if the individual effects are uncorrelated with the regressors and the standard errors are clustered, since a unit’s observations are not independent. If the effects are correlated, pooled OLS can even reverse the sign, as this page’s data set shows.
Book an econometrics tutor in London or online
Panel data questions are won on precise mechanics — demean, estimate, test, defend. One-on-one sessions build exactly that, on your course’s own problem sets and past papers, in London or online. Tell us your module and exam date, and we’ll match you with the right tutor this week.
