Wiki
Core11 min read

Statistical inference

Sample means have their own distribution; the standard error and the confidence interval are how we say how wrong a sample might be.

You measure the average latency of a new API on 500 requests and get 214 ms. Should you announce that the API is faster than the old one at 220 ms? The sample mean is a single number, but it is not the truth — a different 500 requests would have given a different number. Inference is the machinery for saying how much a sample can wobble, so that a comparison can be made with the wobble in view.

The central object is the sampling distribution: the distribution of a statistic (like the mean) over all the samples you might have drawn. You never see it directly, because you only get one sample, but its width is what controls every confidence interval and p-value you will ever report.

One sample, many possibilities

Imagine repeating the experiment a thousand times. Each repeat gives its own mean. Those means scatter around the true mean, and their scatter is the standard error. A confidence interval is just a compact way of reporting that scatter; a p-value is a way of asking how surprising the observed scatter would be if there were no effect at all.

Draw many samples — the histogram of their means tightens as n grows, at a rate of 1/√n

sampling distribution of the mean · 800 samples of size 12 · curve is the CLT normal

Population

population mean
28.081
population σ
18.079
σ/√n
5.219
observed SE
5.175

The observed standard error tracks σ/√n to within sampling noise, and the histogram is normal even when the exponential population is strongly skewed. The picture is illustrative and the population is synthetic, but the arithmetic of the standard error is exactly what a confidence interval is built from.

The standard error and the central limit theorem

If observations are independent with population mean μ\mu and standard deviation σ\sigma, the sample mean Xˉ\bar X of nn draws has

E[Xˉ]=μ,SE(Xˉ)=σn.\mathbb{E}[\bar X] = \mu, \qquad \mathrm{SE}(\bar X) = \frac{\sigma}{\sqrt{n}}.

The n\sqrt{n} is the whole story of why more data helps and why it helps slowly: to halve the error bar you need four times the data. The central limit theorem adds that, for large nn, the sampling distribution is approximately normal regardless of the shape of the population — which is why the ubiquitous ±1.96 SE\pm 1.96\,\mathrm{SE} interval works even for skewed inputs, as the exponential population in the artifact shows.

With unknown σ\sigma we substitute the sample standard deviation ss and use the tt-distribution with n−1n-1 degrees of freedom:

Xˉ±t1−α/2, n−1 sn.\bar X \pm t_{1-\alpha/2,\,n-1}\,\frac{s}{\sqrt{n}}.

p-values, said carefully

A p-value is the probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one you saw:

p=P(T≥tobs∣H0).p = P(T \ge t_{\text{obs}} \mid H_0).

It is not the probability that H0H_0 is true, and it is not the probability that your result will replicate. A small pp means the data are surprising under H0H_0; it does not by itself tell you the effect matters. Always report the effect size and its interval, not just the verdict.

Careful

Testing many hypotheses inflates false positives: at α=0.05\alpha = 0.05, twenty independent null tests are expected to produce about one "significant" result by chance. If you slice a dataset into subgroups until something lights up, you have not made a discovery — you have run a multiple-comparisons machine. Correct for it (Bonferroni, Benjamini-Hochberg) or pre-register the comparison.

When theory is not available: the bootstrap

For statistics with no clean formula — a median, a ratio, an R2R^2 — resample the data with replacement many times, recompute the statistic each time, and read the spread of the resampled values. The bootstrap turns the sampling distribution from a theoretical object into a computational one, at the cost of assuming your sample is representative enough to stand in for the population.

Illustrative vs real

The artifact draws from a synthetic population so the sampling distribution is visible, and the "population σ" is estimated from a 20,000-point pilot, which real studies never have. The formulas — σ/n\sigma/\sqrt n, the normal approximation, the bootstrap — are exact procedures; the comfort they give still depends on independent, unbiased sampling, which is the assumption most often violated in practice.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, Data science foundations

0 / 5 answered

  1. 1A population has σ = 12. What is the standard error of the mean of n = 36 independent observations?
    ms
    Numeric answer
  2. 2With SE = 2 and a large sample, what is the approximate half-width of a 95% confidence interval using the normal critical value?
    Numeric answer
  3. 3A p-value of 0.03 means there is a 3% probability the null hypothesis is true.
    True / false
  4. 4Which method estimates a statistic's sampling distribution by resampling the observed data with replacement?
    Short answer
  5. 5To halve a standard error, by what factor must the sample size be multiplied?
    Numeric answer

From the exam paper

Modeled on NITJ AI-505, End-Sem December 2024

0 / 4 answered

  1. 1A company has IT = 200, HR = 100 and Marketing = 300 employees, and wants a stratified sample of 120 proportional to department size. How many IT employees should be selected?
    Numeric answer
  2. 2For the same survey (IT 200, HR 100, Marketing 300; total sample 120), how many HR employees should be selected?
    Numeric answer
  3. 3For the same survey, how many Marketing employees should be selected?
    Numeric answer
  4. 4What sampling design splits a population into non-overlapping groups and draws from each group in proportion to its size?
    Short answer

Where next: feature generation — where domain knowledge and transformations turn raw columns into signal.