Wiki
Advanced13 min read

Hypothesis testing: p-values, errors, and power

A null hypothesis, a test statistic, a rejection region — and the two ways a test can be wrong.

You change a model, the metric moves, and a dashboard says "significant". But metrics move for no reason all the time. Hypothesis testing is the machinery for asking whether an observed effect is larger than what chance alone would produce, and the discipline of stating in advance how surprised you would need to be.

Assume nothing happened, then measure the surprise

Testing starts by assuming the boring world — the null hypothesis, no effect. Under that assumption the test statistic has a known distribution, so we can ask how unusual the observed value is. The answer is the p-value. If it is small enough, the null looks like a bad explanation of the data.

Set the observed sample mean, the standard error, and α below. The shaded regions are where the null would be rejected; the marker is where the data actually fell.

A z-test under the null — where does the observation fall, and how much tail is beyond it?

z
1.600
p-value
0.1096
critical z
1.960
decision
fail to reject H₀

Amber is the rejection region under H₀: α/2 in each tail. The accent band is the two-tailed p-value for the observed z.

The p-value is the probability, assuming H₀, of a result at least as extreme as the one observed — here it is the tail area beyond z. Shrinking the standard error (more data) pushes z out and the p-value down, which is why statistical significance is partly a statement about sample size. It is not the probability that H₀ is true.

The machinery

A test has five ingredients:

  1. Null hypothesis H0H_0 — the default, usually "no effect" (§=0= 0).
  2. Alternative hypothesis H1H_1 — what you would conclude if H0H_0 is rejected (two-sided, or one-sided).
  3. Test statistic — a number computed from the data whose distribution under H0H_0 is known. For a sample mean this is the z-score z=(xˉ−μ0)/SEz = (\bar{x} - \mu_0) / \mathrm{SE}.
  4. Significance level α\alpha — the false-positive rate you are willing to tolerate, chosen before seeing the data.
  5. Decision rule — reject H0H_0 when the statistic falls in the rejection region, equivalently when the p-value is below α\alpha.

The p-value is the probability, computed under H0H_0, of obtaining a statistic at least as extreme as the one observed. For a two-sided z-test that is twice the tail area beyond ∣z∣|z|.

The two errors

Every decision can go wrong in two ways:

  • Type I error: rejecting a true H0H_0. Its probability is exactly α\alpha, the significance level.
  • Type II error: failing to reject a false H0H_0, with probability β\beta. The power of the test is 1−β1 - \beta — the chance it detects an effect that is really there.

Power rises with the size of the true effect, with the sample size, and with α\alpha; it falls with the variability of the data. A test with low power will fail to detect real effects and is the reason "not significant" is so often misread as "no effect".

What a p-value is not

A p-value is not the probability that H0H_0 is true, and 1−p1 - p is not the probability that the effect is real. It is P(data this extreme∣H0)P(\text{data this extreme} \mid H_0). Conflating the two is the most common misinterpretation in applied work: a small p-value says the data would be surprising under the null, not that the null is probably false. It also says nothing about the size of the effect.

Tests and intervals

A two-sided test at level α\alpha is equivalent to checking whether the null value lies in a (1−α)(1-\alpha) confidence interval for the parameter: reject exactly when the interval excludes it. Confidence intervals are usually more informative, because they show the range of plausible effect sizes rather than a single yes-or-no verdict. Report them alongside p-values.

Illustrative vs real

The widget runs a clean one-sample z-test with a known standard error and a normal null distribution. Real studies have small samples with heavy tails (hence t-tests), multiple comparisons that inflate the family-wise error rate, and optional stopping that quietly invalidates the nominal α\alpha. The logic is the same; the guarantees are only as good as the assumptions.

Check yourself

Eduspheria wiki · Mathematics for AI, Statistics

0 / 5 answered

  1. 1σ = 10, n = 25, sample mean = 5, null mean = 0. What is the z statistic?
    Numeric answer
  2. 2For z = 2.5, what is the two-sided p-value? Give four decimals.
    Numeric answer
  3. 3Which is the definition of a p-value?
    Multiple choice
  4. 4What is the probability of correctly rejecting a false null hypothesis called?
    Short answer
  5. 5Failing to reject H₀ proves that H₀ is true.
    True / false

From the exam paper

Modeled on NITJ AI-501, End-Sem December 2024

0 / 5 answered

  1. 1A sample of 900 members has mean 3.47 cm. The population mean is claimed to be 3.23 cm with standard deviation 2.31 cm. Compute the z statistic. Give three decimals.
    Numeric answer
  2. 2For that sample, at the 1% significance level the sample cannot be regarded as drawn from a population with mean 3.23 cm.
    True / false
  3. 3Ten students' heights in inches are 50, 52, 52, 53, 55, 56, 57, 58, 58 and 59. What is the sample mean?
    Numeric answer
  4. 4For those ten heights, test the suggestion that the school mean is 54 inches. Using the sample standard deviation (n − 1), compute the t statistic. Give two decimals.
    Numeric answer
  5. 5At the 5% level of significance, the ten heights provide enough evidence to reject the suggestion that the school mean height is 54 inches.
    True / false

Where next: with linear algebra and probability in hand, you have the mathematical foundation for the machine-learning domains.