Wiki
Advanced13 min read

Estimation and the orthogonality principle

Point estimates, bias and mean-square error, and the projection view of least squares that makes it optimal.

Given noisy data, what single number should you report as your estimate of an unknown quantity? And among all the ways to combine the data, which is best? The answer turns out to be geometric: form the best estimate by projecting the unknown onto what the data can tell you, and the error you cannot remove is orthogonal to the data.

Estimating is projecting

Think of the unknown quantity as a vector and of everything you can compute from the data as a subspace. The best estimate under squared error is the shadow the unknown casts on that subspace. The leftover — the error — is perpendicular to the subspace, so no adjustment within it can shrink the error further.

Drag the target vector and the coefficient below. At the optimal coefficient the residual becomes perpendicular to the data vector.

Estimate y by c·x — find the c that minimizes the squared error

optimal c*
1.350
squared error
4.250
minimum error
3.025
x · residual
3.500

Not optimal yet: the residual still has a component along x.

The parabola is error as a function of c; its bottom is the projection coefficient c* = (x · y) / (x · x). At that bottom the residual is perpendicular to x, which is exactly the normal equation from linear algebra. In random-variable language the same picture says the conditional mean is the orthogonal projection onto the space of functions of the observed data.

Point estimation

An estimator θ^\hat{\theta} is a function of the data that guesses an unknown parameter θ\theta. Its quality is judged by its sampling distribution:

  • Bias: E[θ^]−θE[\hat{\theta}] - \theta. Zero bias means correct on average across repeated samples.
  • Variance: how much the estimate moves from sample to sample.
  • Mean-square error: MSE⁡(θ^)=E[(θ^−θ)2]=Bias⁡2+Variance⁡\operatorname{MSE}(\hat{\theta}) = E[(\hat{\theta} - \theta)^2] = \operatorname{Bias}^2 + \operatorname{Variance}.

That last identity is the bias–variance tradeoff in its original statistical form: an estimator with a little bias but much lower variance can beat an unbiased one. Modern regularization — ridge regression, shrinkage, early stopping — is this tradeoff made explicit.

Least squares as a projection

The normal equations A⊤Ax^=A⊤bA^\top A \hat{x} = A^\top b from the linear-systems lesson have a second reading. The fitted vector b^=Ax^\hat{b} = A\hat{x} is the orthogonal projection of the observed data bb onto the column space of the model matrix AA. The residual e=b−b^e = b - \hat{b} is perpendicular to every column of AA, written

A⊤e=0.A^\top e = 0.

This is the orthogonality principle: the optimal error is uncorrelated with the data used to form the estimate. It holds whether the data are finite vectors or random variables — for random variables it reads E[(Y−Y^)X]=0E[(Y - \hat{Y})X] = 0, and its solution is the linear minimum mean-square-error estimator.

Why least squares is not arbitrary

Under the Gauss–Markov assumptions — errors with zero mean, equal variance, and zero correlation — the least-squares estimator is the best linear unbiased estimator (BLUE): among all linear unbiased estimators it has the smallest variance. So the choice of squared error is not a modelling whim; for linear estimators it is provably optimal under those conditions. Squared error also makes the optimal estimator the conditional mean, which is why least squares is the default almost everywhere.

Squared error is a choice

Least squares is optimal for squared error. If large errors are catastrophic — a sensor that fails loudly, a delivery that misses by hours — you may prefer absolute error and get the conditional median instead. The orthogonality principle is specific to the squared loss; different losses have different optimal predictors.

Illustrative vs real

The widget projects a single vector onto a line in the plane. Real estimation projects a high-dimensional observation onto a subspace spanned by many features, and the data matrix is solved with a QR or SVD factorization rather than by inverting A⊤AA^\top A. The geometry — projection and orthogonality — is identical; only the dimension and the numerical method change.

Check yourself

Eduspheria wiki · Mathematics for AI, Statistics

0 / 5 answered

  1. 1Estimate y = (3, 4) by c·x with x = (1, 2). What is the optimal c?
    Numeric answer
  2. 2At the least-squares solution, the residual is perpendicular to the column space. What is this condition called?
    Short answer
  3. 3An unbiased estimator always has which property?
    Multiple choice
  4. 4Mean-square error equals squared bias plus variance.
    True / false
  5. 5An estimator has bias 1 and variance 4. What is its mean-square error?
    Numeric answer

Where next: hypothesis testing — deciding whether an observed effect is real or just noise.