Linear systems: solving, and what to do when you cannot

Three measurements of a physical law rarely lie on a perfect line. A handful of equations in two unknowns almost never share one common point. And yet we want a slope and an intercept anyway. Linear systems are the language for this, and the interesting case is not the tidy one where everything is solvable — it is the overdetermined one where nothing is.

Below, five measurements of a line are shown with their least-squares fit. Move the values and watch the residuals — the fit always settles so that its error is orthogonal to the data columns.

Existence and uniqueness #

For $A \in \mathbb{R}^{m\times n}$ and $b \in \mathbb{R}^m$, the system $Ax = b$ is consistent exactly when $b$ lies in the column space of $A$. The number of solutions is then:

• none if $b$ is outside the column space — the inconsistent case;
• exactly one if the columns are independent (full column rank $n$);
• infinitely many if the columns are dependent, one per null-space vector.

Equivalently, if $A$ is square then $Ax = b$ has a unique solution for every $b$ exactly when $\det A \neq 0$. Row-reduction (Gaussian elimination) is the mechanical way to decide which case you are in; the column-space question is the geometric way to understand it.

The inconsistent case: least squares #

When no $x$ satisfies $Ax = b$, we choose the $x$ that makes $Ax$ as close to $b$ as possible, in the Euclidean sense:

$\hat{x} = \arg\min_{x} \lVert Ax - b \rVert^2.$

The minimizer satisfies the normal equations

$A^\top A\, \hat{x} = A^\top b,$

and the fitted vector $\hat{b} = A\hat{x}$ is the orthogonal projection of $b$ onto the column space of $A$. The residual $b - \hat{b}$ is perpendicular to every column — that orthogonality is not a coincidence, it is the definition of "closest point". If you drag the points in the widget above, the readouts $\sum e_i = 0$ and $\sum x_i e_i = 0$ hold no matter what: those are the two normal equations written out.

Geometry of the four subspaces #

Every matrix $A$ carries four subspaces, and they pair up:

• the column space and its orthogonal complement, the left null space;
• the row space and its orthogonal complement, the null space.

The row space and the column space both have dimension equal to the rank. A solution exists iff $b$ has no component in the left null space, and the solution is unique iff the null space is trivial. This is the whole existence-and-uniqueness story in one picture.

Check yourself

Eduspheria wiki · Mathematics for AI, Linear algebra

0 / 5 answered

1. 1Solve x + y = 3, x − y = 1. What is x?

   Check Show answerNumeric answer
2. 2Fit y = m·x through the origin to the points (1, 2) and (2, 4). What is m?

   Check Hint Show answerNumeric answer
3. 3Which two equations define the least-squares solution when Ax = b is inconsistent? Give the standard name.

   Check Show answerShort answer
4. 4At the least-squares solution, the residual b − A·x is orthogonal to which space?

   A.The column space of AB.The null space of AC.The whole space R^mD.The row space of A

   Check Show answerMultiple choice
5. 5If an m×n system has more equations than unknowns, it can still be consistent.

   TrueFalse

   Check Show answerTrue / false

Where next: eigenvalues and eigenvectors — the special directions a matrix stretches without rotating, and the gateway to the SVD.