Wiki
Core13 min read

Eigenvalues and eigenvectors: the directions a matrix cannot rotate

Most vectors get turned by a matrix; a special few only get stretched. Those directions are the matrix's eigenstructure.

Almost every direction gets bent when a matrix acts. Pick a vector, multiply by the matrix, and the result usually points somewhere new. But a few directions are special: along them the matrix does nothing but stretch or shrink. Those directions are the eigenvectors, and the stretch factors are the eigenvalues.

Find the skeleton of a transformation

If you feed an eigenvector back through the matrix, it comes out parallel to itself — same line, new length. That makes eigenvectors the natural coordinate system for the map: in those coordinates the matrix is diagonal, so repeated application is just repeated scaling.

Rotate the test vector below. Most angles give a vector that has been turned; at two particular angles the image stays on the same line.

Rotate the test vector — where does the matrix merely stretch it?

λ₁
3.00
λ₂
1.00
angle(v, Mv)
7.0°
Mv
(2.22, 1.95)

v has been rotated: it is not an eigenvector of this matrix.

The dashed blue and amber lines are the two eigen-directions. A vector exactly along either one comes back parallel to itself, stretched by λ₁ or λ₂; every other vector is also rotated. For a symmetric matrix the eigenvectors come out perpendicular and the eigenvalues real, which is why covariance and PCA matrices behave so nicely.

The definition and the characteristic equation

A non-zero vector vv is an eigenvector of AA with eigenvalue λ\lambda when

Av=λv.A v = \lambda v.

Rearranging, (A−λI)v=0(A - \lambda I) v = 0, which has a non-zero solution only when A−λIA - \lambda I is singular:

det⁡(A−λI)=0.\det(A - \lambda I) = 0.

For a 2×22\times 2 matrix this expands to λ2−(tr⁡A)λ+det⁡A=0\lambda^2 - (\operatorname{tr} A) \lambda + \det A = 0, so the two eigenvalues satisfy

λ1+λ2=tr⁡A,λ1λ2=det⁡A.\lambda_1 + \lambda_2 = \operatorname{tr} A, \qquad \lambda_1 \lambda_2 = \det A.

Those two identities are a fast sanity check on any hand computation: if the eigenvalues do not sum to the trace and multiply to the determinant, something is wrong.

Symmetric matrices are the friendly case

A real symmetric matrix A=A⊤A = A^\top has two properties that make it central to machine learning:

  1. All eigenvalues are real (no complex rotations hiding in the spectrum).
  2. Eigenvectors belonging to distinct eigenvalues are orthogonal, so they can be chosen as an orthonormal basis.

The spectral theorem then says A=QΛQ⊤A = Q \Lambda Q^\top, where QQ is orthogonal and Λ\Lambda is diagonal. This is exactly what makes a covariance matrix diagonalizable by a rotation — and that rotation is PCA. The directions the data varies in most are the eigenvectors of the covariance with the largest eigenvalues. The widget above is a covariance matrix in disguise.

Not every matrix is diagonalizable

Repeated eigenvalues can leave too few independent eigenvectors to form a basis — the matrix is then defective and needs the Jordan form. In practice, the matrices we care about most (symmetric, positive definite, normal) are safe. The SVD, next, sidesteps the issue entirely by working with A⊤AA^\top A and AA⊤AA^\top, both symmetric.

Where eigenvalues show up

  • Stability of iteration. If you repeatedly multiply xx by AA, the component along the largest-magnitude eigenvector dominates. That is power iteration for computing eigenvectors, and it is also why a learning rate that makes the largest eigenvalue of the update too big diverges.
  • Conditioning. The ratio of largest to smallest singular value (or ∣λ∣|\lambda| for symmetric matrices) measures how much the map stretches different directions unevenly — a flatter spectrum means a better-conditioned problem.
  • Quadratic forms. The definiteness of AA is the sign pattern of its eigenvalues, which decides whether a loss surface is a bowl or a saddle.

Illustrative vs real

The widget uses a symmetric 2×22\times 2 matrix so the eigenvectors are real and perpendicular and you can see them. A ResNet weight matrix or a data covariance can be thousands of dimensions wide; you will not draw them, but the same definition Av=λvAv = \lambda v applies, and the symmetric special case is what makes the tooling practical.

Check yourself

Eduspheria wiki · Mathematics for AI, Linear algebra

0 / 5 answered

  1. 1For M = [[2, 1], [1, 2]], what is the larger eigenvalue?
    Numeric answer
  2. 2For M = [[2, 1], [1, 2]], what is the product of the eigenvalues?
    Numeric answer
  3. 3For a symmetric matrix, what is the geometric relationship between eigenvectors for distinct eigenvalues?
    Short answer
  4. 4Multiplying a matrix by an eigenvector changes the vector's direction in general.
    True / false
  5. 5The principal component directions of a dataset are which quantities of its covariance matrix?
    Multiple choice

From the exam paper

Modeled on NITJ AI-501, End-Sem December 2024

0 / 5 answered

  1. 1The matrix [[2, 8], [7, 9]] has two eigenvalues. Give the larger one, correct to two decimal places.
    Numeric answer
  2. 2For the same matrix [[2, 8], [7, 9]], give the smaller eigenvalue, correct to two decimal places.
    Numeric answer
  3. 3What is the product of the two eigenvalues of [[2, 8], [7, 9]]?
    Numeric answer
  4. 4The matrix [[13, 9, 2], [3, 12, 1], [4, 6, 14]] has three eigenvalues. What is their sum?
    Numeric answer
  5. 5For the same 3×3 matrix [[13, 9, 2], [3, 12, 1], [4, 6, 14]], what is the product of its three eigenvalues?
    Numeric answer

Where next: the singular value decomposition, which does the eigen story for every matrix, square or not.