Wiki
Core13 min read

Language models and n-grams

A language model assigns a probability to a sentence. The n-gram approximation, maximum-likelihood estimation, smoothing and perplexity are where every modern model still begins.

Before Transformers there were counts. A language model is any function that assigns a probability to a sequence of words — high to fluent text, low to nonsense. It powers autocomplete, speech recognition, spelling correction and machine translation, and its modern neural descendants still optimise the same objective: make the observed text likely.

The classic approach asks an almost embarrassingly simple question. Given the last word or two, what word usually comes next? Count the corpus and find out.

The chain rule, then a shortcut

The probability of a sentence is the product of each word given everything before it. That is exact and useless: the context is unbounded. The n-gram model makes one bold approximation — only the last n−1 words matter — and suddenly every term is a frequency you can count.

Pick an order and a context; the widget counts next words straight from a tiny corpus and computes probabilities (and the perplexity of a held-out sentence).

Pick the order n and the context; the next-word distribution is counted straight from the corpus, with optional add-one smoothing.

order n
contextcount(“the”) = 10
cat
0.211 (3)
dog
0.158 (2)
log
0.158 (2)
rat
0.158 (2)
mat
0.105 (1)
ate
0.053 (0)
on
0.053 (0)
sat
0.053 (0)
held-out sentence
the cat sat on the mat
perplexity
4.20
vocab size
9

Real n-gram MLE with the exact corpus counts above. The toy corpus is tiny, so high orders quickly hit zero counts — that data sparsity is precisely why smoothing, back-off and eventually neural language models exist.

The maths of a bigram model

The chain rule for a sequence w1,…,wmw_1, \dots, w_m is exact:

P(w1,…,wm)=∏i=1mP(wi∣w1,…,wi−1).P(w_1, \dots, w_m) = \prod_{i=1}^{m} P(w_i \mid w_1, \dots, w_{i-1}).

A bigram model assumes each word depends only on the previous one: P(wi∣w1,…,wi−1)≈P(wi∣wi−1)P(w_i \mid w_1, \dots, w_{i-1}) \approx P(w_i \mid w_{i-1}). Maximum-likelihood estimation is just relative frequency, optionally smoothed:

P(wi∣wi−1)=c(wi−1,wi)c(wi−1)⟶c(wi−1,wi)+αc(wi−1)+α∣V∣,P(w_i \mid w_{i-1}) = \frac{c(w_{i-1}, w_i)}{c(w_{i-1})} \qquad\longrightarrow\qquad \frac{c(w_{i-1}, w_i) + \alpha}{c(w_{i-1}) + \alpha |V|},

where α\alpha is the add-α\alpha (Laplace) smoothing count and ∣V∣|V| is the vocabulary size.

Perplexity: the standard score

A language model is judged by how well it predicts unseen text. The metric is perplexity, the exponential of the average negative log-likelihood:

PP⁡(W)=exp⁡ ⁣(−1m∑i=1mlog⁡P(wi∣wi−1)).\operatorname{PP}(W) = \exp\!\left( -\frac{1}{m} \sum_{i=1}^{m} \log P(w_i \mid w_{i-1}) \right).

Lower is better; a perplexity of kk means the model is on average as uncertain as choosing uniformly among kk words. When an unseen n-gram gets probability zero, perplexity is infinite — the clearest possible signal that smoothing is not optional.

Sparsity grows fast

Even a small vocabulary explodes: with ∣V∣=50,000|V| = 50{,}000 words there are 2.5 billion possible bigrams and 1.25×10141.25 \times 10^{14} trigrams, while a corpus contains only a tiny fraction of them. Most observed n-grams are unseen no-grams of the test set, which is why back-off, interpolation and Kneser–Ney smoothing exist.

Illustrative vs real

The widget is a real count-based n-gram model over roughly thirty tokens, so high orders hit zero counts almost immediately. Production n-gram models are built over billions of tokens with modified Kneser–Ney smoothing and careful vocabulary closure; neural models replace the counts with learned distributions, but perplexity remains their common yardstick.

Check yourself

Eduspheria wiki · Applied AI, Classical NLP

0 / 5 answered

  1. 1In a corpus the bigram 'the cat' occurs 12 times and 'the' occurs 40 times. What is P(cat | the) with no smoothing?
    Numeric answer
  2. 2The Markov assumption in an n-gram model states that…
    Multiple choice
  3. 3Adding Laplace smoothing to an n-gram model tends to increase the perplexity on the training corpus while allowing finite perplexity on unseen text.
    True / false
  4. 4A model assigns every one of 20 test words probability 0.05. What is the perplexity?
    Numeric answer
  5. 5What is the exponential of the average negative log-likelihood of a test set called?
    Short answer

From the exam paper

Modeled on NITJ AI-502, End-Sem May 2025

0 / 5 answered

  1. 1Consider the corpus: 'I tell you to sleep and rest', 'I would like to sleep for an hour', 'Sleep helps one to relax' (each sentence begins with a start-of-sentence token). How many bigram tokens begin with the word 'to'?
    Numeric answer
  2. 2Using that corpus, what is P(sleep | to)? Give three decimal places.
    Numeric answer
  3. 3Using that corpus, what is P(relax | to)? Give three decimal places.
    Numeric answer
  4. 4Based on the conditional probabilities, which word does the bigram model predict next after 'to'?
    Multiple choice
  5. 5In a bigram model each word is conditioned only on the immediately preceding word.
    True / false

Where next: language models score sequences; the next lesson adds labels — tagging each word, parsing structure, and the Naive Bayes classifier underneath.