Wiki
Core9 min read

Instruction tuning

How a text-completer becomes an assistant — supervised fine-tuning on demonstrations, and why it's not enough.

A freshly pretrained model is a glorified autocomplete. Ask it "How do I make bread?" and the statistically likely continuation might be a follow-up question, a history lecture, or a recipe mid-sentence — the internet contains all of those continuations. Instruction tuning (or SFT, supervised fine-tuning) is the first step of turning a base model into an assistant, and it is delightfully simple:

Start here

Instruction tuning = pretraining's objective, different data. Instead of internet text, the model trains to continue conversation transcripts where the assistant responds helpfully. The architecture and loss are unchanged; the distribution shifts.

The recipe

  1. Collect demonstrations — humans write (instruction → good response) pairs: maybe 10k–1M examples, covering Q&A, summarization, coding, refusal of harmful requests, multi-turn chat.
  2. Fine-tune on them — the same next-token cross-entropy loss from the training chapter, applied to response tokens only (the prompt is masked out):

One training example — which tokens does the SFT loss count?

Canyousummarizephotosynthesisinonesentence?
→Sure:plantsconvertlightintosugar,usingwaterandCO₂.<eos>
loss counts: 11 next-token predictionsmasked: 7 prompt tokens

Toggle it off and most of the gradient goes into learning to *predict the instruction* — i.e., learning to be a user, not an assistant. Masking is why SFT teaches "answer well" rather than "complete text that contains questions". Tokens here are word-enumerations for clarity; a real tokenizer counts subword pieces, but the masking structure is exactly this.

  1. Diversity beats volume — models instruction-tuned on many diverse task types (62 in FLAN 2021, scaling past 1,800 in FLAN's 2022 successor) generalize to instructions they were never shown; task diversity matters more than example count.

The result speaks when spoken to. But SFT has a ceiling, and it's worth naming precisely, because it motivates everything in the next lessons:

  • It imitates, it doesn't optimize. SFT clones demonstrator behavior. The model learns what a good answer looks like but never sees which of two outputs is better — so it can't systematically prefer the better one.
  • Good examples ≠ bad examples penalized. Cross-entropy pushes up the demonstrated response; it never pushes down the flawed one the model might otherwise produce.
  • Human writers plateau. For tasks like "give the most accurate answer," even experts' best single drafts are uneven — and you can't specify "helpfulness" as a differentiable formula.

What SFT cannot specify

Consider what makes a response good: accurate, appropriately concise, safe, honest about uncertainty. Some of that is expressible as "imitate this" — much of it is a judgment call between two imperfect outputs, which is exactly what preference data captures and SFT cannot.

Note

A useful mental model: SFT teaches the format and the manners; preference tuning teaches the taste. Every serious assistant model does both, in that order.

The base-vs-tuned illusion

A subtle point worth testing in the wild: the tuned model didn't gain knowledge during SFT. World knowledge came from pretraining; SFT unlocked access to it in conversation form. That's why a 7B instruction-tuned model can feel smarter than a 13B base model at chat while knowing less — capability and usability are different axes.

Check yourself

Eduspheria wiki · Large Language Models, Post-training

0 / 4 answered

  1. 1During instruction tuning, which tokens is the loss computed on?
    Multiple choice
  2. 2SFT imitates demonstrator behavior: it can push up a good response but never directly push down the flawed response a model might produce.
    True / false
  3. 3What is the standard three-letter acronym for instruction tuning on demonstrations?
    Short answer
  4. 4According to the base-versus-tuned illusion, what did instruction tuning actually add?
    Multiple choice

Next: the mechanism that captures "taste" — learning from human preferences with a reward model.