Instruction tuning
How a text-completer becomes an assistant — supervised fine-tuning on demonstrations, and why it's not enough.
A freshly pretrained model is a glorified autocomplete. Ask it "How do I make bread?" and the statistically likely continuation might be a follow-up question, a history lecture, or a recipe mid-sentence — the internet contains all of those continuations. Instruction tuning (or SFT, supervised fine-tuning) is the first step of turning a base model into an assistant, and it is delightfully simple:
Start here
Instruction tuning = pretraining's objective, different data. Instead of internet text, the model trains to continue conversation transcripts where the assistant responds helpfully. The architecture and loss are unchanged; the distribution shifts.
The recipe
- Collect demonstrations — humans write (instruction → good response) pairs: maybe 10k–1M examples, covering Q&A, summarization, coding, refusal of harmful requests, multi-turn chat.
- Fine-tune on them — the same next-token cross-entropy loss from the training chapter, applied to response tokens only (the prompt is masked out):
One training example — which tokens does the SFT loss count?
Toggle it off and most of the gradient goes into learning to *predict the instruction* — i.e., learning to be a user, not an assistant. Masking is why SFT teaches "answer well" rather than "complete text that contains questions". Tokens here are word-enumerations for clarity; a real tokenizer counts subword pieces, but the masking structure is exactly this.
- Diversity beats volume — models fine-tuned on ~1,000+ diverse tasks generalize to instructions they were never shown (the FLAN/FLAN-T5 finding); task diversity matters more than example count.
The result speaks when spoken to. But SFT has a ceiling, and it's worth naming precisely, because it motivates everything in the next lessons:
- It imitates, it doesn't optimize. SFT clones demonstrator behavior. The model learns what a good answer looks like but never sees which of two outputs is better — so it can't systematically prefer the better one.
- Good examples ≠ bad examples penalized. Cross-entropy pushes up the demonstrated response; it never pushes down the flawed one the model might otherwise produce.
- Human writers plateau. For tasks like "give the most accurate answer," even experts' best single drafts are uneven — and you can't specify "helpfulness" as a differentiable formula.
What SFT cannot specify
Consider what makes a response good: accurate, appropriately concise, safe, honest about uncertainty. Some of that is expressible as "imitate this" — much of it is a judgment call between two imperfect outputs, which is exactly what preference data captures and SFT cannot.
Note
A useful mental model: SFT teaches the format and the manners; preference tuning teaches the taste. Every serious assistant model does both, in that order.
The base-vs-tuned illusion
A subtle point worth testing in the wild: the tuned model didn't gain knowledge during SFT. World knowledge came from pretraining; SFT unlocked access to it in conversation form. That's why a 7B instruction-tuned model can feel smarter than a 13B base model at chat while knowing less — capability and usability are different axes.
Next: the mechanism that captures "taste" — learning from human preferences with a reward model.