The model landscape
GPT, Gemini, LLaMA, Claude, DeepSeek — one decoder-only shape, many trade-offs. Open weights versus open source, and how to read a model card.
Type "best LLM" into a search box and you get a leaderboard that reorders itself weekly. Beneath that churn is a surprisingly stable fact: almost every entry is the same machine — a decoder-only transformer trained to predict the next token. The differences that decide which one you deploy live everywhere except the block diagram.
So the useful skill is not memorising a ranking (it will be stale before you finish the page). It is reading a release the way an engineer does: what exactly was released, how large is it, how sparse, how much context, and how was it tuned.
One blueprint, many plots
Picture every model as a house from the same blueprint: decoder-only transformer, attention plus feed-forward layers, a residual stream. The blueprint is nearly universal. What differs is the plot it sits on — the data, the scale, the post-training, and whether every room is open at once (dense) or only a few at a time (mixture-of-experts).
Same block shape, twelve different trade-offs. Filter by openness and sparsity, sort by what you care about, and click a model for its note.
671B total with 37B active per token; the technical report is unusually explicit about architecture and training cost.
Tokenizer: byte-level BPE (128K vocab) · Active share: 5.5% of parameters run per token
Parameter, context and tokenizer figures are illustrativeballparks drawn from public papers and model cards; several closed models never disclose parameter counts, shown here as “undisclosed”. Bars are log-scaled so small and large models share an axis. The lesson is the shape of the trade-offs, not any single number.
What every one of them shares
The shared skeleton is the transformer block you already built: attention mixes information between tokens, a feed-forward network processes each token, residuals and normalization keep the stack trainable, and the last token's vector is projected to scores over the vocabulary. Modern LLMs are this in decoder-only form, trained on the single objective of next-token prediction, then post-trained into assistants.
Because the skeleton is shared, "it's a transformer" tells you almost nothing about capability. Two models can share the block diagram and differ by orders of magnitude in quality, cost and behavior.
What actually differs
-
Data. The pretraining mixture — web text, code, books, multilingual corpora, synthetic text — plus filtering, deduplication and a data cutoff. This is often the single largest quality lever, and the least disclosed.
-
Scale. Parameter count and tokens trained on and compute. The training chapter's scaling laws are why labs chase all three together.
-
Post-training. The supervised fine-tuning and preference tuning (RLHF, DPO, reasoning RL) that turn a raw completer into a model that follows instructions and refuses harm. Same base model, different post-training, very different assistant.
-
Sparsity: dense versus mixture-of-experts. A dense model runs all its parameters on every token. An MoE routes each token to a few experts, so the total parameters measure memory and the active parameters measure per-token compute:
DeepSeek-V3 stores 671B parameters but activates about 37B per token — roughly 5.5%. That is how a large model can be cheap to run.
-
Context length. How many tokens the model can attend over at once. Advertised and effective context differ: performance degrades before the limit ("lost in the middle").
-
Tokenizer. The vocabulary and segmentation learned from the data decide how many tokens a given text costs, whether code or non-English text is efficient, and — importantly — the model's perplexity numbers.
Sparse does not mean small
In an MoE, every token still flows through the full network shape, but only the selected experts compute. So MoE saves inference FLOPs, not memory — all 671B parameters must still be stored. Sparsity is a compute trick, not a compression trick.
Why "open weights" is not "open source"
These two phrases get used interchangeably and should not be. Open source (OSI-style) means you have the freedoms needed to study, modify and redistribute the work — for a model, practically, the training data and training code, not just the result. Open weights releases the trained parameters, usually under a bespoke license with acceptable-use restrictions, and leaves the data and training recipe private.
So "open" is a spectrum, not a boolean:
- Closed / API only — weights never leave the vendor.
- Open weights under a custom license — downloadable, with terms.
- Permissive open weights — Apache-2.0 / MIT-style terms.
- Fully open — weights, license and the data and code to reproduce the run. Genuinely rare, because the data is the expensive asset.
Careful
Read the license before you build on a "open" model — open weights can still restrict commercial use, distillation, or certain applications. And treat any leaderboard position as a dated measurement, not a property of the model.
How to read a model card
A model card is a technical document and a marketing document. Read it in this order:
- What was released — API only, weights, or weights plus data/code? Under what license?
- Architecture and size — dense or MoE; total versus active parameters.
- Context — the window, and what the card says about effective use.
- Data — cutoff date and, if disclosed, the mixture.
- Post-training — method, and any documented refusals or biases.
- Evaluation — the benchmarks, and each one's contamination caveat (the evaluating-models lesson's lenses apply verbatim).
- Cost and latency — the deployment facts the leaderboard omits.
Illustrative vs real
The parameters, context windows and tokenizers in the map above are realistic ballparks from public papers and model cards, not authoritative figures — several closed models never disclose parameter counts, shown as "undisclosed". The axes (openness, sparsity, scale, context, tokenizer) are the durable part; the numbers move every release quarter.
Check yourself
Eduspheria wiki · Large Language Models, Frontier
0 / 5 answered
Where next: you can now place any release on the map — the next lesson asks the harder question the map cannot answer: how do we put a number on the text a model generates?