Context is the constraint
The context window is an agent's working memory, and every step re-reads all of it. Loop length — not question length — drives cost, latency, and forgetting.
A single LLM call is stateless: everything the model knows about your task must be in the prompt. An agent's loop makes this vivid — each step appends its thought, tool call, and observation to a transcript that the next step must read in full. The context window is the agent's working memory, and it is the resource the whole loop spends.
Start here
The context window is not a hard drive you fill up; it is a desk you must keep clear. Too little on the desk and the agent forgets what it is doing; too much and the important facts get lost among the noise — and you pay for every item on the desk on every single step.
An 8-step agent task — toggle a verifier on the critical step
total tokens: 9,000 vs 7,200 clean
cost scales with loop length, not question length
Every step re-reads the whole transcript (the loop-cost lesson: attention over a growing context), and an unchecked early error compounds — the waste bars grow downstream of the bad step. The verifier costs one extra step but resets the trajectory. Token figures are round-number arithmetic on illustrative steps; the compounding shape is the real phenomenon.
Toggle the verifier and watch the loop's cost and error behaviour change. The key asymmetry: a chat answer pays for its context once, while an agent pays for a growing context on every iteration.
Why loops get expensive
- Context re-read — step n reads everything from steps 1 to n−1. A 20-step agent on a long transcript pays for that transcript 20 times.
- Attention cost — self-attention scales with the square of sequence length, so doubling the context more than doubles the compute per step.
- Latency — more tokens to read means slower responses, which matters when a step is on the critical path of a user's request.
- Forgetting — long contexts are not used uniformly.
Lost in the middle
Models do not read a long context like a human skims a document. Recall tends to be strongest at the beginning and end of the prompt and weakest in the middle — so a fact buried in the centre of a 100k-token transcript is the easiest thing for the agent to miss. This is not a memory failure you can prompt away; it is a property of how attention distributes over long sequences.
The levers
- Retrieval — keep the context small and fetch the relevant slice when it is needed, instead of carrying everything (the memory chapter).
- Compaction — periodically summarise old steps and replace the raw transcript with the summary, trading detail for room.
- Tool-result pruning — drop verbose tool outputs once their conclusion is recorded.
- Prompt caching — reuse the unchanged prefix of a prompt across calls so the provider does not recompute it; it cuts cost and latency but not the window.
Careful
A bigger window is not a free upgrade. Filling it with everything "just in case" raises cost on every step and can lower accuracy by burying the signal. The skill is choosing what stays on the desk — and that is context engineering, which the memory chapter takes up in detail.
Check yourself
Eduspheria wiki · Agentic AI, Foundations
0 / 4 answered
Next: knowing when not to build an agent at all.