Building with generative APIs

The model is the easy part. Once a capable checkpoint exists, turning it into a product is ordinary distributed-systems work wrapped around an unusual function: one that charges money per token, fails under rate limits, can hallucinate, and is influenced by text that came from an untrusted user.

Every hosted generative service exposes roughly the same shape — authenticate, send a model id plus inputs plus parameters, receive a structured response with token usage. Learning that shape once is enough to build against text models, image models and multimodal models, because the differences are in the payloads, not in the protocol.

Wire a request through a model and into post-processing. The mock JSON, the token count and the cost estimate are all computed from the controls.

{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this review in one sentence: the battery lasts two days but the screen is dim."
    }
  ],
  "temperature": 0.7,
  "max_tokens": 128,
  "n": 1
}

{
  "id": "mock-request-0001",
  "model": "gpt-4o-mini",
  "created": 1735689600,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "[mock completion] Summarize this review in one sentence: the battery lasts two days but th..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "completion_tokens": 83,
    "total_tokens": 104
  }
}

The anatomy of a request #

Four things are in every call, in text and image generation alike:

1. Authentication. An API key in an Authorization: Bearer … header. It is a credential that spends money, so it lives in a server-side secret store and never in a mobile bundle, a browser page or a URL.
2. Model id. Which checkpoint to run. Ids are versioned; pin them, because "the model" changes under you and silent upgrades break evaluations.
3. Input. A list of messages for chat models, a single prompt for image models, or a mix of text and image parts for multimodal models.
4. Parameters. temperature, max_tokens, n, and a sampling top-p. These trade variety against determinism and cap the cost.

A minimal chat request looks like this:

{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "system", "content": "You are a concise summarizer." },
    { "role": "user", "content": "Summarize: ..." }
  ],
  "temperature": 0.2,
  "max_tokens": 128
}

Response schema, streaming and finish reasons #

The response carries the generated content, a usage block, and a finish_reason. finish_reason: "length" means the model was cut off by max_tokens — the single most common cause of truncated JSON in production. Streaming (server-sent events) sends tokens as they are produced so a UI can start rendering before the full completion is billed and returned, but it makes assembling the final object your responsibility.

Tokens, cost and the arithmetic of scale #

You are billed per token for the prompt and per token for the completion, at different rates. For a single call,

where the prices are in USD per million tokens. Because the prompt is re-sent on every request, an agent that stuffs a thousand tokens of instructions into every call pays for those instructions every time; trimming context is a direct cost optimization. Image models price per generated image instead, so their cost scales with resolution and count, not with tokens. Latency, similarly, scales with output tokens far more than with input tokens.

Rate limits and back-pressure #

Hosted inference is quota-bound. Providers enforce requests-per-minute and tokens-per-minute with a token-bucket limiter and return HTTP 429 with a Retry-After header when you exceed it. The correct response is not to hammer the endpoint but to back off exponentially with jitter:

and to degrade gracefully — queue, cache, or return a partial result. Buying more quota is a capacity decision, not an architectural fix.

Grounding: retrieval over parametric memory #

A model's weights are a lossy, undated memory. Retrieval-augmented generation replaces unaided recall with an explicit lookup: embed the query, search a vector index for the $k$ most similar chunks, and condition the answer on them,

The generation is now traceable — you can cite the chunks — and updatable without retraining, which is why document assistants are built this way. The hard parts are chunking, ranking and evaluation, not the model call: a wrong retrieval produces a fluent, confident, wrong answer.

Caching: exact, semantic and prompt caches #

Most production traffic repeats itself. An exact cache keyed on the request hash removes duplicate calls outright. A semantic cache keys on an embedding of the query so near-duplicates also hit. Many providers additionally offer a prompt cache that bills repeated prompt prefixes at a reduced rate. All three trade a storage layer and an eviction policy for a large cut in cost and latency, at the price of staleness — a cached answer can be out of date the moment the source document changes.

Where next: with prompting and the API surface in hand, the applied-AI domain continues into the video, graph, medical and signal chapters — or, if you want the infrastructure view, the Systems for AI domain picks up serving, GPU computing and concurrency.