Eduspheria Wiki
Core10 min read

Multimodality

Patches, spectrograms, projectors — how one transformer reads images and audio by turning them into tokens.

Nothing in the transformer cares what its tokens came from — the architecture consumes vectors with position. That indifference is the door to multimodality: if you can turn an image or a sound into a sequence of vectors, the same model attends over them, reasons across them, and generates in them. Tokens all the way down.

Start here

There is no "vision model" special sauce. An image is chopped into patches the way text is chopped into words, each patch becomes a vector with a position, and attention does the rest — which is why a multimodal model can answer "what's funny about this chart?" without any new architecture.

From pixels to tokens

A ViT (vision transformer) splits an image into a grid of non-overlapping patches — typically 14×14 or 16×16 pixels — flattens each patch, and linearly projects it into the model's dimension:

An image becomes a sequence of patch tokens — same transformer, new alphabet

the image (16×16 pixels)

→
▦
▦
▦
▦

2×2 = 4 patch tokens

Each patch is flattened to a 768-d vector, linearly projected, and fed to the same transformer as text — with a learned position embedding, since patches also live in order. Smaller patches see more detail but cost a longer sequence: the same tradeoff as resolution in an image file.

The tradeoff the slider shows is the central one: smaller patches → finer detail → longer sequences → quadratic attention cost. Modern VLMs use native dynamic resolution — patch grids that match the image's aspect ratio — because fixed resizing destroys document understanding.

Vision + language in one embedding space

Two designs coexist, and they build different capabilities:

  1. Contrastive pretraining (CLIP-style) — an image encoder and a text encoder train together to place matching image–caption pairs nearby in one shared space, and mismatched pairs far apart:

    L=−12(Li→t+Lt→i),Li→t=−log⁡exp⁡(sii/τ)∑jexp⁡(sij/τ)\mathcal{L} = -\frac{1}{2}\left(\mathcal{L}_{\text{i}\to\text{t}} + \mathcal{L}_{\text{t}\to\text{i}}\right), \quad \mathcal{L}_{\text{i}\to\text{t}} = -\log \frac{\exp(s_{ii}/\tau)}{\sum_j \exp(s_{ij}/\tau)}

    The result is a shared semantic space — the backbone of image retrieval and, later, the grounding layer for retrieval-augmented multimodal models (the retrieval lesson in the frontier chapter).

  2. Vision encoder + projector + LLM (LLaVA-style) — a frozen (or lightly tuned) CLIP-class encoder produces patch vectors; a small projector maps them into the LLM's embedding dimension; the LLM then treats image tokens as just more tokens in the sequence. This is the recipe behind most current chat-with-images products: maximal reuse of a strong LLM.

Audio

Speech follows the same recipe with different tokenizers: spectrogram patches (vision-style) or neural codecs — models that compress raw waveforms into discrete codes, so a sound becomes a token sequence like text. Generation runs the tokenizer in reverse. This is why voice modes can both listen and speak inside one model stack rather than chaining separate speech-to-text and text-to-speech systems.

What it doesn't magically solve

  • "Seeing" is not one skill — VLMs that ace photo description still miscount objects, misread dense OCR, or fail at precise spatial relations ("is the dot left of the square?"). Perception decomposes into sub-skills that scale unevenly.
  • Resolution is attention budget — reading a full-page document at usable fidelity can exceed affordable sequence length; hence tiling, cropping, and task-specific resolution routing.
  • Generation ≠ understanding — producing images usually runs a separate (diffusion or token-decoder) model; "one model for all modalities, both directions" remains frontier work.

Next: the final foundations lesson — how all of this becomes generated text, one sampled token at a time.