Systems for AI
The infrastructure an AI engineer is expected to know: how GPU hardware and the CUDA execution model deliver throughput (AI-619), how threads, races and synchronization primitives make parallel programs correct (AI-506), and how cloud-native serving, vector indexes and request batching keep latency predictable under load (AI-602, part of AI-622). Intuition first, a playable artifact, then the math — with honest numbers about what is illustrative.
GPU computing
Why a GPU is not a fast CPU: the throughput argument, the CUDA grid/block/thread execution model and its memory hierarchy, then synchronization, streams and profiling to find where the time actually goes.
- Why a GPU is not a fast CPUA GPU wins by doing thousands of slow things at once, not one thing quickly. Peak throughput is arithmetic; reaching it is a question of regularity. 11 min · core
- Threads, blocks, and the memory hierarchyThe grid/block/thread mapping turns a loop into thousands of lanes. Where a value lives — register, shared, global — then decides how fast those lanes run. 12 min · core
- Synchronization, streams, and profilingBarriers make shared memory correct, divergence makes lanes idle, streams overlap independent work — and a profiler tells you which of them actually costs you. 13 min · advanced
Concurrent programming
Shared state without shared confusion: interleavings and race conditions, the synchronization primitives that restore order (locks, semaphores, atomics), and the recurring parallel patterns — map, reduce, scan — that scale.
- Threads and race conditionsTwo threads increment the same counter and the answer is wrong. The bug is a schedule, not a line of code — and that is exactly why it is hard. 10 min · intro
- Synchronization primitivesLocks, semaphores and atomics create the ordering edges that make shared state safe — each with a different cost and a different failure mode. 12 min · core
- Parallel patterns: map, reduce, scanMost parallel programs are combinations of three shapes — map, reduce and scan. Their work and span tell you the speedup before you write a line of code. 12 min · core
Serving at scale
From a model artifact to a service: containers and orchestration, vector databases and approximate nearest-neighbour indexes, and the batching/latency/throughput tradeoff that governs every inference endpoint.
- Cloud-native AI: containers and orchestrationA trained model is not a service. Containers freeze the environment, an orchestrator reconciles a declared replica count, and the loop keeps serving through failure. 12 min · core
- Vector databases and approximate searchEmbeddings are only useful if you can find the nearest ones. Exact search is O(N·d); approximate indexes trade a little recall for orders of magnitude fewer comparisons. 12 min · core
- Serving and latency: the batching tradeoffEvery inference server lives on a curve: bigger batches raise throughput and lower latency quality. Queueing theory, continuous batching and tail latency decide where to sit on it. 13 min · advanced