Data pipelines
A pipeline is a DAG of idempotent steps with explicit dependencies — and the wall-clock time is its longest path, not the sum of its parts.
The pipeline below looks like boxes and arrows, and that is exactly the point. Once extraction, validation, transformation, feature building and training are separate steps with declared dependencies, three properties become possible that a single monolithic script can never offer: a failed step stops its dependents instead of feeding them garbage, independent steps run in parallel, and the whole thing can be re-run safely. Click a node to make it fail and press Run.
Click nodes to make them fail, then run the pipeline
Tip: fail feature store — train, evaluate, register and deploy all skip, but catalog still completes on its own branch.
The durations and topology are illustrative, but the scheduling logic is real: a node starts only when every dependency has finished, so the parallel catalog branch runs alongside the model path and the total time is the longest chain, not the sum of all steps.
The DAG is the contract
A directed acyclic graph says two things: which steps need which inputs, and that no step waits on itself. From that one structure you get failure containment, parallelism, and reproducibility for free. Most pipeline bugs are really DAG bugs — a missing edge, a cycle hidden behind a shared table, or a step that reads the output of a step it never declared.
Idempotence and the re-run
The property that separates a data pipeline from a script is idempotence: running a step twice leaves the same result. Idempotence is what makes retries safe. It comes from writing to a fresh partition, upserting on a key, or replacing a table atomically, rather than appending blindly. Without it, a failed-and-retried run silently duplicates rows, and every downstream average is wrong in a way no schema will catch.
The same idea extends to event time versus processing time. Events arrive late and out of order, so a batch that runs at midnight may not have seen all of yesterday's data. A pipeline must decide how long to wait for stragglers, and record that decision, or its "daily" numbers will keep changing after the fact.
Batch, streaming, and the features in between
- Batch processes bounded chunks on a schedule; simple, cheap, and easy to reason about, but always behind.
- Streaming processes unbounded data as it arrives; low latency, but correctness under out-of-order arrival is genuinely hard.
- Feature stores sit between pipelines and models, serving the same feature definition for training (offline, historical) and inference (online, current) so the two cannot diverge. Training-serving skew — a feature computed one way in the notebook and another way in production — is one of the most common causes of a model that works offline and fails online.
Careful
Orchestration is not optional at scale. A cron job with no dependency tracking, no retries and no alerting will fail silently at 2 a.m., and the first sign will be a dashboard that looks slightly off a week later. Make every step observable, give it a retry policy, and make the pipeline itself a monitored artifact rather than a thing someone happens to know how to run.
Illustrative vs real
The artifact uses nine nodes and fixed durations to make scheduling visible. Real pipelines have hundreds of steps, variable runtimes, and a scheduler with resources, retries and backfills. The topological logic and the critical-path arithmetic are real; the durations and topology are a teaching example.
Check yourself
Eduspheria wiki · Data, MLOps & Deployment, Data engineering
0 / 4 answered
From the assignment paper
Modeled on NITJ AI-505, Assignment/Quiz
0 / 5 answered
Where next: data quality and lineage — measuring whether a pipeline's output can be trusted, and tracing it back when it cannot.