Wiki
Core11 min read

Data quality and lineage

Trust is a measurement, not a feeling: define quality dimensions, score them per table, and trace every downstream table a broken source can reach.

A dashboard says revenue dropped 8% overnight. Is the business in trouble, or did an upstream feed fail to load? Without lineage you cannot tell the two apart, and you will spend the morning in a meeting about a bug. Data quality and lineage are what let an organisation answer that question in minutes and degrade gracefully instead of confidently.

The artifact below makes the mechanism tangible. Each table in a small warehouse has a quality score built from completeness, validity and freshness, and each has declared lineage parents. Degrade a source table and watch the risk propagate downstream — the blast radius is computed from the graph, exactly as a real lineage service would compute it.

Select a table, degrade its quality, and trace the lineage blast radius

events

quality score = 0.872 (0.4 · completeness + 0.4 · validity + 0.2 · freshness)

Tables marked at risk: none

Blast radius of events: 4 downstream tables (orders, facts, features, model)

Real quality monitoring evaluates these metrics per column and per rule, with thresholds agreed with the data owners. Here the graph, the weighted score and the risk propagation are all computed, but the metric inputs are illustrative — drag a source table down and the model input turns red along the lineage, which is the lesson.

Quality is per-dimension and per-project

"Good data" means different things to a billing system, a model, and an analyst. Name the dimensions you care about — completeness, uniqueness, validity, timeliness, consistency, accuracy — define a measurable rule for each, and agree a threshold with the data owner. A vague quality goal cannot be monitored and will be quietly abandoned after the first incident.

The dimensions, made measurable

  • Completeness — the fraction of rows or fields that are populated: completeness=1−#nulls#expected\text{completeness} = 1 - \frac{\#\text{nulls}}{\#\text{expected}}.
  • Uniqueness — the fraction of records that are not duplicates of a key.
  • Validity — the fraction satisfying a domain rule (a date in range, a country code in the reference list, a value within an allowed set).
  • Timeliness / freshness — how old the data is relative to its expected cadence, often measured as lag in hours.
  • Consistency — agreement across systems: the order total in the warehouse matching the payments ledger.

A composite score is a weighted sum, and the weights are a policy decision. The artifact uses 0.40.4 completeness, 0.40.4 validity and 0.20.2 freshness; a regulated finance table would weight validity far more heavily.

Lineage: from provenance to blast radius

Lineage is the graph of how each table was produced from others: the nodes are datasets and transformations, the edges are dependencies. It answers two operational questions:

  1. Root cause — going upstream from a broken dashboard to the source that corrupted it.
  2. Blast radius — going downstream from a bad source to everything it contaminated, so you know what to quarantine and what to recompute.

Provenance in the database sense can be why-provenance (which input tuples contributed) or where-provenance (where in the input a value came from); for operations, table-level lineage is usually enough, and column-level lineage is the gold standard when it exists.

Careful

Lineage decays the moment it is maintained by hand. If edges live in a spreadsheet or in someone's memory, they are wrong by the next sprint. Derive lineage from the transformations themselves — SQL parsers, orchestration metadata, or a catalog that instruments the pipeline — and treat a hand-declared edge as a liability rather than documentation.

Monitoring: tests that fail the pipeline

Modern practice turns quality expectations into tests that run as pipeline steps: not-null, unique, accepted-values, foreign-key, and row-count anomaly checks. A failing test halts the downstream DAG (see the previous lesson) so bad data never reaches a model. The alternative — alerting a human after the fact — means the bad number was already published.

Illustrative vs real

The seven-table lineage graph and the per-table scores here are constructed for teaching; the metrics are entered by slider, not measured. The propagation logic is real graph traversal: a table is at risk if its own score is low or any parent is at risk. Real catalogues compute this over thousands of nodes and continuous metrics.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, Data engineering

0 / 5 answered

  1. 1A table expects 5000 rows and 250 fields are null. Using 1 − nulls/expected, what is the completeness score?
    Numeric answer
  2. 2A dashboard shows a wrong number. Which direction do you traverse lineage to find the cause?
    Multiple choice
  3. 3What is the term for the set of downstream datasets affected by a corrupted source?
    Short answer
  4. 4Hand-maintained lineage documentation is a reliable long-term solution.
    True / false
  5. 5A composite score weights completeness 0.4, validity 0.4 and freshness 0.2. With completeness 0.8, validity 1.0 and freshness 0.5 (where freshness is 1 − staleness), what is the score?
    Numeric answer

From the assignment paper

Modeled on NITJ AI-505, Assignment/Quiz

0 / 5 answered

  1. 1A table is expected to hold 8000 records and 320 of its fields are null. Using 1 − nulls/expected, what is the completeness score?
    Numeric answer
  2. 2Which quality dimension is measured by the fraction of records that satisfy a domain rule, such as a date falling inside an allowed range?
    Multiple choice
  3. 3What does a confusion matrix measure?
    Multiple choice
  4. 4Mean imputation is a legitimate way of handling missing values.
    True / false
  5. 5Which statement correctly describes overfitting?
    Multiple choice

Where next: unstructured data — the text and images that do not arrive as tidy columns and have to be encoded before any of this applies.