CASE STUDIES LIST/ HOME
← Case #22#23 / 25Case #24 →
CASE STUDY #23Phase 6: Data Reliability & Black BoxCLASSIFICATION: ACTUAL

Tracking an Error Backward Through Multiple Transformation Layers

Data Lineage Propagation and End-to-End Audit Trail Instrumentation

OpenLineageApache AirflowPySparkPostgreSQL

1. The Problem

When a Gold table metric anomaly appeared, data engineers spent 4 days manually searching through raw Bronze files to find which source batch introduced the bad record.

2. What I Initially Thought

"I thought pipeline execution logs in Airflow were sufficient for debugging. But execution logs tell you when a job ran, not where a specific row originated."

3. What I Learned

Data Lineage attaches metadata tags (`_source_file`, `_batch_id`, `_pipeline_run_id`) to every record as it flows from Bronze to Silver to Gold layers.

End-to-End Data LineageOpenLineage StandardsAudit Metadata InjectionRoot Cause Backward Tracing

4. What I Built

Lineage Metadata Propagation framework stamping source file path and pipeline run ID onto every row throughout transformation stages.

# Inject Lineage Metadata at Bronze Ingestion
bronze_df = raw_df \
  .withColumn("_source_file", input_file_name()) \
  .withColumn("_ingested_at", current_timestamp()) \
  .withColumn("_run_id", lit(pipeline_run_id))

5. The Experiment

BEFORE

Root-cause tracking for data anomalies required 4 days of manual file scanning and grep searching.

CHANGE APPLIED

Instrumented automated lineage tracking (`_source_file`, `_run_id`) across Bronze, Silver, and Gold transformations.

AFTER RESULT

Root-cause payload isolation time reduced from 4 days to 45 seconds using direct SQL queries on lineage metadata.

6. What Went Wrong

Lineage columns were dropped during PySpark `.groupBy()` aggregation operations. Preserved lineage attributes by aggregating `first(_source_file)`.

7. Engineering Decision & Trade-offs

Mandated `_source_file` and `_run_id` as non-nullable standard metadata attributes across all data lake tables.

8. What I Would Do Differently in Production

Integrate OpenLineage API collectors with Apache Airflow to generate interactive visual data lineage graphs automatically.

Questions I Can Now Answer Confidently in an Interview:

  • Why is row-level data lineage critical in multi-stage data lakehouse architectures?
  • How do you preserve lineage attributes through Spark aggregations and joins?
  • What is OpenLineage and how does it capture metadata across data platform stacks?

Expected / Verified Evidence

•Lineage SQL query lookup output
•PySpark lineage injection module (pipelines/common/lineage.py)
•OpenLineage execution DAG snapshot
BACK TO ALL CASE STUDIESNEXT: CASE #24 (Can We Reproduce Yesterday's Production Pipeline Bug?)