1. The Problem
When a Gold table metric anomaly appeared, data engineers spent 4 days manually searching through raw Bronze files to find which source batch introduced the bad record.
2. What I Initially Thought
3. What I Learned
Data Lineage attaches metadata tags (`_source_file`, `_batch_id`, `_pipeline_run_id`) to every record as it flows from Bronze to Silver to Gold layers.
4. What I Built
Lineage Metadata Propagation framework stamping source file path and pipeline run ID onto every row throughout transformation stages.
# Inject Lineage Metadata at Bronze Ingestion
bronze_df = raw_df \
.withColumn("_source_file", input_file_name()) \
.withColumn("_ingested_at", current_timestamp()) \
.withColumn("_run_id", lit(pipeline_run_id))5. The Experiment
Root-cause tracking for data anomalies required 4 days of manual file scanning and grep searching.
Instrumented automated lineage tracking (`_source_file`, `_run_id`) across Bronze, Silver, and Gold transformations.
Root-cause payload isolation time reduced from 4 days to 45 seconds using direct SQL queries on lineage metadata.
6. What Went Wrong
Lineage columns were dropped during PySpark `.groupBy()` aggregation operations. Preserved lineage attributes by aggregating `first(_source_file)`.
7. Engineering Decision & Trade-offs
Mandated `_source_file` and `_run_id` as non-nullable standard metadata attributes across all data lake tables.
8. What I Would Do Differently in Production
Integrate OpenLineage API collectors with Apache Airflow to generate interactive visual data lineage graphs automatically.
Questions I Can Now Answer Confidently in an Interview:
- Why is row-level data lineage critical in multi-stage data lakehouse architectures?
- How do you preserve lineage attributes through Spark aggregations and joins?
- What is OpenLineage and how does it capture metadata across data platform stacks?