1. The Problem
Upstream application team renamed a database column from `user_id` to `account_id` without notification. Ingestion pipeline crashed on null key lookup.
2. What I Initially Thought
3. What I Learned
Data contracts decouple ingestion pipelines from source changes. Validate incoming payloads against an explicit contract schema and route non-conforming records to a Dead Letter Queue (DLQ).
4. What I Built
Schema Drift & Contract Validation module using Pydantic models to validate incoming JSON payloads before raw storage ingestion.
class TaskIngestionContract(BaseModel):
task_id: int
tenant_id: str
status: str
updated_at: datetime
@field_validator('status')
def validate_status(cls, v):
allowed = {'PENDING', 'IN_PROGRESS', 'COMPLETED'}
if v not in allowed:
raise ValueError(f"Invalid status: {v}")
return v5. The Experiment
Pipeline crashed on unexpected upstream schema changes; invalid records halted entire batch processing.
Implemented Pydantic schema contract validator routing non-conforming records to a Dead Letter Queue (DLQ).
99.8% valid batch records ingested successfully; 0.2% schema-drifted records safely quarantined in DLQ with exact error logs.
6. What Went Wrong
Initial strict validator rejected payloads with new optional fields. Updated model configuration to `extra = 'ignore'` to allow backwards-compatible additions.
7. Engineering Decision & Trade-offs
Established explicit JSON Schema / Pydantic contracts at the ingestion boundary for all external API and webhook pipelines.
8. What I Would Do Differently in Production
Implement automated schema evolution notifications that alert data engineers when non-critical new fields are detected in raw payloads.
Questions I Can Now Answer Confidently in an Interview:
- What is Schema Drift and how does it impact data pipelines?
- How do Data Contracts prevent breaking downstream data models?
- What is a Dead Letter Queue (DLQ) and how should quarantined records be reprocessed?