1. The Problem
Raw JSON files, intermediate CSV exports, and final reports were dumping into a single Azure Blob container, causing file search latency and security permission leaks.
2. What I Initially Thought
3. What I Learned
Medallion architecture structures data lakes into distinct zones: Bronze (raw, immutable), Silver (cleansed, standardized Parquet), and Gold (business aggregates).
4. What I Built
Structured Azure ADLS Gen2 storage hierarchy with automated pathing (`abfss://bronze/domain/year=YYYY/month=MM/`) and RBAC storage permissions.
def generate_adls_path(layer: str, domain: str, dt: datetime) -> str:
return f"abfss://{layer}@datapulse.dfs.core.windows.net/{domain}/year={dt.year}/month={dt.month:02d}/batch_{dt.strftime('%Y%m%d_%H%M%S')}.parquet"5. The Experiment
Single flat Blob container storing 200,000 un-partitioned raw and processed files. Directory search duration: 4.2 minutes.
Organized storage into Bronze/Silver/Gold ADLS containers with Hive-style directory partitioning.
Directory path resolution latency dropped to 0.12 seconds; storage lifecycle rules auto-archived raw Bronze files to Cold storage after 90 days.
6. What Went Wrong
Used uppercase characters in ADLS container names, violating Azure naming rules and breaking SDK deployment scripts.
7. Engineering Decision & Trade-offs
Standardized on lowercase Hive partitioning paths (`year=YYYY/month=MM/day=DD`) across all ADLS storage layers.
8. What I Would Do Differently in Production
Configure Azure Lifecycle Management rules to move raw Bronze JSON files to Archive storage after 90 days to reduce storage costs by 80%.
Questions I Can Now Answer Confidently in an Interview:
- What is the purpose of Bronze, Silver, and Gold layers in a Data Lakehouse?
- Why is Hive-style directory partitioning (year=YYYY/month=MM) useful in cloud object storage?
- How do storage tiering rules (Hot, Cool, Archive) reduce cloud data lake operating costs?