1. The Problem
Raw Bronze logs stored as uncompressed JSON occupied 120 GB of cloud storage, costing thousands in storage fees and slowing down analytical queries.
2. What I Initially Thought
3. What I Learned
Parquet stores data column-by-column with dictionary encoding and Snappy compression. Columnar layout allows query engines to read ONLY the required columns off disk.
4. What I Built
Parquet conversion module converting Bronze JSON logs into Snappy-compressed columnar Parquet files in the Silver ADLS layer.
import pyarrow.json as pajson
import pyarrow.parquet as pq
table = pajson.read_json("bronze_events.json")
pq.write_table(table, "silver_events.parquet", compression="snappy", use_dictionary=True)5. The Experiment
Querying 120 GB raw Bronze JSON dataset. Storage size: 120 GB. Query scan time: 3.4 minutes.
Converted raw JSON dataset into Snappy-compressed Apache Parquet format.
Storage footprint reduced from 120 GB to 21.6 GB (82% reduction). Analytical query scan time dropped from 3.4 minutes to 8.2 seconds.
6. What Went Wrong
Initial Parquet write omitted explicit data types, causing PyArrow to convert integer status codes into float64.
7. Engineering Decision & Trade-offs
Mandated Apache Parquet as the standard storage format for all Silver and intermediate data lake layers.
8. What I Would Do Differently in Production
Ensure Parquet file row group sizes are tuned (typically 128 MB to 512 MB) to maximize vectorization in Spark and DuckDB engines.
Questions I Can Now Answer Confidently in an Interview:
- How does columnar Parquet storage differ from row-oriented CSV or JSON storage?
- What are projection pushdown and predicate pushdown in Parquet file reading?
- Why does dictionary encoding dramatically reduce storage volume for repeated string columns?