1. The Problem
Adding `.cache()` to 4 intermediate DataFrames caused PySpark pipeline runtime to increase by 40% and triggered high JVM Garbage Collection (GC) pauses.
2. What I Initially Thought
3. What I Learned
Spark shares JVM memory between Execution (joins/sorts) and Storage (cache). Caching large DataFrames starves Execution memory, forcing Spark to spill join operations to disk.
4. What I Built
Memory-optimized pipeline removing redundant `.cache()` calls and explicitly invoking `.unpersist()` as soon as cached DataFrames were consumed.
# Cache ONLY when DataFrame is reused in MULTIPLE downstream actions
reused_df = expensive_transformation(raw_df).persist(StorageLevel.MEMORY_AND_DISK)
# Action 1
count_val = reused_df.count()
# Action 2
reused_df.write.parquet("abfss://gold/output/")
# EXPLICITLY FREE MEMORY IMMEDIATELY
reused_df.unpersist()5. The Experiment
Caching 4 intermediate DataFrames. Pipeline runtime: 4.8 minutes. JVM GC time: 1.2 minutes. Disk Spill: 8.4 GB.
Removed unnecessary `.cache()` calls and added explicit `.unpersist()` after action completions.
Pipeline runtime dropped from 4.8 minutes to 2.1 minutes; JVM GC time reduced by 88%; disk spill eliminated.
6. What Went Wrong
Cached a DataFrame that was only referenced once in the entire DAG, forcing Spark to serialize and store data pointlessly.
7. Engineering Decision & Trade-offs
Established rule: Only cache a DataFrame if it is evaluated by 2 or more downstream materializing actions, and always call `.unpersist()` after usage.
8. What I Would Do Differently in Production
Monitor `Spark UI → Storage` tab to verify cached block sizes and ensure memory fractions leave sufficient room for execution joins.
Questions I Can Now Answer Confidently in an Interview:
- How does Apache Spark manage JVM memory between Execution Memory and Storage Memory?
- Why can over-using .cache() or .persist() degrade PySpark job performance?
- When is it appropriate to use .cache() in a PySpark workflow?