Most Workers Finished Fast. The Last Worker Took Forever.

1. The Problem

In a multi-tenant dataset, a single enterprise tenant owned 40% of all task records. Joining by `tenant_id` routed all 40% records to a single executor task, creating a 25-minute straggler.

2. What I Initially Thought

3. What I Learned

Data skew occurs when partition key values are unevenly distributed. Salting appends a random integer (`0-N`) to skew keys on the fact table, distributing hot keys evenly across multiple workers.

4. What I Built

Salting transformation pipeline appending random salt keys `(0..7)` to skewed tenant keys, exploding dimension tables, and performing balanced parallel joins.

# Append random salt (0-7) to skewed fact keys
salted_fact = fact_df.withColumn("salt", (rand() * 8).cast("int")) \
                      .withColumn("salted_tenant_id", concat(col("tenant_id"), lit("_"), col("salt")))

# Explode dimension table to match salt range
salted_dim = dim_df.withColumn("salt_array", array([lit(i) for i in range(8)])) \
                    .withColumn("salt", explode("salt_array")) \
                    .withColumn("salted_tenant_id", concat(col("tenant_id"), lit("_"), col("salt")))

# Join on salted key -> Balanced partition distribution!
skew_free_df = salted_fact.join(salted_dim, "salted_tenant_id")

5. The Experiment

6. What Went Wrong

7. Engineering Decision & Trade-offs

Applied salting conditionally only to top 1% high-volume tenant keys identified in metadata profiling.

8. What I Would Do Differently in Production

In Spark 3.x, enable `spark.sql.adaptive.skewJoin.enabled=true` to allow Spark AQE to automatically split skewed partitions at runtime.

Expected / Verified Evidence