Performance, cost, and troubleshooting

Start performance troubleshooting with a symptom and evidence. Slow scans may read too many files or columns; expensive shuffles can result from wide joins, aggregations, or repartitioning; skew occurs when a small number of keys dominate the work. Query plans, Spark UI stages, input/output sizes, task duration, and job history can narrow the cause. Avoid changing cluster size before understanding whether the bottleneck is compute, data layout, or a poor plan. Measure changes on representative data and compare runtime, resource usage, and output correctness. Optimize the largest cost driver first and keep the change reversible. Databricks may provide automatic optimizations, but they do not eliminate the need for good predicates, appropriate joins, and supported table maintenance. Practice task: identify the first metric you would inspect when one join stage is much slower than the others. Applying the pattern: Capture a baseline, inspect the slowest stage and data distribution, change one likely cause, then measure the same workload again. Checks and failure modes: Caching or increasing worker count without evidence may raise cost while leaving the actual skew or scan problem unchanged. Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract. Practice task: What is a sound first step for a slow Spark job?