Transformations, actions, and lazy evaluation

Many DataFrame operations are transformations that describe a new logical result, such as select, filter, and join. Spark can build a plan from these operations without immediately scanning all data. An action, such as writing a result or collecting rows to the driver, triggers execution. The optimizer can then combine operations and choose a physical plan based on the query and available statistics. Keep large-scale processing distributed. Calling collect or converting a large DataFrame to a local Python list can overwhelm the driver. Use explain plans and query metrics to understand scans, shuffles, and joins rather than adding cache reflexively. Practice task: classify select, filter, show, and write as transformations or execution-triggering actions in a given workflow. Applying the pattern: Compose transformations, inspect the resulting plan, then trigger execution with a bounded action or a distributed write. Checks and failure modes: Repeated actions can recompute upstream work, and collecting an unbounded result moves distributed data into driver memory. Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract. Practice task: What typically triggers execution of a lazily planned DataFrame?