Event time, watermarks, and state

Event time is when an event occurred according to its source timestamp; processing time is when the engine handled it. Events may arrive late or out of order because of retries, network delays, or offline devices. Stateful operations such as windowed aggregation retain intermediate data, and a watermark communicates how late the engine should continue considering older event-time data. A watermark is an operational tradeoff: a longer delay can include more late events but increases state retention; a shorter delay reduces state but may exclude delayed records from results. Define how corrections are exposed to consumers and test data arriving both within and beyond the expected lateness. Practice task: choose a watermark for telemetry that is usually delayed by minutes but occasionally arrives hours late. Applying the pattern: Use an event-time field, define allowed lateness, aggregate with bounded state, and verify how late records update or miss output. Checks and failure modes: A watermark is not a promise that all older events will be retained forever. Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract. Practice task: What is a primary role of a watermark in stateful streaming?