Expectations and data quality

Data quality rules turn assumptions into observable checks. Examples include required identifiers, valid status values, nonnegative amounts, and event timestamps within an acceptable range. A rule must specify what happens when a record fails: track it, drop it, or fail the update. The correct action depends on severity and whether downstream consumers can safely use the remaining rows. Track both counts and rates because a small number of failures may be normal at high volume but a sudden rise can signal a source incident. Keep rejected data or diagnostic details where policy permits so teams can investigate and replay. Practice task: choose an invalid-record policy for a missing primary key and for a malformed optional description. Applying the pattern: Define a rule, select its failure action, record metrics, inspect rejected records, and make the quality result visible to downstream owners. Checks and failure modes: Dropping bad rows without a metric or quarantine path makes a pipeline look healthy while silently losing data. Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract. Practice task: Why monitor the rate of records failing a quality expectation?