Delta Lake writes and maintenance · Lesson 3 of 3
Schema enforcement and schema evolution
Protect table contracts while handling legitimate source changes.
Lesson
Delta schema enforcement checks that writes are compatible with a table's schema. Schema evolution can allow intentional changes, such as adding a new field, but should not be enabled casually for unpredictable input. A table contract describes expected columns, types, nullability, and meaning. A technically compatible schema change can still break downstream queries or alter business interpretation.
Separate additive, compatible changes from destructive changes such as renaming or changing a field's meaning. Test downstream consumers and document migration steps before updating a shared table. Capture unexpected fields for investigation when the ingestion method supports it. Practice task: decide whether a new optional source field should be accepted, quarantined, or rejected and explain why.
Applying the pattern: Detect a schema change, classify its compatibility, validate downstream dependencies, and apply an explicit migration or quarantine policy.
Checks and failure modes: Automatic schema evolution can hide upstream drift and silently change the contract relied upon by consumers.
Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract.
Practice task: What is a safe approach to an incompatible schema change?
Visual guide
Review schema drift
-
Step 01
Detect
Compare source fields to the contract.
-
Step 02
Classify
Decide whether the change is compatible.
-
Step 03
Assess
Check downstream tables and consumers.
-
Step 04
Apply
Migrate, accept, quarantine, or reject.
Code examples
Inspect an incoming schema
PYTHON
incoming.printSchema()
print(incoming.columns)
# Compare fields and types with the table contract before writing.Expected output
Review observed fields and types before accepting schema changes.
Key points
- Detect a schema change, classify its compatibility, validate downstream dependencies, and apply an explicit migration or quarantine policy.
- Automatic schema evolution can hide upstream drift and silently change the contract relied upon by consumers.
- Re-run the included practice scenario after changing source data or transformation logic.
Check your understanding
Practice
Your learning progress