Data ingestion and change data capture · Lesson 1 of 3
Batch file ingestion and Auto Loader
Incrementally discover and process new cloud files with checkpointed ingestion.
Lesson
Batch ingestion processes a bounded set of files, while incremental file ingestion repeatedly discovers newly arriving objects. Databricks Auto Loader uses the cloudFiles source with Structured Streaming to incrementally process files from cloud storage. Its schema tracking and checkpoint state help a long-running ingestion avoid rescanning every previously seen file. Configure the input path, schema location, checkpoint, and target deliberately.
A checkpoint is operational state, not a generic cache to delete whenever a run fails. Understand whether a restart should resume from existing progress or intentionally reprocess data. Test duplicate delivery, malformed files, and schema changes. Practice task: outline which storage paths hold source data, inferred schema state, and streaming checkpoint state.
Applying the pattern: Configure the source path and schema policy, use a durable checkpoint, write to a Delta target, and monitor file discovery and failures.
Checks and failure modes: Changing or removing checkpoint state can alter which files are considered processed; do so only with a defined replay plan.
Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract.
Practice task: What is a key purpose of a streaming checkpoint?
Visual guide
Incremental file ingestion
-
Step 01
Discover
Identify new files in cloud storage.
-
Step 02
Parse
Apply the configured schema policy.
-
Step 03
Checkpoint
Persist progress for restart recovery.
-
Step 04
Write
Commit records to the target table.
Code examples
Read files incrementally
PYTHON
incoming = (spark.readStream
.format('cloudFiles')
.option('cloudFiles.format', 'json')
.option('cloudFiles.schemaLocation', schema_path)
.load(source_path))Expected output
Auto Loader discovers new files while persisting schema state separately.
Key points
- Configure the source path and schema policy, use a durable checkpoint, write to a Delta target, and monitor file discovery and failures.
- Changing or removing checkpoint state can alter which files are considered processed; do so only with a defined replay plan.
- Re-run the included practice scenario after changing source data or transformation logic.
Check your understanding
Practice
Your learning progress