Managed connectors and ingestion choices

Data can arrive from databases, SaaS applications, object storage, and message buses. Lakeflow Connect provides managed and customizable ingestion choices; standard connectors and custom pipelines provide other tradeoffs. Start with a managed connector when it supports the source and required behavior, because it can reduce operational burden. Move to a more customizable option when the source, transformation, or delivery guarantees require it. Compare initial-load support, incremental behavior, delete capture, latency, schema drift, source API limits, and operational ownership. Validate what “exactly once” or “incremental” means for the specific connector rather than assuming all connectors behave identically. Practice task: select an ingestion approach for a SaaS source with rate limits and a daily freshness target. Applying the pattern: Document source capabilities and service limits, choose the least operationally complex compatible connector, and validate its incremental semantics. Checks and failure modes: Connector names do not guarantee identical support across regions, source plans, or workspace configurations; verify current availability. Before production, test both a representative input and a boundary case. Confirm the expected row grain, permissions, recovery behavior, and observable output for the task. Record the configuration needed to reproduce the run and compare its result with the agreed data contract. Practice task: What is a good starting point when selecting an ingestion connector?