Data ingestion is the process of moving data from its source systems into a target platform, such as a data lake, data warehouse or lakehouse, where it can be stored, processed and analyzed. In data engineering, ingestion is the first stage of every data pipeline: it extracts records from databases, applications, files, APIs and event streams and lands them reliably in a place that the rest of the platform controls.
Ingestion deals with connectivity, format, volume, timing and reliability, while deeper reshaping of data usually belongs to later transformation stages. A well-designed ingestion layer makes data arrive complete, on time and traceable, so that every downstream report or model can trust its inputs.
How data ingestion works
Every ingestion process follows the same basic sequence: connect, extract, serialize, transfer and write new or changed data to the target, while recording what was loaded. The details differ by source type, but the reliability concerns are constant.
| Stage | What happens | Key concern |
|---|---|---|
| Connection | Authenticate to a database, API, file store or broker | Credentials, network access, rate limits |
| Extraction | Read full data or only changes | Load on the source system |
| Serialization | Encode records as Avro, Parquet, JSON or CSV | Schema and type preservation |
| Transfer | Move data over the network in batches or streams | Throughput and retries |
| Landing | Write to raw storage or staging tables | Atomic writes and partitioning |
| Bookkeeping | Store offsets, watermarks and load metadata | Restartability and lineage |
Ingested data is usually landed as close to its original form as possible, for example in a raw or bronze layer of a medallion architecture. Keeping the raw copy allows later transformations to be rerun when business logic changes, without extracting from the source again.
Batch, micro-batch and streaming ingestion
The main design choice in data ingestion is timing: how soon after an event the data must be available. The choice sets cost, complexity and the freshness that consumers can expect.
| Mode | Typical latency | Strengths | Weaknesses |
|---|---|---|---|
| Batch | Hours to a day | Simple, cheap, easy to rerun | Stale data between runs |
| Micro-batch | About one to fifteen minutes | Near-real-time with batch tooling | Many small files to manage |
| Streaming | Milliseconds to seconds | Fresh data for live use cases | Complex state and operations |
Extraction strategy matters as much as timing. A full load copies the entire source each time and is acceptable only for small tables. Incremental loads use a watermark, such as an updated timestamp or increasing identifier, to read only new or changed rows. Change data capture reads the database transaction log and captures inserts, updates and deletes with minimal load on the source, which makes it the preferred method for large operational databases.
Ingestion compared with related data engineering terms
Data ingestion is often used interchangeably with ETL or pipelines, but it covers a narrower part of the flow. Clear boundaries help teams assign responsibility and choose tools.
| Concept | Scope | Relation to ingestion |
|---|---|---|
| ETL | Extract, transform, then load | Ingestion plus transformation before loading |
| ELT | Extract, load, then transform in the target | Ingestion is the extract and load part |
| Data pipeline | Any chain of data processing steps | Ingestion is the entry step |
| Change data capture | Capturing row changes from logs | A technique for incremental ingestion |
| Reverse ETL | Sending warehouse data back to applications | Ingestion in the opposite direction |
| Data orchestration | Scheduling and dependency management | Triggers and monitors ingestion jobs |
The modern preference for ELT has shifted transformation work out of the ingestion layer. Because cloud warehouses and lakehouses provide cheap storage and elastic compute, teams now ingest data with minimal changes and transform it later with SQL-based tools such as dbt. Ingestion still performs light technical steps, such as type mapping, adding load timestamps and masking sensitive fields that must never reach raw storage.
Designing reliable ingestion
Reliable ingestion is built around idempotency and observability. An idempotent load can be repeated after a failure without creating duplicates, for example by writing to a partition that is replaced as a whole or by merging on a primary key.
- Track delivery guarantees: most systems provide at-least-once delivery, so deduplication must happen downstream.
- Handle schema evolution explicitly, with a data contract or a schema registry that rejects incompatible changes.
- Protect source systems by reading from replicas or logs instead of running heavy queries on primaries.
- Record load metadata such as source, extraction time, row counts and file names for lineage.
- Monitor freshness and volume, alerting when a load is late or row counts deviate sharply from normal.
The choice between managed connectors and custom code depends on the source. Common SaaS applications and databases are well served by prebuilt connectors, while proprietary systems and unusual APIs often need custom extraction with careful handling of pagination, rate limits and retries.
Common ingestion problems
Most ingestion incidents are silent rather than loud: the job succeeds, but the data is incomplete or wrong. Monitoring that checks the data itself catches these cases.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Duplicate rows after a retry | Non-idempotent append loads | Overwrite partitions or merge on keys |
| Missing late records | Watermark set too aggressively | Add a lookback window |
| Deleted rows still present | Incremental loads ignore deletes | Use change data capture or soft-delete flags |
| Broken downstream jobs | Unannounced source schema change | Schema checks and data contracts |
| Thousands of tiny files | Very frequent micro-batches | Scheduled compaction |
The short version
Data ingestion is how information gets from where it is created to where it is analyzed. Sales happen in a shop system, clicks happen on a website and payments happen in a bank system. To study them together, copies must be collected in one place.
Ingestion is like a delivery service for data. It picks up packages from many senders, on a schedule or continuously, and drops them at one warehouse with a note of what arrived and when. If a delivery fails, it tries again without delivering the same package twice.
Where it makes a difference
A mid-sized retailer wants daily sales dashboards that combine its PostgreSQL order database, an online advertising platform and CSV files from warehouse partners. At first an analyst exports files manually each morning, and reports are often a day late or missing a source.
The data team sets up change data capture from a PostgreSQL replica into the lakehouse, a managed connector that pulls advertising data every hour, and an automatic loader that picks up partner files from object storage. Each load writes source, row counts and extraction time to a metadata table, and alerts fire when a source is more than two hours late. Dashboards are now refreshed every hour without manual work, and the team can prove which data each report contains.