A data pipeline is an automated sequence of processing steps that extracts data from one or more sources, transforms or validates it, and delivers it to a destination such as a data warehouse, data lake, search index or application. In data engineering, the pipeline is the basic unit of production work: it turns raw, scattered records into datasets that analysts, models and products can trust.
A pipeline is more than a script that copies data. A production pipeline defines when it runs, what it depends on, how it recovers from failure and how its output is checked. ETL and ELT are specific pipeline patterns; the broader term covers batch jobs, streaming flows, reverse flows back into operational tools, and machine learning feature preparation.
Stages of a data pipeline
Most pipelines follow the same logical stages, even when tools differ. Each stage has a clear input, a clear output and its own failure modes, which makes it possible to test, monitor and rerun stages independently.
| Stage | What it does | Typical techniques |
|---|---|---|
| Ingestion | Reads data from sources | API calls, database extracts, change data capture, file drops |
| Landing | Stores raw data unchanged | Object storage, raw schema in a warehouse |
| Validation | Checks schema, completeness and ranges | Schema tests, row counts, null and uniqueness checks |
| Transformation | Cleans, joins and reshapes data | SQL models, Spark jobs, deduplication |
| Serving | Publishes data for consumers | Warehouse tables, feature stores, exports to tools |
| Observation | Tracks health and freshness | Run logs, alerts, lineage, data quality metrics |
Keeping a raw landing layer is one of the most valuable design decisions. When a transformation bug is discovered weeks later, the team can rebuild corrected outputs from the untouched source copy instead of asking source systems to resend history.
Batch, micro-batch and streaming pipelines
The most consequential design choice is how often data moves. Latency requirements drive cost, complexity and tool selection, so they should come from a real business need rather than a preference for real time.
| Mode | Typical latency | Strengths | Trade-offs |
|---|---|---|---|
| Batch | Hours to a day | Simple, cheap, easy to rerun | Data is stale between runs |
| Micro-batch | About one to fifteen minutes | Near-fresh data with batch semantics | More frequent jobs, higher overhead |
| Streaming | Milliseconds to seconds | Immediate reaction to events | State management, ordering, harder debugging |
A practical rule is to start with the slowest mode that meets the requirement. A daily sales report gains nothing from streaming, while fraud scoring on card payments cannot wait for a nightly batch. Many organizations run both: streaming for operational decisions and batch for complete, reconciled reporting.
Properties of a reliable pipeline
Reliability in data engineering rests on a few properties that experienced teams design for from the start. The most important is idempotency: running the same job twice for the same period must produce the same result, not duplicate rows.
- Idempotent writes: overwrite or merge a partition for a given date instead of blindly appending.
- Incremental processing: read only new or changed data using watermarks or change data capture, which keeps run times stable as volumes grow.
- Backfill support: parameterize jobs by date range so history can be reprocessed deliberately.
- Explicit dependencies: an orchestrator starts a step only after its upstream data is complete.
- Data contracts: producers and consumers agree on schema, meaning and freshness so changes do not break downstream jobs silently.
- Lineage: every output table records which sources and jobs produced it.
Orchestration tools schedule tasks as a directed acyclic graph (DAG), retry failed steps and record each run. They coordinate work but do not guarantee correctness: data quality tests must still verify that the output is complete and plausible.
Data pipelines compared with related concepts
The term data pipeline is often used interchangeably with neighboring terms, but each one describes a different level or pattern. The distinction helps when choosing architecture and when assigning ownership.
| Term | What it describes | Relationship to a data pipeline |
|---|---|---|
| ETL | Transform before loading into the target | A pipeline pattern suited to strict target schemas |
| ELT | Load raw data, transform inside the warehouse | The dominant pattern with cloud warehouses |
| Data lake | Storage for raw files at scale | A common landing destination |
| Data warehouse | Modeled storage for analytics | A common serving destination |
| Lakehouse | Lake storage with table and transaction features | Destination that combines both roles |
| Orchestrator | Scheduler and dependency manager | Runs and monitors pipelines, is not one itself |
Common failures and how to prevent them
The most damaging pipeline failures are silent ones, where jobs succeed technically but deliver wrong or incomplete data. A failed job raises an alert; a job that loads half the expected rows often does not.
| Symptom | Likely cause | Prevention |
|---|---|---|
| Duplicate rows after a rerun | Append-only writes, no idempotency | Partition overwrite or merge on keys |
| Sudden drop in row count | Source outage or changed filter | Volume checks with expected ranges |
| Columns suddenly null | Upstream schema change | Schema contracts and validation at ingestion |
| Dashboards show old numbers | Late or skipped run | Freshness monitoring with alerts |
| Run times keep growing | Full reloads of growing tables | Incremental loading and partitioning |
A further risk is pipeline sprawl: hundreds of small jobs built by different people without ownership or documentation. Assigning an owner to every pipeline and retiring unused outputs keeps the system maintainable and controls compute cost.
The short version
A data pipeline works like a water supply system for information. Water is collected from several sources. It is filtered and treated. Then pipes deliver it to homes, where it is ready to use.
A data pipeline does the same with data. It collects records from different systems and cleans them. It then delivers them to the place where people build reports. It runs automatically and checks itself, so nobody has to copy files by hand. When something breaks, it should raise an alarm.
Where it makes a difference
An online retailer sells through its own web shop and two marketplaces. Orders sit in three different systems, and the finance team used to merge spreadsheet exports every Monday, which took most of a working day and produced frequent mismatches.
The data team builds a pipeline that pulls new orders from all three sources every hour, lands the raw records in object storage, validates order identifiers and currencies, converts amounts to a single currency and merges everything into one orders table in the warehouse. Volume and freshness checks alert the team if any source delivers fewer orders than usual.
Finance now sees consolidated revenue within about an hour of each sale. When a marketplace changed its export format, the schema check stopped the load before wrong data reached the reports, and the fix took minutes instead of a week of corrections.