Batch processing is a data processing model in which a job reads a bounded set of data, such as all transactions from yesterday or every file in a folder, processes it as a single unit of work, and writes the complete result before finishing. Jobs typically run on a schedule or on a trigger, and results become available only after the whole batch completes.
In data processing and data engineering, batch processing is the established approach for ETL and ELT pipelines, warehouse loads, reporting, billing runs and model training. It trades latency of minutes to hours for high throughput, simple recovery and reproducible results, in contrast to stream processing, which handles unbounded data continuously.
How batch processing works
A batch job follows a read, transform, write cycle over a defined input. The input is fixed when the job starts, typically identified by a time partition (for example the date 2026-11-09) or a list of files. The job applies transformations such as data cleaning, deduplication, joins, data aggregation and enrichment, and then writes output to a target table, file or system.
Because the input is bounded, the job knows when it is complete, and it can sort, join and aggregate the whole dataset at once. Large batch jobs are executed by distributed engines that split the data into partitions, process them in parallel on many machines and combine the results, a pattern popularized by MapReduce and refined by Apache Spark.
The main components of a batch system are summarized below.
| Component | Role | Common examples |
|---|---|---|
| Storage layer | Holds input and output data | Object storage, distributed file systems, warehouses |
| Processing engine | Executes transformations in parallel | Apache Spark, MapReduce, warehouse SQL engines |
| Orchestrator | Schedules jobs and manages dependencies | Apache Airflow, Dagster, cron |
| Data format | Determines read efficiency | Parquet, ORC, Avro, CSV |
| Metadata and catalog | Tracks tables, schemas and partitions | Hive metastore, data catalogs |
| Monitoring | Detects failures and late runs | Job logs, SLA alerts, data quality checks |
Key parameters of a batch job
Batch design revolves around a few parameters that determine freshness, cost and reliability. The most important is the batch window: the interval of data each run covers and the time available to process it before downstream consumers need the result.
| Parameter | Typical range | What it affects |
|---|---|---|
| Schedule frequency | Every 15 minutes to monthly, most often daily | Data freshness and compute cost |
| Batch size | Megabytes to many terabytes per run | Runtime, memory and parallelism |
| Partition size | Commonly about 100 MB to 1 GB per task | Parallel efficiency and overhead |
| Processing window | Minutes to several hours | Deadline for downstream reports |
| Retry policy | One to several automatic retries | Resilience to transient failures |
| Load strategy | Full reload or incremental | Runtime and correctness of updates |
Incremental loads, which process only new or changed records since the last run, reduce cost dramatically but require reliable change detection through timestamps, change data capture or watermark columns. Full reloads are simpler and self-correcting but become impractical as data grows.
Batch processing compared with stream processing
Batch and stream processing differ mainly in how data is bounded and therefore in latency. Many organizations run both: streaming for operational reactions within seconds and batch for authoritative, reconciled results.
| Aspect | Batch processing | Micro-batch | Stream processing |
|---|---|---|---|
| Input | Bounded dataset | Small bounded slices | Unbounded event stream |
| Typical latency | Minutes to hours | Seconds to minutes | Milliseconds to seconds |
| Completeness | Known when the job ends | Per slice | Estimated with watermarks |
| Recovery | Rerun the job | Rerun the slice | Checkpoints and replay |
| Operational complexity | Lowest | Moderate | Highest |
| Typical uses | Warehouse loads, billing, training | Near-real-time dashboards | Fraud detection, alerting |
Two architecture patterns combine them. The Lambda architecture runs a batch layer and a speed layer in parallel and merges their views, at the cost of maintaining two code paths. The Kappa architecture treats everything as a stream and reprocesses history by replaying the log. Modern engines narrow the gap by running the same code in both batch and streaming modes.
Designing reliable batch pipelines
The most important property of a production batch job is idempotency: running it twice for the same input must produce the same output, not duplicate it. This is usually achieved by overwriting a whole partition or by merging on a key, rather than appending rows blindly.
- Parameterize each run by its logical date, so any day can be reprocessed or backfilled.
- Write to a staging location first and publish atomically, so consumers never read half-written results.
- Validate data before publishing, checking row counts, null rates and key uniqueness.
- Declare dependencies explicitly in the orchestrator instead of relying on fixed start times.
- Use columnar formats and partition pruning so jobs read only the data they need.
Common failure symptoms point to specific causes, as the table below shows.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Duplicated rows after a retry | Non-idempotent append | Overwrite partitions or merge on keys |
| One task runs far longer than others | Data skew on a hot key | Salt keys or repartition |
| Job misses its deadline | Growing data with full reloads | Switch to incremental processing |
| Reports miss recent records | Late-arriving data after the run | Reprocess a trailing window |
| Many tiny output files | Excessive partitioning | Compact files after writing |
Limitations of batch processing
The fundamental limitation is latency: a daily job means decisions are based on data that is up to a day old, which is unacceptable for fraud prevention, live personalization or operational alerting. Batch systems also concentrate load in peak windows, often overnight, which can create contention when many jobs depend on the same sources. Finally, long dependency chains mean that one late upstream job delays everything downstream.
In plain words
Batch processing means collecting work and doing it all at once. A laundromat does not wash each sock the moment it gets dirty. It waits for a full load and runs the machine.
Computers do the same with data. Every night, a system might take all of the day’s sales and total them in one run. It is efficient and easy to repeat, but the results are only as fresh as the last run.
A real-world example
A mobile network operator produces customer bills once a month from call, data and messaging records. Each day a batch job loads roughly a billion usage records into partitioned Parquet files. On the billing date a Spark job joins the month’s usage with tariff plans and discounts, computes charges per subscriber and writes invoices to a staging table.
Validation checks compare totals with the previous month and flag subscribers whose bill changed by more than a set threshold. Only after the checks pass is the staging table published atomically to the invoicing system. When a tariff error was once discovered, the team reran the job for the affected month and regenerated correct invoices without any manual data repair.