A batch job is a unit of non-interactive work that processes a collection of records or tasks in a single run, without a user waiting for the result. In scheduling, a batch job is typically started by a task scheduler at a defined time, after another job completes or when enough input data has accumulated, and it ends when all items are processed or an error stops it.
Batch processing is one of the oldest patterns in computing, dating from mainframe job control, and remains central to payroll, billing, reporting, data warehousing and machine learning pipelines. Its core trade-off is throughput over latency: work is grouped for efficiency, so results arrive later than in real-time processing.
How a batch job runs
A batch job follows a predictable lifecycle. A scheduler decides that the job should start, places it in a job queue, and a worker with available capacity picks it up. The job reads its input, processes items one by one or in chunks, writes its output and reports a final status that downstream jobs can depend on.
The table below shows the usual states a batch job passes through and what each state means for operators.
| State | Meaning | Operator concern |
|---|---|---|
| Scheduled | Start time or condition defined | Correct schedule and time zone |
| Queued | Waiting for a free worker | Queue length and wait time |
| Running | Processing input | Progress and resource use |
| Succeeded | All items processed | Output verified |
| Failed | Stopped by an error | Retry or manual fix |
| Timed out | Exceeded its allowed duration | Stuck process or input growth |
Large batch jobs usually process data in chunks or partitions, for example one thousand records or one day of data at a time. Chunking limits memory use, allows progress to be recorded after each chunk and makes it possible to resume from the last checkpoint instead of restarting the whole job after a failure.
Scheduling and triggering batch jobs
Batch jobs are started in several ways. The classic method is a time-based schedule, often written as a cron expression such as 0 2 * * * for every day at 02:00. Modern systems also trigger jobs from events, such as the arrival of a file, or from job dependencies, where a job starts only after its upstream jobs succeed.
| Trigger type | How it works | Typical use |
|---|---|---|
| Time-based | Cron expression or calendar rule | Nightly reports, billing runs |
| Dependency-based | Starts after upstream jobs succeed | Data pipeline stages |
| Event-based | File arrival or message on a queue | Processing incoming uploads |
| Threshold-based | Starts when enough items accumulate | Bulk email or payment batches |
| Manual or backfill | Operator starts a run for a period | Reprocessing historical data |
Scheduling decisions must account for time zones and daylight saving transitions, which can skip or duplicate a run scheduled at 02:30 local time. Many teams therefore schedule critical jobs in UTC. Concurrency limits prevent two instances of the same job from overlapping when a run takes longer than the schedule interval.
Batch jobs compared with other processing models
Batch jobs are one of several models for doing work in the background. Choosing among them depends on how fresh the results must be and how the work arrives.
| Model | Latency | Best suited for |
|---|---|---|
| Batch job | Minutes to hours | Large periodic workloads |
| Background task | Seconds | Individual requests offloaded from a web app |
| Delayed job | Set delay | Reminders, follow-up actions |
| Micro-batch | Seconds to minutes | Near-real-time analytics |
| Stream processing | Milliseconds to seconds | Continuous event handling |
The boundaries are not strict. Frameworks such as Celery and Sidekiq run individual background tasks but are frequently used to orchestrate batch work, and a distributed job scheduler can run thousands of batch jobs across a cluster. The distinguishing property of a batch job remains that it processes a bounded set of input and then finishes.
Designing robust batch jobs
A well-designed batch job can be rerun safely. This property, called idempotency, means that processing the same input twice produces the same result rather than duplicate payments or records. It is achieved with unique keys, upsert operations or by writing output to a new location that replaces the old one atomically.
Robust jobs also define their failure behavior explicitly. A job retry policy with exponential backoff handles transient errors such as network timeouts, while a job timeout stops runs that hang. Records that fail validation are often moved to an error table instead of stopping the entire job, so one malformed row does not block thousands of valid ones.
| Parameter | Typical setting | Purpose |
|---|---|---|
| Schedule | Cron expression in UTC | Predictable start times |
| Chunk size | Hundreds to tens of thousands of records | Memory use and checkpoint frequency |
| Retry count | Two to five attempts with backoff | Recover from transient failures |
| Timeout | A multiple of normal run time | Detect stuck runs |
| Concurrency | One instance per job unless partitioned | Avoid overlapping writes |
| Priority | Higher for business-critical jobs | Order in a shared queue |
Common problems and monitoring
Batch failures are often discovered late, because nobody watches a job that runs at night. Effective monitoring tracks not only failures but also missing runs and unusual durations.
- Silent non-runs: the scheduler stopped, and no job failed because none started.
- Growing run times as input volume increases until jobs overlap with business hours.
- Partial output after a mid-run crash, read by downstream jobs as complete.
- Duplicate processing when a non-idempotent job is retried.
- Hidden dependencies on files or tables that another team changes without notice.
A dependable safeguard is a heartbeat or dead-man check, which raises an alert when a job has not reported success within its expected window. Recording row counts and durations for every run turns the job history into an early warning of trouble.
The idea in brief
A batch job is a chore that a computer does in one go, usually when nobody is waiting for it. Instead of handling each item the moment it arrives, the system collects the work and processes it all at once.
It is like doing the laundry once a week instead of washing each shirt separately. Grouping the work saves effort, but the results are ready only after the whole load is finished.
A practical scenario
A utility company bills 300,000 customers each month. Meter readings arrive throughout the day, but invoices are produced by a nightly batch job scheduled at 01:00 UTC. The job processes customers in chunks of 5,000, records a checkpoint after each chunk and writes invoices with a unique key per customer and billing period.
One night the database becomes unavailable for several minutes partway through the run. The job retries with backoff, resumes from the last checkpoint and finishes about 40 minutes late. Because the invoice writes are idempotent, no customer receives a duplicate bill, and a dead-man alert confirms success before the support team starts work in the morning.