A backfill is the execution of a scheduled job for past time intervals that were never processed, failed, or must be recomputed after a change in logic or source data. In scheduling, a job normally runs once per interval going forward, for example every hour or every night; a backfill instead replays a range of historical intervals, each with the parameters it would have received had it run on time.
Backfills are routine in data pipelines, reporting, billing and synchronization jobs. They are also among the riskiest scheduler operations, because a single command can launch hundreds of runs against production systems and overwrite data that people rely on.
How a backfill works in a scheduler
A backfill relies on jobs that are parameterized by a logical date or interval, not by the moment they actually run. A nightly job for 3 March processes the data of 3 March whether it executes on 4 March or six months later. The scheduler generates one run per interval in the requested range and executes them with the usual task scheduler, workers and job queue.
Three ingredients decide whether a backfill is safe: interval-based parameters, idempotent writes that can be repeated without duplicating results, and controlled concurrency so that historical runs do not starve current ones. When any of them is missing, backfilling becomes manual data surgery instead of a routine operation.
| Concept | Meaning | Why it matters |
|---|---|---|
| Logical date | The interval a run is responsible for | Separates data time from execution time |
| Interval range | Start and end of the backfilled period | Defines how many runs are created |
| Idempotency | Same input produces the same final state | Allows safe reruns of any interval |
| Partition | Unit of output, such as a day of data | Enables replace-instead-of-append writes |
| Catch-up | Automatic runs for missed intervals | Can trigger an unintended backfill |
Common reasons for running a backfill
Backfills are triggered by gaps, errors or changes. Each reason implies a different scope and different checks before starting.
| Trigger | Typical scope | What to verify first |
|---|---|---|
| Outage or failed runs | Hours to days of missing intervals | Root cause is fixed |
| New job or pipeline | Full available history | Source systems can handle the load |
| Bug fix in logic | All intervals produced by the bug | Downstream consumers are informed |
| Late-arriving source data | Specific recent intervals | Data is now complete |
| Schema or metric definition change | Whole history for consistency | Old and new results are comparable |
A backfill after a definition change is the most sensitive case, because it changes numbers that people have already seen in reports. Announcing the change and keeping the previous version available for comparison avoids confusion and mistrust.
Backfill strategies and their trade-offs
There is more than one way to execute a backfill, and the right one depends on data volume, dependencies and how much the job’s own logic can be trusted with large ranges.
| Strategy | How it runs | Advantage | Risk |
|---|---|---|---|
| Interval by interval | One run per original interval | Same code path as normal runs | Slow for long histories |
| Parallel intervals | Many intervals at once, capped | Much faster completion | Load spikes on sources |
| Bulk range run | One run covering the whole range | Efficient for large scans | Different code path, harder retries |
| Reverse chronological | Newest intervals first | Recent data usable sooner | Breaks jobs that depend on prior state |
| Shadow backfill | Writes to a separate target | Compare before switching | Needs extra storage |
Jobs with a job dependency on the previous interval, such as running balances or cumulative counters, must be backfilled in strict chronological order with a concurrency limit of one. Stateless daily aggregations, by contrast, can run in parallel safely.
Backfill compared with related scheduling operations
Backfill is easily confused with neighboring operations that also re-execute work. The distinction lies in which intervals are processed and why.
| Operation | What is executed | Typical cause |
|---|---|---|
| Backfill | A range of historical intervals | Gaps, fixes or new jobs |
| Job retry | The same failed run again | Transient error |
| Catch-up run | Intervals missed while paused | Scheduler downtime or new deployment |
| Rerun or clear | Selected existing runs | Manual correction of specific results |
| Batch job | A large unit of work, not tied to past dates | Regular bulk processing |
Running backfills safely
A careful backfill is planned like a small release. The following practices prevent the most frequent incidents.
- Dry-run the range to list the runs the scheduler will create before executing any of them.
- Start with one interval, verify the output, then widen the range gradually.
- Cap concurrency with a dedicated queue or pool, and use a lower job priority so current runs are not delayed.
- Write by partition with overwrite semantics, never by blind appends, to keep reruns idempotent.
- Protect source systems by throttling API calls and scheduling heavy reads outside peak hours.
- Disable side effects such as customer emails, invoices or alerts for historical intervals.
The last point causes some of the most visible failures: a job that sends a notification for each processed day can send months of stale messages within minutes if it is backfilled without a guard. Jobs should know whether they run on schedule or as part of a backfill and skip external side effects in the latter case.
Scheduler configuration also deserves attention. An unintended backfill often starts when a job with catch-up enabled and an old start date is deployed or unpaused, and the scheduler immediately creates runs for every missed interval. Setting an explicit, recent start date and disabling automatic catch-up for jobs that do not need history removes this risk.
The idea in brief
Imagine a diary that someone is supposed to write every evening, but they missed two weeks while traveling. Filling in those missing entries afterwards, day by day, is a backfill.
Computers do the same with regular jobs. If a nightly report did not run for a week, or ran with a mistake, the system goes back and runs the job again for each missed day, as if it had happened on time. The important part is doing it carefully, so that the past is corrected without disturbing the present.
A practical scenario
An online retailer computes daily revenue per product category in a nightly job. A developer discovers that refunds have been ignored for the last 90 days because of a filter bug. After the fix is deployed, the data team plans a backfill of 90 daily intervals.
They first run a single day and compare it with the finance system, then launch the remaining days with a concurrency limit of four on a separate low-priority queue, so that tonight’s regular run is unaffected. Each run overwrites its own date partition, and the alerting step is skipped for historical dates. The backfill finishes in a few hours, dashboards show corrected revenue for the whole quarter, and finance receives a note explaining why previously reported numbers changed.