A job dependency is a rule that makes one job wait for another job, event or condition before it is allowed to start. In scheduling, dependencies turn a set of independent tasks into an ordered workflow: a report runs only after the data load finishes, and the data load runs only after the source file arrives.
Dependencies are what separate orchestration from simple timing. A cron expression says when to try; a dependency says whether the job is ready. Together they let schedulers run large chains of batch jobs correctly even when upstream steps finish late, fail or are rerun.
How job dependencies control execution
A scheduler evaluates each job’s dependencies before dispatching it to a worker. If every upstream requirement is satisfied, the job becomes runnable; otherwise it stays waiting or blocked. When an upstream job finishes, the scheduler re-evaluates its downstream jobs, which is how a whole chain advances without fixed start times.
Most systems model dependencies as a directed acyclic graph (DAG): jobs are nodes and dependencies are edges pointing from upstream to downstream. The graph must be acyclic, because a cycle means two jobs wait for each other forever. Schedulers detect cycles at definition time and reject the workflow.
Two graph shapes appear constantly. In fan-out, one upstream job releases many downstream jobs that can run in parallel, limited only by the concurrency limit and the number of workers. In fan-in, one downstream job waits for several upstream jobs, such as a consolidation step that needs results from every region before it starts.
The scheduler also needs a status model to decide readiness. Typical states include pending, running, succeeded, failed, skipped and upstream-failed, and each dependency rule refers to one or more of these states.
| State | Meaning | Effect on downstream jobs |
|---|---|---|
| Pending | Waiting for dependencies or a slot | Downstream also waits |
| Running | Executing on a worker | Downstream waits |
| Succeeded | Finished without error | Satisfies success dependencies |
| Failed | Ended with an error after retries | Blocks downstream by default |
| Skipped | Intentionally not run | Handled according to the rule |
| Upstream failed | Cannot run because a parent failed | Propagates further down |
Types of job dependencies
Dependencies differ in what they wait for. The most common type waits for another job to succeed, but production schedulers also wait for files, external data, time windows, resources and even jobs in other workflows.
| Dependency type | Waits for | Example |
|---|---|---|
| Job-to-job | Completion of an upstream job | Transform after extract |
| Cross-workflow | A job in another workflow or schedule | Finance report after nightly ETL |
| File or data | Arrival of a file or partition | Load when the daily export lands |
| Time window | A clock time or business calendar | Not before 06:00 on working days |
| Resource | A free slot, lock or license | One database maintenance job at a time |
| External event | A signal from another system | Start after an approval is recorded |
Dependencies are evaluated with trigger rules. The default is usually all upstream jobs succeeded, but alternatives include any succeeded, all done regardless of result, or one failed, which is useful for cleanup and alerting jobs that must run when something breaks.
Job dependencies in different schedulers
Support for dependencies varies widely. Plain cron has no dependency model at all, so teams historically chained jobs with fixed time gaps and hoped upstream steps finished in time. Workflow orchestrators and enterprise schedulers make dependencies a first-class concept, while task-queue frameworks such as Celery express them as chains and chords in code.
| Scheduler category | How dependencies are expressed | Typical limitation |
|---|---|---|
| Cron and task scheduler | Not supported, only time gaps | Fragile when jobs run late |
| Task queue frameworks | Chains, groups and callbacks in code | Limited visibility of the whole graph |
| Workflow orchestrators | DAG definitions with trigger rules | Requires a running orchestration service |
| Enterprise job schedulers | Graphical flows, conditions, calendars | License cost and proprietary formats |
| Distributed job schedulers | Graph plus cluster-wide coordination | Operational complexity |
Designing dependency chains well
Good dependency design keeps graphs explicit, shallow and observable. Replacing time gaps with real dependencies is the single largest reliability improvement for most batch environments, because a late upstream job then delays rather than corrupts downstream results.
- Depend on data readiness, such as a partition or completion marker, rather than on job names when several producers exist.
- Set timeouts on waiting, so a missing file raises an alert instead of blocking a chain indefinitely.
- Keep the critical path visible: the longest chain of dependencies decides when the final output is ready.
- Make jobs idempotent, so a retry or backfill of an upstream job can safely re-trigger downstream work.
During a backfill, dependencies decide whether historical runs proceed in date order. Many orchestrators offer a setting that makes each run depend on the previous run of the same job, which preserves order for cumulative calculations at the cost of parallelism.
Common failures and how to fix them
Most dependency problems show up as jobs that never start or jobs that start too early. The table maps typical symptoms to causes and fixes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Chain stuck in waiting | Upstream failed or file never arrived | Wait timeouts and failure alerts |
| Report built on partial data | Time gap used instead of dependency | Depend on a completion marker |
| Workflow rejected at load | Circular dependency | Split the cycle into separate stages |
| Cleanup never runs | Default all-success trigger rule | Use an all-done trigger rule |
| Duplicate downstream runs | Upstream rerun re-triggers children | Idempotent jobs and run keys |
Explained without jargon
Think of cooking a dinner: you cannot serve the pasta before the water boils, and you cannot boil the water before you fill the pot. Each step depends on an earlier one being finished. A job dependency is the same idea for computers.
Instead of starting every task at a fixed time and hoping the earlier ones are done, the scheduler waits until each step is really complete before starting the next. If one step fails, the steps after it simply do not start, so nobody receives a half-finished result.
A typical use case
A retail chain produces a sales dashboard every morning. Store systems upload transaction files overnight, a load job imports them, a transformation job builds daily aggregates, and a final job refreshes the dashboard. Originally each job ran at a fixed time, one hour apart.
On nights when uploads ran late, the dashboard showed incomplete figures, and managers made decisions on wrong numbers. The data team rebuilds the chain in an orchestrator with real dependencies: the load waits for a completion marker from every store, with a timeout at 05:00 that alerts the on-call engineer. Transformation and refresh depend on the previous job succeeding. The dashboard is now sometimes a little later, but it is never published with partial data.