A concurrency limit is a configured maximum number of jobs or workflow runs that may execute at the same time within a defined scope, such as one worker, one queue, one job type or one customer. When the limit is reached, additional work waits, is rejected or is rescheduled instead of starting immediately.
In scheduling and job processing, the concurrency limit is the main control that turns an unbounded stream of work into a predictable load. It protects CPUs, memory, database connections and external APIs, and it is one of the first settings to examine when a job queue grows or a downstream service starts failing under pressure.
How a concurrency limit controls execution
Conceptually, a concurrency limit works like a counting semaphore. Each running job holds one slot; a new job may start only when a slot is free, and it releases the slot when it finishes, fails or times out. The scheduler or worker checks the counter before dispatching each job, so the number of active jobs never exceeds the configured value.
Limits can be enforced locally, inside one worker process, or globally, across a fleet of machines. Local limits are simple and fast because the counter lives in memory. Global limits need a shared coordination point such as a database row, a Redis key or a lease service, and they must handle workers that crash while holding a slot, usually by attaching an expiry to every slot.
The scope of the limit decides what it protects. The table below shows common scopes and their typical purpose.
| Scope | Example setting | What it protects |
|---|---|---|
| Per worker process | 10 threads or child processes | CPU and memory of one host |
| Per queue | 50 jobs across all workers | Shared database or service |
| Per job type | 2 report exports at once | Expensive resource such as a large query |
| Per tenant or key | 1 sync job per customer | Fairness and data consistency |
| Per workflow | 1 run at a time | Ordering and avoidance of overlapping runs |
Concurrency limit compared with related controls
A concurrency limit counts work in progress, not work over time. That distinguishes it from rate limits, which count starts per second or per minute, and from queue size limits, which cap waiting work. Mature systems usually combine several of these controls.
| Control | What it limits | Typical unit |
|---|---|---|
| Concurrency limit | Simultaneously running jobs | Number of active jobs |
| Rate limit | Job starts or requests over time | Requests per second or minute |
| Queue length limit | Jobs waiting to run | Number of queued jobs |
| Job timeout | Duration of a single job | Seconds or minutes |
| Job priority | Order of dispatch, not quantity | Priority level or weight |
The controls interact. Under steady conditions, Little’s law links them: average concurrency equals the arrival rate multiplied by the average job duration. A queue that receives 20 jobs per second, each lasting 3 seconds, needs about 60 concurrent slots to keep up; a lower limit means the backlog will grow without bound.
Where concurrency limits are configured
Almost every scheduler and job framework exposes a concurrency setting, but the names differ. Knowing where the knob lives prevents teams from tuning the wrong layer, for example raising worker threads while a database pool still caps real throughput.
| System type | Typical setting | Effect |
|---|---|---|
| Task queue workers | Worker concurrency, prefork pool size | Jobs processed in parallel per worker |
| Thread-based job processors | Number of threads per process | Parallel jobs sharing one memory space |
| Workflow automation platforms | Concurrent runs per workflow | Overlapping executions of one flow |
| Container orchestrators | Parallelism of batch jobs | Pods running at once |
| CI and build systems | Concurrency groups | Builds for the same branch or environment |
The useful value depends on the bottleneck resource. CPU-bound jobs rarely benefit from more slots than CPU cores, while I/O-bound jobs that mostly wait on networks can run tens or hundreds of slots per process. The limit should also stay below the size of every shared pool the jobs use, such as database connections.
Choosing and tuning the limit
A good concurrency limit is found by measurement, not guessed. Teams start from the capacity of the weakest dependency, then raise the limit step by step while watching latency, error rates and queue depth.
| Observation | Likely meaning | Adjustment |
|---|---|---|
| Queue grows, CPU idle | Limit too low for I/O-bound work | Raise concurrency gradually |
| Latency rises, errors from database | Downstream saturation | Lower limit or add a per-resource limit |
| Jobs for one customer starve others | No fairness scope | Add a per-tenant limit |
| Duplicate runs corrupt data | Overlapping executions | Set a limit of 1 per key or workflow |
| Slots stay occupied after crashes | Leaked slots | Use leases with expiry and heartbeats |
Some systems apply adaptive concurrency, adjusting the limit automatically from observed latency, similar to congestion control in networking. Adaptive limits react well to changing conditions but are harder to reason about, so many teams keep a fixed upper bound even when adaptation is enabled.
Common mistakes
Most problems with concurrency limits come from mismatched layers or missing failure handling rather than from the chosen number itself.
- Multiplying limits unintentionally: 10 workers with a concurrency of 20 each produce 200 simultaneous jobs, not 20.
- Retry storms: failed jobs retried immediately consume slots again and keep a struggling service overloaded; use backoff.
- Limit equals pool size: if jobs need two connections each, the connection pool must be larger than the job limit.
- Ignoring long jobs: a few hour-long jobs can occupy every slot and block short urgent work; separate queues help.
- Treating the limit as a rate limit: short jobs can still exceed an external API quota even under a low concurrency limit.
Explained without jargon
A concurrency limit is a rule about how many things may happen at once. Think of a car wash with four bays. Only four cars can be washed at a time. The others wait in line until a bay is free.
Computers use the same idea for background work. Without a limit, a system might try to run thousands of tasks together and crash. With a sensible limit, work is slower to start but steady and safe, and nothing gets overwhelmed.
A typical use case
An online shop sends order data to a partner’s shipping API, which allows no more than 10 simultaneous connections per client. The shop’s job queue runs three worker machines, each with a default concurrency of 8, so up to 24 shipping jobs could call the API at the same moment.
During a holiday sale, the partner starts rejecting requests, failed jobs retry immediately and the queue grows into the thousands. The engineering team adds a global concurrency limit of 8 for the shipping job type, enforced through a shared counter with expiring leases, and switches retries to exponential backoff.
The next peak runs without rejected requests. Shipping labels are created a little later than order confirmations, but every order is processed exactly once, and the other job types keep their full worker capacity because the limit applies only to shipping.