cgroups

On this page 7

cgroups, short for control groups, are a Linux kernel feature that organizes processes into a hierarchy and lets the kernel limit, prioritize and account for the resources each group uses, including CPU time, memory, block I/O and the number of processes. In the topic of Linux processes, cgroups answer a question that per-process tools cannot: how much can this whole set of related processes consume together.

Every process on a modern Linux system belongs to exactly one cgroup. systemd places each service, user session and container in its own group, so cgroups quietly shape resource behavior on almost every current distribution, even when no administrator configured them directly.

How cgroups control groups of processes

cgroups are exposed through a virtual file system, normally mounted at /sys/fs/cgroup. Each directory is a cgroup, and creating a subdirectory creates a child group. Writing a process ID into the file cgroup.procs moves that process into the group, and child processes inherit their parent’s cgroup when they are forked.

Resource rules are applied by controllers, which are kernel subsystems that read the interface files of each group. A controller enforces limits on the whole group, so ten worker processes in one cgroup share a single memory limit rather than receiving ten separate limits. Limits flow down the tree: a child can never use more than its parent allows.

The table below lists the controllers administrators meet most often and the main interface files they provide in the cgroup v2 unified hierarchy.

ControllerResourceKey interface filesEffect
cpuProcessor timecpu.weight, cpu.maxRelative share or hard quota
memoryRAM and swapmemory.max, memory.high, memory.lowLimit, throttle or protect memory
ioBlock device I/Oio.weight, io.maxBandwidth or IOPS control per device
pidsNumber of taskspids.maxStops fork bombs
cpusetCPUs and NUMA nodescpuset.cpus, cpuset.memsPins a group to specific cores
Main cgroup v2 controllers

Cgroup v1 and cgroup v2

Linux has two cgroup designs. Cgroup v1 allowed a separate hierarchy for each controller, which made consistent policy difficult. Cgroup v2, stable since kernel 4.5, uses one unified hierarchy in which all controllers see the same tree.

AspectCgroup v1Cgroup v2
HierarchyOne tree per controllerSingle unified tree
Process placementProcesses in any nodeProcesses only in leaf groups
Memory handlingHard limit with abrupt behaviormemory.high throttling before memory.max
Pressure dataNot availablePressure stall information per group
DelegationError-proneSafe delegation to unprivileged users
Differences between cgroup v1 and v2

Most current distributions boot with cgroup v2 only. Older container runtimes and monitoring agents that expect v1 paths may fail on these systems, so checking for the file /sys/fs/cgroup/cgroup.controllers is a quick way to confirm the unified hierarchy is in use.

cgroups sit next to several older process controls in Linux. The difference is scope: nice values, CPU affinity and rlimits act on single processes, whereas cgroups act on a whole group, including future children.

MechanismScopeWhat it controlsHard limit
cgroupsGroup of processesCPU, memory, I/O, task countYes
Nice and process prioritySingle processScheduler preferenceNo
CPU affinitySingle process or threadAllowed CPUsPlacement only
Rlimits (ulimit)Single processOpen files, stack, core sizeYes, per process
Linux namespaceGroup of processesWhat processes can seeNot a resource limit
cgroups and neighboring mechanisms

The pairing of namespaces and cgroups is the foundation of containers: namespaces decide what a container can see, while cgroups decide how much it can consume. Neither alone provides isolation that is comparable to a virtual machine.

Using cgroups through systemd

On systemd distributions, administrators rarely edit /sys/fs/cgroup by hand. systemd owns the cgroup tree and maps it into slices, services and scopes, so resource limits are set as unit properties.

  • Slices such as system.slice and user.slice group units and split resources between them.
  • Services receive their own cgroup automatically, which also lets systemd kill every process of a service reliably on stop.
  • Scopes wrap externally started processes, such as user sessions or containers.
  • systemctl set-property nginx.service MemoryMax=2G CPUQuota=150% applies limits at runtime and persists them.
  • systemd-cgtop shows live resource usage per cgroup, and systemd-cgls prints the tree.

Note that CPUQuota=150% means one and a half CPUs of time per period, because quotas are expressed against a single CPU. The underlying file cpu.max stores the same idea as a quota and a period in microseconds, such as 150000 100000.

Limitations and common mistakes

cgroups are powerful, but misconfigured limits cause failures that look unrelated to resource control. The table maps typical symptoms to their cause.

SymptomLikely causeRemedy
Process killed with OOM in a groupmemory.max set too lowRaise the limit or add memory.high
Latency spikes under loadCPU quota throttlingIncrease quota or use cpu.weight
Cannot fork new processespids.max reachedRaise pids.max or fix the leak
Limits seem ignoredController not enabled in parentEnable it in cgroup.subtree_control
Symptoms of cgroup misconfiguration

Another common misunderstanding is that cpu.weight guarantees CPU time. Weights only matter under contention; on an idle system a low-weight group can still use every core. Only cpu.max sets a hard ceiling.

In everyday language

Imagine a shared office kitchen where each team gets its own shelf in the refrigerator and a fixed number of microwave minutes. No single person decides how much the team uses; the whole team shares one allowance.

cgroups do the same thing for programs on a Linux computer. They put related programs into a group and give that group a budget for processor time, memory and disk activity. If one group gets greedy, it hits its own limit instead of slowing down everything else on the machine.

Seen in the real world

A company runs a database and a nightly reporting job on the same 16-core server with 64 GB of memory. Every night the reporting job consumed so much memory that the kernel began reclaiming the database cache, and query times rose sharply.

The administrator moved the reporting job into its own systemd service and set MemoryHigh=12G, MemoryMax=16G and CPUQuota=400%, which limits it to four cores of processor time. The database service received MemoryLow=32G, which protects its cache from reclaim.

After the change, the report takes somewhat longer to finish, but database query times stay stable throughout the night, and systemd-cgtop makes the effect easy to verify.