sar

On this page 7

sar, the System Activity Reporter, is a command-line tool from the sysstat package that collects, stores and reports historical performance statistics on Linux. In system monitoring it answers a question that live tools such as top or htop cannot: what was the server doing at 03:40 last night, when nobody was watching.

Unlike interactive viewers, sar works as a long-running recorder. A scheduled collector samples kernel counters at fixed intervals and writes them to binary daily files, and the sar command later reads those files and prints CPU, memory, paging, disk, network and load figures for any chosen time window.

How sar collects and stores activity data

sar relies on a two-stage design: a data collector writes samples, and a reporter reads them. The collector is sadc (System Activity Data Collector), which reads cumulative counters from procfs, such as /proc/stat, /proc/meminfo, /proc/diskstats and /proc/net/dev, and stores raw values in a compact binary format.

Most distributions run the collector through the wrapper script sa1, triggered by cron or a systemd timer. The default sampling interval is usually every 10 minutes, and a second script, sa2, writes a daily text summary. Because the kernel counters are cumulative, sar computes rates by subtracting consecutive samples and dividing by the elapsed time, so every reported value is an average over the interval.

The following table shows the main components of the sysstat collection chain and the role each plays.

ComponentRoleTypical location
sadcReads kernel counters and writes binary samples/usr/lib/sysstat/sadc or /usr/lib64/sa/sadc
sa1Wrapper that runs sadc on a scheduleCalled by cron or sysstat-collect.timer
sa2Writes the daily text reportCalled once per day
saDD filesBinary daily data, one file per day of month/var/log/sa or /var/log/sysstat
sarReads binary files and prints reports/usr/bin/sar
Components of the sar data pipeline

Retention is controlled by the HISTORY setting in the sysstat configuration file, commonly 7 to 28 days by default. Files are named after the day of the month, such as sa18, which means older data is overwritten after a month unless retention is extended and longer names are enabled.

Key sar options and the metrics they report

Each sar report flag selects one family of metrics, and the same flags work for live sampling and historical files. Running sar -u 2 5 takes five live CPU samples two seconds apart, while sar -u -f /var/log/sa/sa17 reads yesterday’s recorded data.

FlagMetric familyKey fields
-uCPU utilization%user, %system, %iowait, %steal, %idle
-P ALLPer-core CPUSame fields for each logical CPU
-rMemory usagekbmemfree, kbavail, %memused, kbcommit
-BPaging activitypgpgin/s, majflt/s, pgscank/s
-WSwappingpswpin/s, pswpout/s
-dBlock devicestps, rkB/s, wkB/s, aqu-sz, await, %util
-n DEVNetwork interfacesrxkB/s, txkB/s, rxpck/s
-qRun queue and loadrunq-sz, plist-sz, ldavg-1
-AAll collected metricsComplete report
Common sar flags and what they show

Time filtering uses -s and -e for start and end times, for example sar -q -s 02:00:00 -e 04:00:00. The companion tool sadf exports the same binary data as CSV, JSON or XML, which makes it suitable for spreadsheets, scripts and graphing tools.

How sar compares with other Linux monitoring tools

sar is the only standard sysstat tool built for retrospective analysis, while its neighbors focus on the present moment or on one subsystem. Choosing the right tool depends on whether the problem is happening now or happened earlier.

ToolTime scopeFocusBest use
sarHistorical and liveWhole systemTrend and incident review
top and htopLive onlyProcessesFinding a busy process now
vmstatLive onlyMemory, CPU, run queueQuick system snapshot
iostatLive onlyDisks and CPUDetailed device latency
iotopLive onlyPer-process I/OFinding the disk-heavy process
Pressure stall informationLive countersResource stall timeDetecting contention directly
sar versus related monitoring tools

The key limitation in this comparison is that sar reports the system, not processes. It can prove that %iowait spiked at 03:40, but it cannot name the process responsible. Its sibling pidstat, from the same package, adds per-process statistics, yet it does not keep history unless it is scheduled separately.

Reading sar output during an investigation

Effective sar analysis means correlating several metric families across the same time window rather than reading one report alone. A high load average, for instance, may come from CPU saturation, blocked disk I/O or memory reclaim, and only the combined view separates them.

The decision table below links typical symptoms in sar output to their likely cause and the next step.

Observed patternLikely causeNext step
High %iowait with high awaitSlow or saturated storageCheck sar -d per device and iostat
High %stealHypervisor contention on a virtual machineReview host placement or instance size
Rising majflt/s and pswpout/sMemory pressure and swappingCompare sar -r with kbcommit
runq-sz above core countCPU saturationInspect %user and %system per core
High %system with low I/OKernel overhead or interruptsUse sar -I or strace on suspects
Interpreting typical sar patterns

A frequent misreading concerns %iowait. It counts idle CPU time during which I/O was outstanding, so it can fall when CPU load rises even though the disk is just as slow. Device latency in await is the more reliable storage indicator.

Limitations and configuration pitfalls

sar is only as useful as its collection settings, and the defaults often hide short events. A 10-minute interval averages away spikes lasting a few seconds, so a 30-second burst at 100 percent CPU may appear as a modest 15 percent.

  • Collection is often disabled after installation, for example ENABLED="false" in /etc/default/sysstat on Debian and Ubuntu.
  • Disk statistics need the right collector option, such as -S DISK or -S XDISK, in the sadc configuration on older releases.
  • Binary files depend on the sysstat version; files from a newer release may not open with an older sar, so use sadf or the same version for analysis.
  • Time zones and clock changes can shift reported hours; setting LC_TIME or S_TIME_FORMAT=ISO keeps output unambiguous.
  • Shortening the interval to one minute is usually safe, because sadc overhead is negligible and daily files grow only to a few megabytes.

The simple explanation

sar works like a flight recorder for a Linux server. Every few minutes it quietly writes down how busy the processor was, how much memory was in use, how hard the disks worked and how much network traffic passed through.

When something goes wrong while nobody is watching, an administrator can replay the recording for that exact hour. Live tools show only the present moment, while sar shows what happened in the past, which is often the only way to explain a night-time slowdown.

How it is used in practice

An operations engineer at an online retailer receives reports that the checkout page was slow between 02:00 and 02:30. By morning the server looks healthy in top, so the engineer runs sar -u -s 01:30:00 -e 03:00:00 on the recorded data and sees %iowait jump from 2 to 38 percent at 02:05.

Running sar -d -p for the same window shows the database volume with await above 120 ms, while sar -r shows normal memory. Cross-checking the cron schedule reveals that a nightly backup job was reading the same volume.

The engineer moves the backup to a snapshot-based method and later lowers the sampling interval to one minute. On the following nights, sar confirms await stays below 5 ms during the backup window, and the checkout complaints stop.