sar, the System Activity Reporter, is a command-line tool from the sysstat package that collects, stores and reports historical performance statistics on Linux. In system monitoring it answers a question that live tools such as top or htop cannot: what was the server doing at 03:40 last night, when nobody was watching.
Unlike interactive viewers, sar works as a long-running recorder. A scheduled collector samples kernel counters at fixed intervals and writes them to binary daily files, and the sar command later reads those files and prints CPU, memory, paging, disk, network and load figures for any chosen time window.
How sar collects and stores activity data
sar relies on a two-stage design: a data collector writes samples, and a reporter reads them. The collector is sadc (System Activity Data Collector), which reads cumulative counters from procfs, such as /proc/stat, /proc/meminfo, /proc/diskstats and /proc/net/dev, and stores raw values in a compact binary format.
Most distributions run the collector through the wrapper script sa1, triggered by cron or a systemd timer. The default sampling interval is usually every 10 minutes, and a second script, sa2, writes a daily text summary. Because the kernel counters are cumulative, sar computes rates by subtracting consecutive samples and dividing by the elapsed time, so every reported value is an average over the interval.
The following table shows the main components of the sysstat collection chain and the role each plays.
| Component | Role | Typical location |
|---|---|---|
| sadc | Reads kernel counters and writes binary samples | /usr/lib/sysstat/sadc or /usr/lib64/sa/sadc |
| sa1 | Wrapper that runs sadc on a schedule | Called by cron or sysstat-collect.timer |
| sa2 | Writes the daily text report | Called once per day |
| saDD files | Binary daily data, one file per day of month | /var/log/sa or /var/log/sysstat |
| sar | Reads binary files and prints reports | /usr/bin/sar |
Retention is controlled by the HISTORY setting in the sysstat configuration file, commonly 7 to 28 days by default. Files are named after the day of the month, such as sa18, which means older data is overwritten after a month unless retention is extended and longer names are enabled.
Key sar options and the metrics they report
Each sar report flag selects one family of metrics, and the same flags work for live sampling and historical files. Running sar -u 2 5 takes five live CPU samples two seconds apart, while sar -u -f /var/log/sa/sa17 reads yesterday’s recorded data.
| Flag | Metric family | Key fields |
|---|---|---|
| -u | CPU utilization | %user, %system, %iowait, %steal, %idle |
| -P ALL | Per-core CPU | Same fields for each logical CPU |
| -r | Memory usage | kbmemfree, kbavail, %memused, kbcommit |
| -B | Paging activity | pgpgin/s, majflt/s, pgscank/s |
| -W | Swapping | pswpin/s, pswpout/s |
| -d | Block devices | tps, rkB/s, wkB/s, aqu-sz, await, %util |
| -n DEV | Network interfaces | rxkB/s, txkB/s, rxpck/s |
| -q | Run queue and load | runq-sz, plist-sz, ldavg-1 |
| -A | All collected metrics | Complete report |
Time filtering uses -s and -e for start and end times, for example sar -q -s 02:00:00 -e 04:00:00. The companion tool sadf exports the same binary data as CSV, JSON or XML, which makes it suitable for spreadsheets, scripts and graphing tools.
How sar compares with other Linux monitoring tools
sar is the only standard sysstat tool built for retrospective analysis, while its neighbors focus on the present moment or on one subsystem. Choosing the right tool depends on whether the problem is happening now or happened earlier.
| Tool | Time scope | Focus | Best use |
|---|---|---|---|
| sar | Historical and live | Whole system | Trend and incident review |
| top and htop | Live only | Processes | Finding a busy process now |
| vmstat | Live only | Memory, CPU, run queue | Quick system snapshot |
| iostat | Live only | Disks and CPU | Detailed device latency |
| iotop | Live only | Per-process I/O | Finding the disk-heavy process |
| Pressure stall information | Live counters | Resource stall time | Detecting contention directly |
The key limitation in this comparison is that sar reports the system, not processes. It can prove that %iowait spiked at 03:40, but it cannot name the process responsible. Its sibling pidstat, from the same package, adds per-process statistics, yet it does not keep history unless it is scheduled separately.
Reading sar output during an investigation
Effective sar analysis means correlating several metric families across the same time window rather than reading one report alone. A high load average, for instance, may come from CPU saturation, blocked disk I/O or memory reclaim, and only the combined view separates them.
The decision table below links typical symptoms in sar output to their likely cause and the next step.
| Observed pattern | Likely cause | Next step |
|---|---|---|
| High %iowait with high await | Slow or saturated storage | Check sar -d per device and iostat |
| High %steal | Hypervisor contention on a virtual machine | Review host placement or instance size |
| Rising majflt/s and pswpout/s | Memory pressure and swapping | Compare sar -r with kbcommit |
| runq-sz above core count | CPU saturation | Inspect %user and %system per core |
| High %system with low I/O | Kernel overhead or interrupts | Use sar -I or strace on suspects |
A frequent misreading concerns %iowait. It counts idle CPU time during which I/O was outstanding, so it can fall when CPU load rises even though the disk is just as slow. Device latency in await is the more reliable storage indicator.
Limitations and configuration pitfalls
sar is only as useful as its collection settings, and the defaults often hide short events. A 10-minute interval averages away spikes lasting a few seconds, so a 30-second burst at 100 percent CPU may appear as a modest 15 percent.
- Collection is often disabled after installation, for example
ENABLED="false"in/etc/default/sysstaton Debian and Ubuntu. - Disk statistics need the right collector option, such as
-S DISKor-S XDISK, in the sadc configuration on older releases. - Binary files depend on the sysstat version; files from a newer release may not open with an older sar, so use
sadfor the same version for analysis. - Time zones and clock changes can shift reported hours; setting
LC_TIMEorS_TIME_FORMAT=ISOkeeps output unambiguous. - Shortening the interval to one minute is usually safe, because sadc overhead is negligible and daily files grow only to a few megabytes.
The simple explanation
sar works like a flight recorder for a Linux server. Every few minutes it quietly writes down how busy the processor was, how much memory was in use, how hard the disks worked and how much network traffic passed through.
When something goes wrong while nobody is watching, an administrator can replay the recording for that exact hour. Live tools show only the present moment, while sar shows what happened in the past, which is often the only way to explain a night-time slowdown.
How it is used in practice
An operations engineer at an online retailer receives reports that the checkout page was slow between 02:00 and 02:30. By morning the server looks healthy in top, so the engineer runs sar -u -s 01:30:00 -e 03:00:00 on the recorded data and sees %iowait jump from 2 to 38 percent at 02:05.
Running sar -d -p for the same window shows the database volume with await above 120 ms, while sar -r shows normal memory. Cross-checking the cron schedule reveals that a nightly backup job was reading the same volume.
The engineer moves the backup to a snapshot-based method and later lowers the sampling interval to one minute. On the following nights, sar confirms await stays below 5 ms during the backup window, and the checkout complaints stop.