Pipe · shell

A pipe in the shell is the | operator that connects the standard output of one command to the standard input of the next, so data flows directly between processes without temporary files. In the Shell & terminal topic it is the basic building block of a pipeline, the Unix way of combining small, single-purpose tools such as grep, sed, awk and sort into a larger task.

Behind the operator stands a kernel object of the same name: an anonymous, one-way byte channel with a write end and a read end. The term Unix pipe covers both the shell syntax and this underlying mechanism, which every POSIX shell, including Bash and Zsh, uses in the same way.

How the shell builds a pipeline

When the shell reads ls -l | grep conf | wc -l, it calls the pipe() system call once for each |, receiving two file descriptors per pipe. It then forks one child process per command and uses dup2() to rewire the descriptors: the first command’s standard output becomes the write end of the first pipe, and the second command’s standard input becomes its read end, and so on along the chain.

All commands in a pipeline run concurrently, not one after another. The kernel holds data in a pipe buffer; on Linux the default capacity is 64 KiB. A writer that fills the buffer is blocked until the reader consumes data, and a reader on an empty pipe waits until data arrives. This flow control lets a pipeline process gigabytes of data with a small, constant amount of memory.

The end of a pipeline is signaled through descriptors as well. When every write end is closed, the reader receives end of file. When the reader exits early, the next write raises SIGPIPE, which normally terminates the writer quietly; this is why yes | head -n 5 finishes immediately instead of running forever.

Pipe syntax and variants in the shell

The basic operator passes only standard output. Error messages on standard error still go to the terminal unless they are redirected explicitly. Shells provide several related forms for other needs.

A named pipe, or FIFO, behaves like an anonymous pipe but has a path, so unrelated processes can open it. It is useful for connecting a long-running producer to a consumer started separately, while ordinary pipes only connect processes started by the same shell command.

Pipes compared with redirection and temporary files

Pipes and input/output redirection are often confused because both change where data goes. The difference is the endpoint: a pipe connects two processes, while redirection connects a process to a file or device.

Temporary files remain useful when data must be read more than once or inspected after a failure, because a pipe’s contents disappear as soon as they are consumed. For one-pass streaming transformations, a pipe is faster and leaner.

Exit status, subshells and buffering

By default the exit status of a pipeline is that of the last command only. A failing curl piped into jq can therefore go unnoticed in a script. Bash and Zsh offer `set -o pipefail`, which makes the pipeline fail if any command fails, and Bash records every status in the PIPESTATUS array.

In Bash, each part of a pipeline runs in a subshell. A loop such as cmd | while read line; do count=$((count+1)); done updates a variable that disappears when the loop ends. Remedies include process substitution, shopt -s lastpipe in non-interactive Bash, or restructuring the loop; Zsh runs the last command in the current shell by default.

Output buffering surprises many users. Programs that use the C standard library switch from line buffering to block buffering when their output is a pipe rather than a terminal, so results may appear in bursts. Options such as grep --line-buffered or the stdbuf -oL wrapper restore line-by-line output when it matters, for example when following a log with tail -f.

Good practice and common mistakes

• Use pipefail in scripts so that failures early in a pipeline stop the script.
• Avoid unnecessary cat: grep pattern file is simpler than cat file | grep pattern.
• Quote patterns and variables inside pipeline commands to avoid word splitting.
• Handle file names with spaces using null-separated output, such as find -print0 | xargs -0.
• Filter early: placing grep before sort reduces the amount of data sorted.

Pipes carry unstructured bytes, so every command must parse text produced by the previous one. This makes pipelines fragile when output formats change, and it is the main reason tools increasingly offer machine-readable output such as JSON for processing with jq.

Understanding it simply

A pipe works like a conveyor belt between workers in a factory. The first worker finishes a piece and puts it on the belt. The next worker picks it up right away and does the next step.

In the shell, the workers are small programs. One lists files, another searches, another counts. The | sign links them, so simple tools combine into a powerful chain without saving anything in between.

Applied in practice

A system administrator notices heavy load on a web server and wants to know which client addresses send the most requests. The access log contains several million lines, and copying it to another tool would take time.

A single pipeline answers the question in seconds: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 10. Awk extracts the client address, sort groups identical addresses, uniq counts them, the second sort orders them by frequency and head shows the top ten. Because the stages stream data through pipes with a small, fixed buffer, the command uses little memory even on a very large file. The administrator identifies one misbehaving crawler responsible for most of the traffic and blocks it at the firewall.