Pipes and filters

Pipes and filters is a software design pattern invented by Douglas McIlroy.

In the context of Unix operating systems, a pipe signifies that the output of one program feeds directly as input to another program. The Unix shell uses the pipe character (|) to join programs together. A sequence of commands joined together by pipes is known as a pipeline. For creating this mechanism, all Unix tools have access to three distinct files:

• stdin the standard input file
• stdout the standard output file
• stderr the standard error file

By joining one tools stdout to another tools stdin, a pipeline is formed. Errors are sent to a side track and accumulated.

Often filter programs form the constituent programs in a pipeline.

An example of a pipeline:

 cat * | grep "alice" | grep -v "wonderland" | wc -l

will print out the number of lines in all files is a directory which contain the text "alice" without the text "wonderland".

The pipeline has four parts:

• cat * concatenates the text of all files to its standard output
• grep "alice" treats its input as a set of lines, and produces copies of only those lines which contain the word "alice"
• grep -v "wonderland" produces copies of only those remaining lines which do not contain the word "wonderland"
• wc -l counts the lines on its standard input, and prints out a line count.

Pipes and filters can be viewed as a form of functional programming, using byte streams as data objects.