4

When a command is piped to another command in what way or in what format does the piped output exist/get sent? Is it a temporary file? Is it a string? And how does the command that receives the piped output decode/read that output?

Example:

echo "Someone string" | ./program | tail

Does the program "program" receive the output of echo as a file? How would the program read in that input?

AymenTM
  • 277
  • The part about temporary files depends from whether one is asking about very old versions of Unix. (-: https://unix.stackexchange.com/a/450900/5132 – JdeBP Sep 23 '18 at 11:32
  • Related: https://unix.stackexchange.com/questions/470767/how-does-a-program-know-what-to-read-in-a-pipeline – Kusalananda Sep 23 '18 at 11:55

3 Answers3

5

A pipe is a form of redirection (transfer of standard output to some other destination) that is used in Linux and other Unix-like operating systems to send the output of one command/program/process to another command/program/process for further processing. The Unix/Linux systems allow stdout of a command to be connected to stdin of another command. You can make it do so by using the pipe character ‘|’.

Pipe is used to combine two or more command and in this the output of one command act as input to another command and this command output may act as input to next command and so on. It can also be visualized as a temporary connection between two or more commands/ programs/ processes. The command line programs that do the further processing are referred to as filters.

This direct connection between commands/ programs/ processes allows them to operate simultaneously and permits data to be transferred between them continuously rather than having to pass it through temporary text files or through the display screen. Pipes are unidirectional i.e data flow from left to right through the pipeline.

Syntax :

command_1 | command_2 | command_3 | .... | command_N

Example : Listing all files and directories and give it as input to more command.

$ ls -l | more

The more command takes output of ls -l as its input. The net effect of this command is that the output of ls -l is displayed one screen at a time. The pipes act as a container which take output of ls -l and giving it to more as input. This command does not use a disk to connect standard output of ls -l to standard input of more because pipe is implemented in the main memory. In terms of I/O redirection operators, the above command is equivalent to the following command sequence.

$ ls -l -> temp
more -> temp (or more temp)
[contents of temp]
rm temp

More 1, 2, 3

4

./program receives the output of echo as a file, namely the standard stream file /dev/stdin. Demo:

echo foo | wc -c
echo foo | wc -c /dev/stdin

Outputs:

4
4 /dev/stdin

Note: a stream file is somewhat different from a file stored on a disk. We can access data randomly in a regular file, but not in a streaming file. It's like the difference between a CD player and radio playing a song. On the CD player we can rewind, fast forward, seek, and skip songs. The radio just outputs (or streams) whatever the broadcaster is sending right now.

Where is the stream stored? The OS stores it in a temporary data buffer, the size of which varies between OSes. See How big is the pipe buffer?

On a lower level, the OS uses file descriptors, not filenames. File descriptors are just a stack of numbers, for which each process identifier has its own stack. Linux also allows accessing the /dev/stdin stream, (which is always file descriptor "0"), that way:

echo foo | wc -c /proc/self/fd/0

Output:

4 /proc/self/fd/0

One more Linux abstraction can be seen with the realpathutil:

realpath /dev/stdin /proc/self/fd/0

Output (on the current terminal on my system):

/dev/pts/2
/dev/pts/2

The letters pts stand for Pseudo Terminals. That last 2 number will always vary between terminals, so it's not a reliable name, but those numbers can be used for tricks like: How to send output from one terminal to another without making any new pipe or file.

agc
  • 7,223
3

It goes as a bit stream. That's how it's sent and how it's received. The interfaces are file descriptors (or handles). A simple file is also handled that way.

See:

  • So (correct me if I'm wrong) if I wanted a program (that I wrote) to read in the output of a command/program/process (let's say that simply prints to standard out a shape, say a square); "./print_square | ./program" ; From my program, I would need to read the piped output from standard-in as a string of bytes (a bit string) ? And from that string, I'd need to parse and figure out how to understand that string of bytes to be a square shape ? – AymenTM Sep 23 '18 at 10:40
  • 1
    I like your succinct answer, but would it be more accurate to call it a byte stream, since that's the smallest unit that can be written ? – X Tian Sep 23 '18 at 10:41
  • @Lion Yes, the program you write should read from standard input, you can develop it by typing at the terminal or redirecting stdin from a file with a "canned" input. Then use it in a pipe stream later to read the output of the previous command. – X Tian Sep 23 '18 at 10:47
  • @XTian I'll keep it bit, as I mean the bit level, as opposed to higher level communication. And this could be in octets of bits or other sets, depending on architecture. –  Sep 23 '18 at 11:04
  • @Lion Yes, it's always about parsing. There are standards like ASCII or UTF-8 for text, but there are always single bits underneath and that has to be remembered. –  Sep 23 '18 at 11:06