What happens when writing gigabytes of data to a pipe?

Question

Let's say I have a makefile with a recipe called hour_long_recipe, which as its name suggests takes an hour to run. At random points throughout the recipe it asks yes/no questions. Let's say it asks 10 questions in total.

One possible (and often recommended) way to run it is:

yes | make hour_long_recipe

which answers all questions with y. However, from my understanding, yes keeps outputting to its stdout at up to 10.2 GiB per second regardless of whether make is actually using that data from its stdin.

Even if it was just 10 MiB/s (much slower than any implementation of yes if that reddit thread is to be believed), during the course of an hour it would add up to over 35 GiB, of which just 20 bytes will be read. Where does the data go? It's possible to save it to disk but that's wasteful, and if the disk fills up fast enough it can even cause make to fail.

Presumably the operating system will stop it from getting to that, but how? What is the limit, and what happens when that limit is reached?

Answer 1

tl;dr: at some point, yes will be blocked from writing if the data isn't being read on the other side. It will not be able to continue executing until either that data is read, or it receives a signal, so you typically don't need to worry about yes writing gigabytes and gigabytes of data.

---

The important thing to remember is that a pipe is a FIFO data structure, not simply a pure stream which drops data if not immediately read on the receiver. That is, while it may appear in most cases to be a seamless stream of data from the writing application to the reading application, it does require intermediate storage to perform that, and that intermediate storage is of finite size.*

If we look at the pipe(7) man page, we can read the following about the size of that internal buffer (emphasis added):

In Linux versions before 2.6.11, the capacity of a pipe was the same as the system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes). Since Linux 2.6.35, the default pipe capacity is 16 pages, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.

Assuming you're using a standard x86_64 system, it's very likely that you use 4KiB pages, so the 2^16 upper limit on pipe capacity likely is correct unless either side of the pipeline at some point used fcntl(F_SETPIPE_SZ). Either way, the principle stands: the intermediate storage between two sides of a pipe is finite, and is stored in memory.

In an abstract pipeline a | b, this storage is used in the period between a writing some data, and b actually reading it. Assuming, then, that your make invocation (and any children also connected to this pipe by inheritance) don't actually try to read stdin, or only do so sparingly, the write syscall from yes will eventually simply not wake up yes from sleep when buffer space is exhausted. yes will then wait to be woken up, either when buffer space is available again, or a signal is received.** All of this is handled by the kernel's process scheduler. You can see this in pipe_write(), which is the write() handler for pipes:

static ssize_t
pipe_write(struct kiocb *iocb, struct iov_iter *from)
{
    /* ... */
if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
    wake_next_writer = false;

if (wake_next_writer)
    wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);

/* ... */

}

When the make side eventually terminates, yes will be sent SIGPIPE as a result of writing to a pipe with nothing remaining on the other end. This will then — depending on yes implementation — invoke either its own signal handler or the default kernel signal handler, and it will terminate.***

---

^{* In simple circumstances, where the receiver is processing the data at roughly the same rate that it's being written, this transfer can also be zero-copy with no intermediate buffer by using virtual memory to map and make available the physical page from the writing process available to the receiver. However, the case you're describing will certainly eventually need to use the pipe buffer to store the unread data.}

^{** It's also possible that the writing is done with the O_NONBLOCK flag set on the file descriptor, which enables non-blocking mode. In this case, you'll probably get one incomplete write, and then write will return EAGAIN and the application will need to deal with that itself. It will likely either do that by suspending or running some other code of its choosing to handle the pipe being full. In the case of every modern yes version I can find and most other applications, though, the description above is what happens, since they don't use O_NONBLOCK.}

^{*** An application can do whatever it likes upon receiving SIGPIPE -- it may even theoretically decide not to terminate. However, all common yes use the default SIGPIPE handler, which just terminates without executing any more userspace instructions.}