How big is the pipe buffer?

Question

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote:

Another reason to avoid | true is that if the command produced enough output to fill up the pipe buffer, it would block waiting for true to read it.

Do we have some way of finding out what the size of the pipe buffer is?

score 200 · Accepted Answer · edited May 16 '23 at 18:07

The capacity of a pipe buffer varies across systems (and can even vary on the same system). I am not sure if there is a quick, easy, and cross-platform way to just lookup the capacity of a pipe.

Mac OS X, for example, uses a capacity of 16384 bytes by default, but can switch to 65336 byte capacities if large write are made to the pipe, or will switch to a capacity of a single system page if too much kernel memory is already being used by pipe buffers (see xnu/bsd/sys/pipe.h, and xnu/bsd/kern/sys_pipe.c; since these are from FreeBSD, the same behavior may happen there, too).

One Linux pipe(7) man page says that pipe capacity is 65536 bytes since Linux 2.6.11 and a single system page prior to that (e.g. 4096 bytes on (32-bit) x86 systems). The code (include/linux/pipe_fs_i.h, and fs/pipe.c) seems to use 16 system pages (i.e. 64 KiB if a system page is 4 KiB), but the buffer for each pipe can be adjusted via a fcntl on the pipe (up to a maximum capacity which defaults to 1048576 bytes, but can be changed via /proc/sys/fs/pipe-max-size)).

Here is a little bash/perl combination that I used to test the pipe capacity on my system:

#!/bin/bash
test $# -ge 1 || { echo "usage: $0 write-size [wait-time]"; exit 1; }
test $# -ge 2 || set -- "$@" 1
bytes_written=$(
{
    exec 3>&1
    {
        perl -e '
            $size = $ARGV[0];
            $block = q(a) x $size;
            $num_written = 0;
            sub report { print STDERR $num_written * $size, qq(\n); }
            report; while (defined syswrite STDOUT, $block) {
                $num_written++; report;
            }
        ' "$1" 2>&3
    } | (sleep "$2"; exec 0<&-);
} | tail -1
)
printf "write size: %10d; bytes successfully before error: %d\n" \
    "$1" "$bytes_written"

Here is what I found running it with various write sizes on a Mac OS X 10.6.7 system (note the change for writes larger than 16KiB):

% /bin/bash -c 'for p in {0..18}; do /tmp/ts.sh $((2 ** $p)) 0.5; done'
write size:          1; bytes successfully before error: 16384
write size:          2; bytes successfully before error: 16384
write size:          4; bytes successfully before error: 16384
write size:          8; bytes successfully before error: 16384
write size:         16; bytes successfully before error: 16384
write size:         32; bytes successfully before error: 16384
write size:         64; bytes successfully before error: 16384
write size:        128; bytes successfully before error: 16384
write size:        256; bytes successfully before error: 16384
write size:        512; bytes successfully before error: 16384
write size:       1024; bytes successfully before error: 16384
write size:       2048; bytes successfully before error: 16384
write size:       4096; bytes successfully before error: 16384
write size:       8192; bytes successfully before error: 16384
write size:      16384; bytes successfully before error: 16384
write size:      32768; bytes successfully before error: 65536
write size:      65536; bytes successfully before error: 65536
write size:     131072; bytes successfully before error: 0
write size:     262144; bytes successfully before error: 0

The same script on Linux 3.19:

/bin/bash -c 'for p in {0..18}; do /tmp/ts.sh $((2 ** $p)) 0.5; done'
write size:          1; bytes successfully before error: 65536
write size:          2; bytes successfully before error: 65536
write size:          4; bytes successfully before error: 65536
write size:          8; bytes successfully before error: 65536
write size:         16; bytes successfully before error: 65536
write size:         32; bytes successfully before error: 65536
write size:         64; bytes successfully before error: 65536
write size:        128; bytes successfully before error: 65536
write size:        256; bytes successfully before error: 65536
write size:        512; bytes successfully before error: 65536
write size:       1024; bytes successfully before error: 65536
write size:       2048; bytes successfully before error: 65536
write size:       4096; bytes successfully before error: 65536
write size:       8192; bytes successfully before error: 65536
write size:      16384; bytes successfully before error: 65536
write size:      32768; bytes successfully before error: 65536
write size:      65536; bytes successfully before error: 65536
write size:     131072; bytes successfully before error: 0
write size:     262144; bytes successfully before error: 0

Note: The PIPE_BUF value defined in the C header files (and the pathconf value for _PC_PIPE_BUF), does not specify the capacity of pipes, but the maximum number of bytes that can be written atomically (see POSIX write(2)).

Quote from include/linux/pipe_fs_i.h:

/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual
   memory allocation, whereas PIPE_BUF makes atomicity guarantees.  */

Great answer. Especially for the link to POSIX write(2), which says: The effective size of a pipe or FIFO (the maximum amount that can be written in one operation without blocking) may vary dynamically, depending on the implementation, so it is not possible to specify a fixed value for it. — Mikel, Apr 25 '11 at 05:59
Thanks for mentioning fcntl() on Linux; I had spent a while looking for userspace buffering programs because I thought the built-in pipes didn't have a large enough buffer. Now I see that they do, if I have CAP_SYS_RESOURCE or root is willing to expand the maximum pipe size. As what I want will only be run on a specific Linux computer (mine), this shouldn't be a problem. — Daniel H, Aug 04 '13 at 18:43
Can you please explain the basic idea of your script? I am staring at it and I cannot figure out how it works? Especially what is the purpose using curly brackets here VAR=$({})? Thank you. — Wakan Tanka, Mar 12 '15 at 00:24
@WakanTanka: It is a bit much to describe in a comment, but that particular construct is a parameter assignment (var=…) of the output of a command substitution ($(…)) that includes grouped commands ({…}, and (…)). It also uses several (less common) redirections (i.e. 0<&- and 3>&1). — Chris Johnsen, Mar 12 '15 at 00:51
@ChrisJohnsen thank you for explaining syntax, can you please simply describe the main idea? — Wakan Tanka, Mar 12 '15 at 00:55
@WakanTanka: The Perl program writes to its stdout (a shell-created pipe—the one that is being tested) in blocks of a given size and reports to its stderr a running total of how much it has written (until it gets an error—usually because the pipe’s buffer is full or possibly because the reading end of the pipe has been closed after a short time (exec 0<&-)). The final report is collected (tail -1) and printed along with the write size. — Chris Johnsen, Mar 12 '15 at 01:27
Is it possible for a process to write to the named pipe (that has no readers) a piece of data that is smaller than the pipe buffer size, and just exit straight away? How does the blocking semantics of a named pipe determine when when a write has finished and allow the reader to read? What if a process write 1 kb, waited 10 seconds, wrote another 1kb without ever exiting? Does the reader get 1kb, then 1kb, or does it block until getting 2kb? — CMCDragonkai, May 01 '15 at 03:38
@CMCDragonkai: That is probably too much to address in a comment. Perhaps ask it as a new question? You might find some answers in the POSIX spec: FIFO definition, open(2), read(2), and write(2) (search for FIFO in these later ones). — Chris Johnsen, May 01 '15 at 06:46
Great answer. I think there's a minor typo and that you probably meant 65536, not 65336 for the big pipe size for mac. In the header file it's defined as 64*1024, which is 65536. 65536 is also what your empirical tests showed. — Max Heiber, Feb 03 '22 at 19:54

Asain Kujovic · Answer 2 · 2017-10-27T12:53:09.990

48

this shell-line can show pipe buffer size too:

M=0; while true; do dd if=/dev/zero bs=1k count=1 2>/dev/null; \
       M=$(($M+1)); echo -en "\r$M KB" 1>&2; done | sleep 999

(sending 1k chunks to blocked pipe until buffer full) ...some test outputs:

64K (intel-debian), 32K (aix-ppc), 64K (jslinux bellard.org)      ...Ctrl+C.

shortest bash-one-liner using printf:

M=0; while printf A; do >&2 printf "\r$((++M)) B"; done | sleep 999

edited Oct 27 '17 at 12:53

answered Apr 17 '13 at 22:52

Asain Kujovic

1,731

16

Very nice! (dd if=/dev/zero bs=1 | sleep 999) & then wait a second and killall -SIGUSR1 dd gives 65536 bytes (66 kB) copied, 5.4987 s, 11.9 kB/s - same as your solution, but at 1 byte resolution ;) – frostschutz Apr 17 '13 at 23:21
3

For the record, on Solaris 10/11 SPARC/x86 the dd command blocks at 16 KiB. On Fedora 23/25 x86-64, it blocks at 64 KiB. – maxschlepzig Jan 01 '17 at 12:08
2

@frostschutz: That's a nice simplification. Pragmatically, you could just run dd if=/dev/zero bs=1 | sleep 999 in the foreground, wait a second, then press ^C. If you wanted a one-liner on Linux and BSD/macOS (more robust than using killall): dd if=/dev/zero bs=1 | sleep 999 & sleep 1 && pkill -INT -P $$ -x dd – mklement0 Jan 23 '17 at 18:39
For unwary users, this seems like an off-label use of sleep for creating a blocked pipe for a set amount of time (999s here). Before killing dd, dd progress can be monitored using USR1 signal (https://askubuntu.com/a/215521/134766). This will show the transfer stats frozen at pipe buffer size. – akhan Apr 08 '22 at 18:24

score 8 · Answer 3 · answered Oct 28 '14 at 14:16

8

Here are some further alternatives to explore the actual pipe buffer capacity using shell commands only:

# get pipe buffer size using Bash
yes produce_this_string_as_output | tee >(sleep 1) | wc -c

# portable version
( (sleep 1; exec yes produce_this_string_as_output) & echo $! ) | 
     (pid=$(head -1); sleep 2; kill "$pid"; wc -c </dev/stdin)

# get buffer size of named pipe
sh -c '
  rm -f fifo
  mkfifo fifo
  yes produce_this_string_as_output | tee fifo | wc -c &
  exec 3<&- 3<fifo
  sleep 1
  exec 3<&-
  rm -f fifo
'

# Mac OS X
#getconf PIPE_BUF /
#open -e /usr/include/limits.h /usr/include/sys/pipe.h
# PIPE_SIZE
# BIG_PIPE_SIZE
# SMALL_PIPE_SIZE
# PIPE_MINDIRECT

answered Oct 28 '14 at 14:16

chan

81

On Solaris 10, getconf PIPE_BUF / prints 5120 which matches the ulimit -a | grep pipe output but doesn't match the 16 KiB after which dd .. | sleep ... blocks. – maxschlepzig Jan 01 '17 at 12:16
On Fedora 25, your first yes method prints 73728 instead of the 64 KiB determined with dd if=/dev/zero bs=4096 status=none | pv -bn | sleep 1 – maxschlepzig Jan 01 '17 at 12:30
1

I too get 73728 from yes | tee >(sleep 1) | wc -c instead of 65536. I think this is because we measure the buffer of >( ) instead of | even though both should use the same mechanism under the hood... strange. With yes | tee >(wc -c >&2) | sleep 1 I get the correct number. – Socowi Aug 06 '21 at 20:08

score 7 · Answer 4 · edited Sep 11 '14 at 02:31

This is a quick and dirty hack on Ubuntu 12.04, YMMV

cat >pipesize.c

#include <unistd.h>
#include <errno.h>
#include </usr/include/linux/fcntl.h>
#include <stdio.h>

void main( int argc, char *argv[] ){
  int fd ;
  long pipesize ;

  if( argc>1 ){
  // if command line arg, associate a file descriptor with it
    fprintf( stderr, "sizing %s ... ", argv[1] );
    fd = open( argv[1], O_RDONLY|O_NONBLOCK );
  }else{
  // else use STDIN as the file descriptor
    fprintf( stderr, "sizing STDIN ... " );
    fd = 0 ;
  }

  fprintf( stderr, "%ld bytes\n", (long)fcntl( fd, F_GETPIPE_SZ ));
  if( errno )fprintf( stderr, "Uh oh, errno is %d\n", errno );
  if( fd )close( fd );
}

gcc -o pipesize pipesize.c

mkfifo /tmp/foo

./pipesize /tmp/foo

>sizing /tmp/foo ... 65536 bytes

date | ./pipesize

>sizing STDIN ... 65536 bytes

score 6 · Answer 5 · answered Mar 22 '17 at 13:52

If you need the value in Python>=3.3, here's a simple method (assuming you can run call out to dd):

from subprocess import Popen, PIPE, TimeoutExpired
p = Popen(["dd", "if=/dev/zero", "bs=1"], stdin=PIPE, stdout=PIPE)
try: 
    p.wait(timeout=1)
except TimeoutExpired: 
    p.kill()
    print(len(p.stdout.read()))

score 1 · Answer 6 · answered Sep 23 '23 at 16:49

For finding mklement0's solution more easily, I turned their comment into this answer:

Pragmatically, you could just run dd if=/dev/zero bs=1 | sleep 999 in the foreground, wait a second, then press ^C [= ctrl+C].

If you wanted a one-liner on Linux and BSD/macOS (more robust than using killall):
dd if=/dev/zero bs=1 | sleep 999 & sleep 1 && pkill -INT -P $$ -x dd

With GNU timeout you can simplify this to ...

timeout -s INT 3 dd if=/dev/zero bs=1 | sleep 99

Either way, the output will be something like below.

65537+0 records in
65536+0 records out
65536 bytes (66 kB, 64 KiB) copied, 2.99738 s, 21.9 kB/s

In above example, my buffer was 65536 bytes.
dd couldn't write the 65537th byte, because the buffer was full.

score -1 · Answer 7 · edited Mar 13 '21 at 11:21

The accepted answer adapted to zsh:

#+begin_src bsh.dash :results verbatim :exports both :wrap example
test-buffer() {
    test $# -ge 1 || { echo "usage: $0 write-size [wait-time]"; return 1; }
    test $# -ge 2 || set -- "$@" 1
    bytes_written=$(
        {
            exec 3>&1
            {
                perl -e '
            $size = $ARGV[0];
            $block = q(a) x $size;
            $num_written = 0;
            sub report { print STDERR $num_written * $size, qq(\n); }
            report; while (defined syswrite STDOUT, $block) {
                $num_written++; report;
            }
        ' "$1" 2>&3
            } | (sleep "$2"; exec 0<&-);
        } | tail -1
                 )
    printf "write size: %10d; bytes successfully before error: %d\n" \
        "$1" "$bytes_written"
}
uname
for i in {1..18} ; do
    test-buffer $((2**i)) 1
done
#+end_src

Results:

#+begin_example
Darwin
write size:          2; bytes successfully before error: 65536
write size:          4; bytes successfully before error: 65536
write size:          8; bytes successfully before error: 65536
write size:         16; bytes successfully before error: 65536
write size:         32; bytes successfully before error: 65536
write size:         64; bytes successfully before error: 65536
write size:        128; bytes successfully before error: 65536
write size:        256; bytes successfully before error: 65536
write size:        512; bytes successfully before error: 65536
write size:       1024; bytes successfully before error: 65536
write size:       2048; bytes successfully before error: 65536
write size:       4096; bytes successfully before error: 65536
write size:       8192; bytes successfully before error: 65536
write size:      16384; bytes successfully before error: 65536
write size:      32768; bytes successfully before error: 65536
write size:      65536; bytes successfully before error: 65536
write size:     131072; bytes successfully before error: 0
write size:     262144; bytes successfully before error: 0
#+end_example

score -1 · Answer 8 · answered Aug 15 '21 at 14:25

-1

As simple as executing this:

echo "$(($(ulimit -p)*512)) bytes"

answered Aug 15 '21 at 14:25

Akib Azmain Turja

241

4

Unfortunately this is not correct, the value reported by ulimit -p is PIPE_BUF which is the largest size for which writes are guaranteed to be atomic. See https://stackoverflow.com/a/4625116/813810 – Diego Jan 04 '22 at 14:22

How big is the pipe buffer?

8 Answers8

Linked