1

I have a command which streams data from a server and writes lines to stdout. There is a large number of records that I would like to pipe to less, but I would like to prevent my tool from continuing to stream data until the user goes to the next page, otherwise the tool will continue to download enormous amounts of data while the user is not viewing it yet.

Does less block the input process from writing to stdout until the user goes to the next page, or does less attempt to buffer the entire input somewhere?

If it's the latter, is it possible to change the behavior to the former via less flags, or some implementation details in my code, or do I need to implement my own pagination system?

Gilles 'SO- stop being evil'
  • 829,060
Brad
  • 155

1 Answers1

1

Less reads its input lazily. It only reads as much as it needs. If the input is a pipe, the program writing to the pipe will eventually block once it fills the pipe's fixed-size buffer (typically 512B to a few kB).

The laziness stops when the user uses a command that requires getting to the end of the file, such as > or an unsuccessful search. To fulfill such a request, less must read until the end of the file. If the input is infinite (because the program writing to the pipe keeps writing forever), the request will never complete.

If a command takes too long, the user can interrupt it with Ctrl+C. This causes less to close the input, which in turn means that the program writing to the pipe will receive a SIGPIPE signal if the pipe buffer fills up. After interrupting with Ctrl+C, there's no way to resume reading the input if the input comes from a pipe.

Gilles 'SO- stop being evil'
  • 829,060
  • I think the standard OS pipe buffer (for years) is 8k. There might be 8k on both the read side and write side of the pipe, but usually not. – user10489 Dec 07 '21 at 00:18
  • @user10489 It's classically 512B, still that size on FreeBSD 12. 4kB on modern Linux systems. – Gilles 'SO- stop being evil' Dec 07 '21 at 12:17
  • Odd. BSD 4.2 system I used 25 years ago was 8k. Or maybe it was 4k on each side. At any rate, the buffering should not be less than a page size. Does FreeBSD 12 use 512B pages? – user10489 Dec 07 '21 at 12:38
  • https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer says it could be up to 64k, although google agrees BSD 12 uses 512b. – user10489 Dec 07 '21 at 12:46
  • @user10489 There's no “side”: the buffer is attached to the pipe itself, not to each side of the pipe. Are you confusing the in-kernel buffer associated to the pipe with read-ahead buffers and write-pending buffers managed by libc inside each process? The size is not related to MMU pages, if that's the kind of pages you mean. – Gilles 'SO- stop being evil' Dec 07 '21 at 12:49
  • You are both right and wrong. This is implementation dependent. Many unixes use a page size for this pipe buffering because it is convenient. Some don't apparently. If this is a direct pipe between processes, then yes there's one buffer in the kernel. If this is two processes on opposite ends of a network socket, there could be a transmit buffer and a receive buffer. Linux explicitly uses (a multiple of) a page size and I remember old versions of BSD doing the same. – user10489 Dec 07 '21 at 12:58