Reading n lines at a time from a named pipe in Ubuntu

Question

I am experimenting with named pipes in Ubuntu and have no prior experience with them. I did the following:

mkfifo pipe
ls>pipe&
cat pipe

This enters a list of all files in my folder into pipe and displays them. I am trying to read the pipe 2 lines at a time. So, I want a command when executed once, will give the names of first and second files in the folder. If the command is executed again, I want third and fourth filenames. Is there any way to do it?

I tried head -2 pipe. It displayed first two filenames. But when it was executed again, it hanged. What would be the right direction to proceed?

You may want to read line by line with the shell built-in read. — AlexP, Oct 22 '19 at 21:37

score 5 · Accepted Answer · answered Oct 23 '19 at 01:39

5

You want to keep the fifo open on the reading side. One way is to let bash keep an open file descriptor to it.

 #!/bin/bash
 mkfifo pipe
 ls > pipe &
 exec 8< pipe
 r2(){
   read -ru 8 fn1
   read -ru 8 fn2
   echo "$fn1" "$fn2"
 }
 r2 # print out first 2 files
 sleep 1 # do something
 r2 # print out next 2 files.

answered Oct 23 '19 at 01:39

icarus

17,920

Thank you so much. That works. – skr Oct 23 '19 at 14:02

Stéphane Chazelas · Answer 2 · 2021-02-27T13:50:24.563

You'd need a command that reads two lines and no more than two lines.

To be able to do that when the input is a pipe which is not seekable, commands have three choices:

read one byte at a time until they finds the second newline character.
peek at (without consuming) the contents of the pipe to see where that second newline is and then read as much data in one go.
read input in blocks, but if they find they have read more than the two lines, put back what they've read too much onto the pipe.

1 is expensive on many systems as that's one read() system call per byte. 2 and 3 don't work on all systems as not all systems support peeking at pipe or put back data onto the pipe and that's also not reliable when more than one process reads from the pipe at the same time.

So in general, except for a few exceptions, commands generally don't bother reading exactly how many lines they're being told to when their input is not seekable.

With most implementations of head, head -n2 will output the first two lines of its input, but may consume more input. A notable exception is the head builtin of the ksh93 shell which will peek if it can and read one byte at a time otherwise.

There are commands that can be told to make sure they don't read past the last line they're told to process.

With GNU sed, that's with the -u option:

sed -u 2q

will read 2 lines (1 byte at a time) and no more.

The line utility (no longer a standard one, so not always found these days) is one that reads exactly one line.

The read builtin of sh is required not to read more than one line. So that line command above can be implemented with:

IFS= read -r line; printf '%s\n' "$line"

(with the caveat that it adds a newline character even if the input line wasn't terminated).

(POSIX actually justified removing line on the basis that it could be achieved with read).

For a line implementation that doesn't add extra newlines and return false on eof.

line() {
  if IFS= read -r line; then
    printf '%s\n' "$line"
  else
    printf %s "$line"
    false
  fi
}

And twolines is just:

twolines() { line && line; }

And then you can do:

mkfifo pipe
ls > pipe &
while
  echo next two lines:
  twolines
do
  continue
done < pipe

Reading n lines at a time from a named pipe in Ubuntu

2 Answers2