Reading multiple files one line at a time in bash

Question

I know the basic way of reading from a command in bash:

cal | while IFS= read -r line ; do 
    echo X${line}X
done

But what if I want to read one line from several files/commands in a loop? I've tried named pipes but they'd only spit out one line.

$ cal > myfifo &
$ IFS= read -r line < myfifo
[cal exits]
$ echo $line
   February 2015

So what I'd really want is something like:

while [all pipes are not done]; do
    IFS=
    read line1 < pipe1    # reading one line from this one
    read line2 < pipe2    # and one line from this one
    read line3 < pipe3    # and one line from this one
    print things with $line1 $line2 and $line3
done

Big picture what I'm trying to do is process three different months from cal to do some colorizing for use with Conky. It's a bit of yak shaving, honestly, so has become academic and a 'learning experience' at this point.

You may want to read Why is using a shell loop to process text considered bad practice? — Stéphane Chazelas, Feb 04 '15 at 10:44

score 7 · Answer 1 · answered Feb 03 '15 at 01:49

paste would be the tidiest way to do it. But, with bash file descriptors and process substitutions:

exec 3< <(cal 1 2015)
exec 4< <(cal 2 2015)
exec 5< <(cal 3 2015)

while IFS= read -r -u3 line1; do
    IFS= read -r -u4 line2
    IFS= read -r -u5 line3
    printf "%-25s%-25s%-25s\n" "$line1" "$line2" "$line3"
done

exec 3<&-
exec 4<&-
exec 5<&-

    January 2015             February 2015             March 2015          
Su Mo Tu We Th Fr Sa     Su Mo Tu We Th Fr Sa     Su Mo Tu We Th Fr Sa     
             1  2  3      1  2  3  4  5  6  7      1  2  3  4  5  6  7     
 4  5  6  7  8  9 10      8  9 10 11 12 13 14      8  9 10 11 12 13 14     
11 12 13 14 15 16 17     15 16 17 18 19 20 21     15 16 17 18 19 20 21     
18 19 20 21 22 23 24     22 23 24 25 26 27 28     22 23 24 25 26 27 28     
25 26 27 28 29 30 31                              29 30 31

This only works when all three programs produce output with the same number of lines. — Chris Davies, Feb 04 '15 at 10:19

muru · Answer 2 · 2015-02-03T04:41:13.030

2

You could use paste to combine the output, and then read in lines:

paste -d $'\n' <(foo) <(bar) <(cat baz) | while IFS= read -r line1
do
    IFS= read -r line2 || break
    IFS= read -r line3 || break
    # ..
done

edited Feb 03 '15 at 04:41

answered Feb 02 '15 at 23:54

muru

72,889

Chris Davies · Answer 3 · 2015-02-04T10:14:59.993

0

This is the sort of structure that you'll need.

You were on the right lines with your FIFO but the reason cal exited when you read one line from it is that the read -r line < myfifo opened the FIFO, read a line, and then closed it again. Closing a pipe sends a signal to the other side indicating that no more writing (reading) can be done. So cal exited.

# Create the FIFOs
mknod pipe1 p
mknod pipe2 p
mknod pipe3 p

# Start off the commands
command1 >pipe1 &
command2 >pipe2 &
command3 >pipe3 &

# Attach file descriptors to the other side of the FIFOs
exec 11<pipe1 12<pipe2 13<pipe3

# Loop
IS_MORE=0
while [[ 0 eq $IS_MORE ]]
do
    IS_MORE=1
    read LINE1 <&11 && IS_MORE=0
    read LINE2 <&12 && IS_MORE=0
    read LINE3 <&13 && IS_MORE=0
    # ...
done

# Close descriptors and remove the FIFOs
exec 11<&- 12<&- 13<&-
rm -f pipe1 pipe2 pipe3

# Clean up the dead processes (zombies)
wait

# ...

edited Feb 04 '15 at 10:14

answered Feb 03 '15 at 00:00

Chris Davies

116,213
16
160
287

exec pipe1 <&11 doesn't work: "Bad file descriptor". – Greg Bell Feb 04 '15 at 07:04
@Greg sorry about that. It should be exec 11< pipe1. I've fixed the answer to reflect the correction. – Chris Davies Feb 04 '15 at 10:15
The eq needs to be a -eq. – Greg Bell Feb 06 '15 at 03:21
The & has always caused me grief. The man page doesn't really explain what it's for (and multiple uses of & in bash make it extra confusing). I get it now, thanks to you. Articles make noise like its a dereference operator, but really it's just a way of syntactically telling the difference between a file named N and a file descriptor N. Without it, ls >5 could be either one. In practice, it's just what you put before the file descriptor on the right side of a redirection. Thus exec 11<pipe1 doesn't need it but read LINE <&11 does. – Greg Bell Feb 06 '15 at 03:22
My mistake. The manual does talk about it, but says it duplicates file descriptors. That's confusing to me. Really what we're doing is pointing one fd at WHERE another points. Sort of like a dereference operator, but again not really necessary except for bash to tell the difference between a file and a file descriptor. – Greg Bell Feb 06 '15 at 04:28
@Greg it can be really confusing, yes. As you pointed out I got it wrong on my first attempt. Rightly or wrongly I think of & as an indicator that an FD is coming up, so I parse <&11 as meaning from FD 11. That then breaks with a construct like 11<pipe1, unfortunately, and at that point I run off to re-read the manpage again. (And again.) – Chris Davies Feb 06 '15 at 10:41

score 0 · Answer 4 · answered Jan 10 '18 at 19:06

while \
    IFS= read -r -u3 line1;do
    IFS= read -r -u4 line2
    IFS= read -r -u5 line3
    printf "%-25s%-25s%-25s\n" "$line1" "$line2" "$line3"
done 3< <(cal -h 1 2018) 4< <(cal -h 2 2018) 5< <(cal -h 3 2018)

Or perhaps the following if the 3 months are adjacent ones:

while IFS= read -r;do
    printf "%s\n" "${REPLY}"
done < <(cal -A 1 -B 1 -h 8 2018)

Reading multiple files one line at a time in bash

4 Answers4