1

I have a file called test containing

test
test

and I am running this command

while read line
do
echo "$line"
done </tmp/test

This should output "test" two times but it only outputs it once. Using while IFS= read -r line doesn't change anything. Something that fixes it is to output a third empty line, but it should work regardless.

DisplayName
  • 11,688
  • 9
    Is the second test line terminated? What's the output of od -vtc < /tmp/test? Note that lines have to end in a newline character, otherwise they're just garbage after the last line. – Stéphane Chazelas Nov 04 '16 at 13:02
  • As Stéphane Chazelas said : od output should appear as 0000000 t e s t \n t e s t \n – steve Nov 04 '16 at 14:49
  • 4
    I'm voting to close this question as off-topic because after almost two years, the user has not returned with more information about the data as requested in comments. – Kusalananda Oct 30 '18 at 06:37
  • Seems on topic as it's about "Shell scripting." Seems hostile to have closed as off topic when it seems on topic as per the help center.. Original user is not required to update as anyone with enough score can edit question with requested clarification. Then again, unix.se has always seemed hostile compared with stackoverflow or others – JGurtz May 15 '19 at 18:43

1 Answers1

2

Try:

while IFS='' read -r line || [ "$line" ]

Yes, it is reading from standard input (stdin), the <test redirection goes to stdin.

What might be happening is that read is designed to work with text files. Correct text files must end on a newline.

To make a file end on a newline turns out to be very fast:

[ -n "$(tail -c1 file)" ] && printf '\n' >>file

But if you can not edit the file to add a newline (if missing), the solution is to test (additionally) if something was read:

while read line || [ "$line" ]

That will make the loop execute if read failed (got to the end of the file without reading a newline (by default)) but something was read in (in "$line").

That will work correctly for files that do end on a newline or not.

No, a IFS='' read will not (directly) affect the reading of the last line, in fact, the IFS='' (compared to a default IFS[a] of space, tab, newline) will only affect the split into several variables (beside affecting the removal of leading and/or trailing white-space (if IFS contains only white space[b])). As there is only one variable ("$line") there is no splitting to be performed.

The -d option to read (in bash since 2.04) will not help either. There is no specific "end of file" character to match (the last byte could be any).

The only options left are either:

  • to repair the file to make it a correct text file
  • test if anything was read into the variable line.

The correct script, then, will be:

#!/bin/bash
while IFS='' read -r line || [ "$line" ]
do
    printf '%s\n' "$line"
done <test

IFS='' used to avoid problems with un-common values of IFS.

[a][b]Of course, If IFS may have any value, many other characters may get removed. Try (for values that end on 'x')

printf "test\ntesx\ntest" | while IFS="x" read -r line || [ "$line" ]; do echo "$line"; done

No, zsh does something different.