86

I am trying to read the output of a command in bash using a while loop.

while read -r line
do
    echo "$line"
done <<< $(find . -type f)

The output I got 

ranveer@ranveer:~/tmp$ bash test.sh
./test.py ./test1.py ./out1 ./test.sh ./out ./out2 ./hello
ranveer@ranveer:~/tmp$

After this I tried 

$(find . -type f) | 
while read -r line
do
    echo "$line"
done

but it generated an error test.sh: line 5: ./test.py: Permission denied.

So, how do I read it line by line because I think currently it is slurping the entire line at once.

Required output:

./test.py
./test1.py
./out1
./test.sh
./out
./out2
./hello
Gilles 'SO- stop being evil'
  • 829,060
RanRag
  • 5,875

5 Answers5

105

There's a mistake, you need < <(command) not <<<$(command)

< <( ) is a Process Substitution, $() is a command substitution and <<< is a here-string.

Gilles Quénot
  • 33,867
22

There's no need for command substitution if you want to use pipe. read reads from stdin by default, so you just pipe into it:

find . -type f -print0 |
while read -r -d '' line
do
    echo "$line"
done

or in one line

find . -type f -print0 | while read -r -d '' line; do echo "$line"; done

Using -print0 and -d '' is a protective measure against potential newline character in filenames, which by default indicates end of line to read (this can be changed with -d). '' used above is effectively null byte in bash (equivalent to $'\0').

Another already mentioned approach is to use find's -exec option. Most likely that would be the best option performance-wise (remember to use -exec cmd {} + variant so that it won't fork a process for each line being processed). But it really depends on what you do inside the while loop and how much data you process. The difference might be negligible or acceptable for your case.

ﾤ ﾡ
  • 364
16

Note that there's nothing stopping file names from containing newline characters. The canonical way to run a command for each file found by find is.

find . -type f -exec cmd {} \;

And if you want things done in bash:

find . -type f -exec bash -c '
  for file do
    something with "$file"
  done' bash {} +

Also, the canonical way to call the "read" command in scripts (if you don't want it to do extra processing on the input) is:

IFS= read -r var

-r is to stop read from treating backslash characters specially (as an escape character for separators and newline), And IFS= to set the list of separators to the empty string for read (otherwise if any whitespace character was in that list, they would be stripped from the beginning and end of the input).

Using loops in shells is usually a bad idea (not how things are done in shells where you make several tools work collectively and concurrently to a task rather than running one or more tools hundreds of times in sequence).

Stéphane Chazelas
  • 544,893
0

Remember that invoking commands has a cost (irrespective of what they do). This can have very negative impact on performance if you put them into a loop that will have multiple iterations. In case you need to process the output in some way, consider using awk. It is very powerful and can quickly handle a lot of input:

find . -type f | awk '{print}'
ﾤ ﾡ
  • 364
0

A small modification of Stéphane Chazelas's answer.

find . -type f -exec bash -c 'echo $0; head $0;' {} \;

This shows how the script you give to bash -c takes the first argument.

plhn
  • 203