for vs find in Bash

Question

When looping through files there are two ways:

1. use a for-loop:

   for f in *; do
       echo "$f"
   done
   

2. use find:

   find * -prune | while read f; do 
       echo "$f"
   done
   

Assuming these two loops will find the same list of files, what are the differences in those two options in perfomance and handling?

Answer 1

I tried this on a directory with 2259 entries, and used the time command.

The output of time for f in *; do echo "$f"; done (minus the files!) is:

real    0m0.062s
user    0m0.036s
sys     0m0.012s

The output of time find * -prune | while read f; do echo "$f"; done (minus the files!) is:

real    0m0.131s
user    0m0.056s
sys     0m0.060s

I ran each command several times, so as to eliminate cache misses. This suggests keeping it in bash (for i in ...) is quicker than using find and piping the output (to bash)

Just for completeness, I dropped the pipe from find, since in your example, it's wholly redundant. The output of of just find * -prune is:

real    0m0.053s
user    0m0.016s
sys     0m0.024s

Also, time echo * (output isn't newline separated, alas):

real    0m0.009s
user    0m0.008s
sys     0m0.000s

At this point, I suspect the reason echo * is quicker is it's not outputting so many newlines, so the output isn't scrolling as much. Let's test...

time find * -prune | while read f; do echo "$f"; done > /dev/null

yields:

real    0m0.109s
user    0m0.076s
sys     0m0.032s

while time find * -prune > /dev/null yields:

real    0m0.027s
user    0m0.008s
sys     0m0.012s

and time for f in *; do echo "$f"; done > /dev/null yields:

real    0m0.040s
user    0m0.036s
sys     0m0.004s

and finally: time echo * > /dev/null yields:

real    0m0.011s
user    0m0.012s
sys     0m0.000s

Some of the variation can be accounted for by random factors, but it seems clear:

• output is slow
• piping costs a bit
• for f in *; do ... is slower than find * -prune, on its own, but for the constructions above involving pipes, is quicker.

Also, as an aside, both approaches appear to handle names with spaces just fine.

EDIT:

Timings for find . -maxdepth 1 > /dev/null vs. find * -prune > /dev/null:

time find . -maxdepth 1 > /dev/null:

real    0m0.018s
user    0m0.008s
sys     0m0.008s

find * -prune > /dev/null:

real    0m0.031s
user    0m0.020s
sys     0m0.008s

So, additional conclusion:

• find * -prune is slower than find . -maxdepth 1 - in the former, the shell is processing a glob, then building a (large) command line for find. NB: find . -prune returns just ..

More tests: time find . -maxdepth 1 -exec echo {} \; >/dev/null:

real    0m3.389s
user    0m0.040s
sys     0m0.412s

Conclusion:

• slowest way to do it so far. As was pointed out in the comments for the answer where this approach was suggested, each argument spawns a shell.

Answer 2

1.

The first one:

for f in *; do
  echo "$f"
done

fails for files called -n, -e and variants like -nene and with some bash deployments, with filenames containing backslashes.

The second:

find * -prune | while read f; do 
  echo "$f"
done

fails for even more cases (files called !, -H, -name, (, file names that start or end with blanks or contain newline characters...)

It's the shell that expands *, find does nothing but print the files it receives as arguments. You could as well have used printf '%s\n' instead which as printf is builtin would also avoid the too many args potential error.

2.

The expansion of * is sorted, you can make it a bit faster if you don't need the sorting. In zsh:

for f (*(oN)) printf '%s\n' $f

or simply:

printf '%s\n' *(oN)

bash has no equivalent as far as I can tell, so you'd need to resort to find.

3.

find . ! -name . -prune ! -name '.*' -print0 |
  while IFS= read -rd '' f; do
    printf '%s\n' "$f"
  done

(above using a GNU/BSD -print0 non-standard extension).

That still involves spawning a find command and use a slow while read loop, so it will probably be slower than using the for loop unless the list of files is huge.

4.

Also, contrary to shell wildcard expansion, find will do a lstat system call on each file, so it's unlikely that the non-sorting will compensate for that.

With GNU/BSD find, that can be avoided by using their -maxdepth extension which will trigger an optimization saving the lstat:

find . -maxdepth 1 ! -name '.*' -print0 |
  while IFS= read -rd '' f; do
    printf '%s\n' "$f"
  done

Because find starts outputting file names as soon as it finds them (except for the stdio output buffering), where it may be faster is if what you do in the loop is time consuming and the list of file names is more than a stdio buffer (4/8 kB). In that case, the processing within the loop will start before find has finished finding all the files. On GNU and FreeBSD systems, you may use stdbuf to cause that to happen sooner (disabling stdio buffering).

5.

The POSIX/standard/portable way to run commands for each file with find is to use the -exec predicate:

find . ! -name . -prune ! -name '.*' -exec some-cmd {} ';'

In the case of echo though, that's less efficient than doing the looping in the shell as the shell will have a builtin version of echo while find will need to spawn a new process and execute /bin/echo in it for each file.

If you need to run several commands, you can do:

find . ! -name . -prune ! -name '.*' -exec cmd1 {} ';' -exec cmd2 {} ';'

But beware that cmd2 is only executed if cmd1 is successful.

6.

A canonical way to run complex commands for each file is to call a shell with -exec ... {} +:

find . ! -name . -prune ! -name '.*' -exec sh -c '
  for f do
    cmd1 "$f"
    cmd2 "$f"
  done' sh {} +

That time, we're back to being efficient with echo since we're using sh's builtin one and the -exec + version spawns as few sh as possible.

7.

In my tests on a directory with 200.000 files with short names on ext4, the zsh one (paragraph 2.) is by far the fastest, followed by the first simple for i in * loop (though as usual, bash is a lot slower than other shells for that).

Answer 3

I would go definitely with find although I would change your find to just this:

find . -maxdepth 1 -exec echo {} \;

Performance wise, find is a lot faster depending on your needs of course. What you have currently with for it will only display the files/directories in the current directory but not the directories contents. If you use find it will also show the contents of the sub-directories.

I say find is better since with your for the * will have to be expanded first and I'm afraid that if you have a directory with a huge amount of files it might give the error argument list too long. Same goes for find *

As an example, in one of the systems that I currently use there is a couple of directories with over 2million files (<100k each):

find *
-bash: /usr/bin/find: Argument list too long

Answer 4

find * -prune | while read f; do 
    echo "$f"
done

is a useless use of find - What you're saying is effectively "for each file in the directory (*), don't find any files. Also, it's not safe for several reasons:

• Backslashes in paths are treated specially without the -r option to read. This is not an issue with the for loop.
• Newlines in paths would break any non-trivial functionality inside the loop. This is not an issue with the for loop.

Handling any file name with find is difficult, so you should use the for loop option whenever possible for that reason alone. Also, running an external program like find will in general be slower than running an internal loop command like for.

Answer 5

But we are suckers for performance questions! This request-for-experiment makes at least two assumptions that make it not terribly valid.

A. Assume that they find the same files…

Well, they will find the same files at first, because they're both iterating over the same glob, namely *. But find * -prune | while read f suffers from several flaws that make it quite possible it won't find all the files you expect:

1. POSIX find isn't guaranteed to accept more than one path argument. Most find implementations do, but still, you shouldn't rely on that.
2. find * can break when you hit ARG_MAX. for f in * will not, because ARG_MAX applies to exec, not builtins.
3. while read f can break with filenames starting and ending with whitespace, which will get stripped off. You could overcome this with while read and its default parameter REPLY, but that still won't help you when it comes to filenames with newlines in them.

B. echo. Nobody's going to do this just to echo the name of the file. If you want that, just do one of these:

printf '%s\n' *
find . -mindepth 1 -maxdepth 1 # for dotted names, too

The pipe to the while loop here creates an implicit subshell that closes when the loop ends, which might be unintuitive for some.

To answer the question, here are the results in a directory of mine that has 184 files and directories in it.

$ time bash -c 'for i in {0..1000}; do find * -prune | while read f; do echo "$f"; done >/dev/null; done'

real    0m7.998s
user    0m5.204s
sys 0m2.996s
$ time bash -c 'for i in {0..1000}; do for f in *; do echo "$f"; done >/dev/null; done'

real    0m2.734s
user    0m2.553s
sys 0m0.181s
$ time bash -c 'for i in {0..1000}; do printf '%s\n' * > /dev/null; done'

real    0m1.468s
user    0m1.401s
sys 0m0.067s

$ time bash -c 'for i in {0..1000}; do find . -mindepth 1 -maxdepth 1 >/dev/null; done '

real    0m1.946s
user    0m0.847s
sys 0m0.933s

Answer 6

find * will not work right if * produces tokens which look like predicates rather than paths.

You cannot use the usual -- argument to fix this because -- indicates the end of options, and find's options come before paths.

To fix this issue you can use find ./* instead. But then it's not producing exactly the same strings as for x in *.

Note that find ./* -prune | while read f .. does not actually use the scanning functionality of find. It is the globbing syntax ./* which actually traverses the directory and generates names. Then the find program will have to perform at least a stat check on each one of those names. You have the overhead of launching the program and having it access these files, and then doing I/O to read its output.

It is hard to imagine how it could be anything but less efficient than for x in ./* ....

Answer 7

Well for starters for is a shell keyword, built into Bash, while find is a separate executable.

$ type -a for
for is a shell keyword

$ type -a find
find is /usr/bin/find

The for loop will only find the files from the globstar character when it expands, it won't recurse into any directories it finds.

Find on the otherhand will also be given a list expanded by the globstar, but it will recursively find all the files and directories below this expanded list and pipe each one through to the while loop.

Both these approach might be considered dangerous in the sense that they don't handle paths or filenames that contain spaces.

That's about all I can think of worth commenting on these 2 approaches.

Answer 8

If all of the files returned by find can be processed by a single command (obviously not applicable to your echo example above), you can use xargs:

find * |xargs some-command

Answer 9

For years I've been using this:-

find . -name 'filename'|xargs grep 'pattern'|more

to look for certain files (e.g. *.txt) that contain a pattern that grep can look for and pipe it into more so that it doesn't scroll off the screen. Sometimes I use the >> pipe to write the results to another file that I can look at later.

Here's a sample of the result:-

./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:Message-ID: <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:In-Reply-To: <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:  <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:  <A165CE5C-61C5-4794-8651-66F5678ABCBF@usit.net>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2008-August.txt:Message-ID: <448E53556A3F442ABC58203D6281923E@hypermax>
./Documents/Organ_docos/Rodgerstrio321A/rodgersmylist/2011-April.txt:URL: http://mylist.net/private/rodgersorganusers/attachments/20110420/3f