Command substitution in for loop not working

Question

I want to keep all files not ending with .bat

I tried

for f in $(ls | egrep -v .bat); do echo $f; done

and

for f in $(eval ls | egrep -v .bat); do echo $f; done

But both approaches yield the same result, as they print everything. Whereas ls | egrep -v .bat and eval ls | egrep -v .bat work per se, if used apart from the for loop.

EDIT

It's interesting to see that if I leave out the -v flag, the loop does what it should and lists all files ending with .bat.

---

^{Feel free to edit the question title, as I was not sure what the problem is.}

I'm using GNU bash, version 4.1.10(4)-release (i686-pc-cygwin).

---

EXAMPLE

$ ls -l | egrep -v ".bat"
total 60K
-rwx------+ 1 SYSTEM SYSTEM 5.3K Jun  6 20:31 fsc*
-rwx------+ 1 SYSTEM SYSTEM 5.3K Jun  6 20:31 scala*
-rwx------+ 1 SYSTEM SYSTEM 5.3K Jun  6 20:31 scalac*
-rwx------+ 1 SYSTEM SYSTEM 5.3K Jun  6 20:31 scaladoc*
-rwx------+ 1 SYSTEM SYSTEM 5.3K Jun  6 20:31 scalap*

Command is working, but not in the for loop.

$ for f in $(ls | egrep -v .bat); do echo $f; done
fsc
fsc.bat
scala
scala.bat
scalac
scalac.bat
scaladoc
scaladoc.bat
scalap
scalap.bat
scalac
scalac.bat
scaladoc
scaladoc.bat
scalap
scalap.bat

DEBUG $ set -x

mike@pc /cygdrive/c/Program Files (x86)/scala/bin
$ for f in $(ls | egrep -v .bat); do echo $f; done
++ ls -hF --color=tty
++ egrep --color=auto -v .bat
+ for f in '$(ls | egrep -v .bat)'
+ echo fsc
fsc
+ for f in '$(ls | egrep -v .bat)'
+ echo fsc.bat
fsc.bat
// and so on

Answer 1

A few things wrong in your code:

Using unquoted command substitution ($(...)) without setting $IFS

Leaving expansions unquoted is the split+glob operator. The default is to split on space, tab and newline. Here, you only want to split on newline, so you need to set IFS to that as otherwise that means that will not work properly if filenames contain space or tab characters

Using unquoted command substitution without set -f.

Leaving expansions unquoted is the split+glob operator. Here you don't want globbing, that is the expansion of wildcards such as scala* into the list of matching files. When you do not want the shell to do globbing, you have to disable it with set -f

ls aliased to ls -F

The issue above is aggravated by the fact that you have ls aliased to ls -F. Which adds / to directories and * to executable files. So, typically, because scala is executable, ls -F outputs scala*, and as a globbing pattern, it is expanded to all the filenames that start with scala which explains why it seems like egrep -v is not filtering files out.

Assuming filenames don't contain newline characters

newline is as valid a character as any in a filename. So parsing the output of ls typically doesn't work. As for instance the output of ls in a directory that contains a and b files is the same as in a directory that contains one file called a\nb.

Above egrep will filter the lines of the filenames, not the filenames

Using egrep instead of grep -E

egrep is deprecated. grep -E is the standard equivalent.

Not escaping the . regex operator.

Above, you used egrep to enable extended regular expressions, but you don't use any of the extended RE specific operator. The only RE operator you're using is . to match any character, while it looks like that's not what you intended. So you might as well have used grep -F here. Or use grep -v '\.bat'.

Not anchoring the regexp on end-of-line

egrep .bat will match any line that contains any character followed by bat, so that's the regexp that means anything that contains bat not in first position. It should have been grep -v '\.bat$'.

Leaving $f unquoted

Leaving an expansion unquoted is the split+glob operator. There, you want neither, so $f should be quoted ("$f").

Use echo

echo expands the ANSI C escape sequences in its arguments and/or treats strings like -n or -e specially depending on the echo implementation (and/or the environment).

Use printf instead.

So a better solution:

for f in *; do
  case $f in
    (*.bat);;
    (*) printf '%s\n' "$f"
  esac
done

Though if there's no non-hidden file in the current directory, that will still output *. You can work around that in zsh by changing * to *(N) or in bash by running shopt -s nullglob.

Answer 2

You should not use ls to parse files this way.

First set extglob globstar by shopt -s extglob globstar then

for f in !(*.bat)
  do
   printf '%s\n' "$f"
  done

Using find

find . -type f ! -name '*.bat' 

Use the negation operator for a safer treat of files.

Answer 3

There's nothing inherently wrong with your for loop:

$ for f in $(ls | egrep -v .bat); do echo $f; done

Here's my output for the above command:

$ for f in $(ls | egrep -v .bat); do echo $f; done
fsc
scala
scalac
scaladoc
scalap

Possible fix

One suggestion would be to quote the argument to egrep, like so:

$ for f in $(ls | egrep -v '.bat'); do echo $f; done

Also take a look at the Bash Pitfalls wiki for other potential issues when scripting things in Bash such as this. In particular this sounds like the most plausible reason why you're encountering this problem. Check to see if any of your files being returned contain spaces.

Debugging

Another thing to try is to enable verbose debugging of your bash command, prior to running it so you can see what's going on behind the scenes.

This enables debugging:

$ set -x

Then run your command:

$ for f in $(ls | egrep -v .bat); do echo $f; done
++ ls --color=auto
++ egrep --color=auto -v .bat
+ for f in '$(ls | egrep -v .bat)'
+ echo fsc
fsc
+ for f in '$(ls | egrep -v .bat)'
+ echo scala
scala
+ for f in '$(ls | egrep -v .bat)'
+ echo scalac
scalac
+ for f in '$(ls | egrep -v .bat)'
+ echo scaladoc
scaladoc
+ for f in '$(ls | egrep -v .bat)'
+ echo scalap
scalap

Then disable it:

$ set +x

Issue with Cygwin?

Given the examples work fine on a number of native Linux systems the problem is most likely rooted in having something to do with Cygwin and/or it's particular versions of bash and egrep. I'd pay particular attention to the field separator in Bash, $IFS to see if there is an issue between the various line separators on Windows (0x0d,0x0a) vs. Unix (0x0a).

I do not have an installation of Cygwin so I have no method for proving this hypothesis.