3

I have a script that's supposed to get the list of files of two directories, get differences and execute some code for certain files.

These are the commands to get the file lists:

list_in=$(find input/ -maxdepth 1 - type f | sed 's/input\///' | sort -u);
list_out=$(find output/ -maxdepth 1 - type f | sed 's/output\///' | sort -u);

I execute the script in the correct directory, so this shouldn't fail. The unprocessed files are determined by

list_todo=$(comm -23 <(echo "$list_in") <(echo "$list_out"));

since the option -23 for comm only prints lines of the first argument, that don't appear in both arguments and doesn't print lines that uniquely appear in the second argument.

However, only occasionally I get an error saying

command substitution: line 3: syntax error near unexpected token `('
command substitution: line 3: `comm -23 <(echo "$list_in") <(echo "$list_out")'

This really puzzles me, since the exact same script worked fine for the last 3 weeks. I'm using this on a cluster, so several processes might execute the script simultaneously. May the error be caused by this?

Update. The script is called with ./script and I've obviously set chmod +x script before.

(Disclaimer: Even though I'm working on a cluster and these first three lines of my script don't include any locking mechanisms: Of course, no file is ever processed twice)

stefan
  • 1,009
  • try executing the file using bash file.sh or chmod +x file.sh;./file.sh because if you use sh it starts up in a special, POSIX-compliant mode and See here, #22: "process substitution is not available". – harish.venkat Jan 15 '13 at 17:57
  • @harish.venkat thanks for your comment. I've done this with your second suggestion (except for file extension which shouldn't matter afaik). I've updated the question accordingly – stefan Jan 15 '13 at 18:00
  • if you execute by ./file.sh then you have shebang #!/bin/bash just for clarification – harish.venkat Jan 15 '13 at 18:10
  • 1
    Please post the complete file, or at least a real file that exhibit the problem. You can't have #!/bin/bash, the line to set list_in and the line to set list_out all above the line that sets list_todo if that line is line 3. Also, just in case, put a space between the two closing parentheses, it's conceivable that some version of bash may want )) to close (( or $(( and not two separate )s. – Gilles 'SO- stop being evil' Jan 16 '13 at 00:19
  • @Gilles: I've not specified #!bin/bash in the first line, so the excerpts shown in the question are my actual script (except for the directory name input beeing data, but who cares). But as harish.venkat sais: calling with ./script has implicit /bin/bash shebang. The weird thing is that this isn't consistent behaviour. It obviously can't be complete crap since it worked for over 3 weeks now, meaning roughly 20000 successful runs.. – stefan Jan 16 '13 at 00:24
  • @stefan No, this is wrong (and this is not what harish.venkat said). Calling ./script has an implicit #!/bin/sh shebang, unless you run it from bash. – Gilles 'SO- stop being evil' Jan 16 '13 at 00:31
  • @Gilles: Then I had some magic going on the last couple of weeks. I've added the bash shebang and calling explicitely with bash script. I'll report if the error is still occuring. – stefan Jan 16 '13 at 00:34

2 Answers2

3

The kernel recognizes certain file formats that it can execute natively. This includes at least one binary format. Additionally, files that begin with #! (shebang) are considered scripts; for example, if a file is located at /path/to/script and begins with #!/bin/bash then the kernel executes /bin/bash /path/to/script arg1 arg2 when you invoke /path/to/script arg1 arg2.

If the kernel doesn't recognize the file format, it returns ENOEXEC (exec format error) from the execve system call. Following a tradition from the ancient days of Unix kernels that didn't have the shebang feature, most programs make a second attempt to execute a program when the first attempt failed with the error ENOEXEC: they try to execute /bin/sh as the script interpreter.

Bash is a notable exception: it runs the script in itself, not in /bin/sh. So your script was working by accident when you invoked it from bash.

If you leave off the shebang line, your script may be executed under /bin/bash or under /bin/sh, depending on what program it was executed from. And /bin/sh evidently doesn't support process substitution on your system. It's probably still bash (the error message looks like the one from bash), but when bash is invoked under the name sh, it goes into POSIX compatibility mode which doesn't support process substitution.

The moral of the story: if you write a bash script, you must put #!/bin/bash on the first line.

Gilles 'SO- stop being evil'
  • 829,060
1

The $() and <() construct is bash only. That means that you need your shebang line to be:

#!/bin/bash

Particularly this will not work:

#!/bin/sh

If you need it to work with sh then you can use ` instead of $():

list_in=`find input/ -maxdepth 1 - type f | sed 's/input\///' | sort -u`

You can use mkfifo in place of <().

Ole Tange
  • 35,514