3

I would like to put shebang #!/bin/sh -eufo pipefail in my script. But there're several things strange:

  1. The script would fail with that shebang in FreeBSD but not when run on MacOS
  2. on FreeBSd, the same shebang works when directly executed from command line (also /bin/sh).
>>> sh -eufo pipefail -c 'echo hi'  # this works
hi

>>> cat <<EOF > script

#!/bin/sh -eufo pipefail
echo hi
EOF


>>> chmod +x ./script 
>>> ./script  # this doesn't work on FreeBSD but works on MacOS
Illegal option -o ./script


>>> cat ./script 
#!/bin/sh -eufo pipefail
echo hi


>>> uname -a
FreeBS 11.3-RELEASE-p7
KFL
  • 269
  • You forgot to try sh '-eufo pipefail' ./script. – JdeBP Aug 22 '20 at 11:41
  • My impression is that you do not get a 100% usable answer because your question does not give the needed information. ash e.g. does not support set -o pipefail and people cannot help you because you hide the error messages you got. – schily Aug 22 '20 at 22:32
  • I did not hide anything. I spent time coming up with a easy to reproduce code snippet and included the output in the question in full fidelity. I thought that should be pretty obvious. – KFL Aug 22 '20 at 22:37
  • 1
    Oh sorry, I missed the error message illeal option. So it it obvious that the problem is not the #! but rather the fact that you are using a non-standard option that is not supported by ash on FreeBSD. – schily Aug 23 '20 at 06:32
  • 1
    @schily, note that it says Illegal option -o ./script, not e.g. Illegal option -o pipefail, which is what I get for e.g. Debian's dash when started with dash -o pipefail (command name + two args). And I know, dash isn't the FreeBSD sh, but an ash derivative nonetheless, so it might be related. – ilkkachu Aug 23 '20 at 09:25
  • @ilkkachu This is a result of the one arg only "limitation" from the #! feature, so the shell does not see the second arg from the #! line and consumes the script argument instead. My tests on FreeBSD with /bin/sh -p pipefail show that ash does not support that option. – schily Aug 23 '20 at 09:45

2 Answers2

1

MacOS still retains the old FreeBSD behaviour from before 2005. In 2005, there was a major change in the way that the FreeBSD kernel handled #! at the start of an executable file passed to execve(), to bring it more into line with some other operating system kernels, including Linux and the NetBSD kernel.

Commentary in the NetBSD kernel source code tries to paint this as a universal:

 * collect the shell argument.  everything after the shell name
 * is passed as ONE argument; that's the correct (historical)
 * behaviour.
— kern/exec-script.c. NetBSD. lines 189 et seq..

It actually is not. Sven Mascheck did some testing about a decade ago and there are four basic behaviours, the AT&T Unix System 5 one having as much claim to being "correct historical" behaviour as the 4.2BSD one has:

  • Ignore the characters (before 4.2BSD and AT&T Unix System 5).
  • Pass the whole string in a single argument (4.2BSD, NetBSD, Linux and FreeBSD from 2005 onwards).
  • Split the string up by whitespace and pass it as multiple arguments (FreeBSD before 2005 and MacOS).
  • Split the string up by whitespace and pass just the first argument (AT&T Unix System 5 and Solaris)

I've only included the operating systems relevant to this answer in parentheses. M. Mascheck checked a lot more, as did Ahmon Dancy in discussion of FreeBSD Problem Report 16393. See the further reading for the full lists.

What brought things to a head in FreeBSD in 2005 was that, ironically, FreeBSD wasn't quite as simple as that. It had had a change introduced that was intended to make things written in popular books about Perl actually work: arguments were skipped after a comment character. The books had recommended things like: 

#!/bin/sh -- # -*- perl -*-
— Larry Wall, Tom Christiansen, Jon Orwant (2000). Programming Perl: 3rd Edition. O'Reilly Media. ISBN 9780596000271. p. 488.

PR 16393 in 2000 was a way of making the kernel handle executable Perl scripts, written in the way that Larry Wall no less had said would work. However, it broke other stuff and didn't completely work.

There was some back and forth on this. Finally, in 2005 the mechanism to make Larry Wall et al.'s idea work was moved out of the kernel, which was made to behave compatibly with Linux, NetBSD, and 4.2BSD (rather than Solaris and AT&T Unix System 5) and made the responsibility of sh.

The behaviour since 2005 has thus been that the shell gets three arguments, the second argument being the entire tail of the #! line, and invoking your script directly with execve() is effectively the same as invoking:

sh '-eufo pipefail' ./script

It should be fairly obvious why the Almquist shell (which is what sh is on FreeBSD) is thinking that ./script is the option argument for the -o option, and that it is treating the pipefail part as further single-letter options collected behind - (which it hasn't got around to processing yet).

An also obvious alternative is to have set -o pipefail as the first command in the script, as pointed out at https://unix.stackexchange.com/a/533418/5132 for the Bourne Again shell. This was only added to the FreeBSD Almquist shell in 2019, however and thus is only available in very recent versions of FreeBSD. (The Debian Almquist shell has not yet had it added, as of 2020.)

Further reading

JdeBP
  • 68,745
  • The table from Sven Mascheck is not correct... SunOS-4 of course supports suid scripts and introduced to let the kernel typicaly use /dev/fd/5 as argument instead of the shellscipt. /dev/fd/5 was opened by the kernel for security reasons. You however need to specify the shell option -p in the first line of the script as the shell forbids scripts where effective and real user ID differ. So for correct usability, you only ever need one single option from the #! line: -p. – schily Aug 22 '20 at 13:49
  • The question is about FreeBSD and MacOS, M. Schilling. – JdeBP Aug 22 '20 at 13:57
  • 1
    If people like to write something that works everywhere, it is a good practice to follow the common demnominator Mr. anonymous. And BTW: if you understand the historical background it is possible to understand an alleged limitation. – schily Aug 22 '20 at 14:05
  • 1
    hmm, what are the characters in "Ignore the characters" in the first point? Did those OS's ignore the whole hashbang line, ignore the characters after the first space in the line, or ignore characters after the second space,or ...? – ilkkachu Aug 22 '20 at 16:33
  • 1
    "It should be fairly obvious why the Almquist shell is thinking that ./script is the option argument for the -o option" -- actually, I don't find that obvious at all... Ksh and Zsh seem to interpret that as the -o option with an argument ␣pipefail (with a leading space), and as far as I can read getopts and getopt(), that's how they're supposed to interpret it. – ilkkachu Aug 22 '20 at 16:36
  • It was obvious to you how this sort of parsing works back when you wrote https://unix.stackexchange.com/a/413516/5132 . (-: – JdeBP Aug 22 '20 at 16:39
  • @JdeBP, um, yes? I'm not sure how that answer is relevant? What I wrote there is what I tried to say above, that by my reading of what getopt() and getopts should do, -eufo pipefail as one argument should be interpreted by them as containing the four options and the associated argument. What makes it obvious that ash should interpret its command line arguments in a manner different from getopt()? – ilkkachu Aug 22 '20 at 20:18
  • Thanks for the answer and in depth context! – KFL Aug 22 '20 at 23:14
  • @ilkkachu You are closer to the real problem than the answer you are commenting,but you are still not right. This is not about getopt() and -eufo pipefail as one argument will not be accepted by shells,nor is it compatible to the POSIX rules for option processing. – schily Aug 23 '20 at 07:07
  • @schily, well, it's not accepted all right, but in Ksh and Zsh because of the space, not because they'd take the next command line argument as the argument to -o. Let me put it this way: this answer says that it should be obvious that ash doesn't interpret options like getopt does, but I don't find it obvious at all. I tried to look, but didn't find any docs on how the shells (should) interpret an arg like -eufo pipefail. If it's just due to historical reasons, that's all fine, but it doesn't cost much to say it out loud instead of just going with "should be obvious". – ilkkachu Aug 23 '20 at 09:11
  • @schily, as for POSIX rules, Utility Syntax Guidelines has "Guideline 6: Each option and option-argument should be a separate argument", but then it refers to getopt() as assisting in handling options that conform to those guidelines, and to Utility Argument Syntax, which is described as "terminology used throughout POSIX.1-2017" and not a rule as such. Is there some other place I should be reading? – ilkkachu Aug 23 '20 at 09:17
  • @ilkkachu I am willing to help you to understand the relation between the POSIX Utility guidelines and getopt(), but this definitely is too much for a serires of comments, so please find a better discussion base than comments. In short: there are the POSIX rules and there is getopt(), but POSIX does not require to use getopt(). The Bourne Shell e.g.does not use getopt() to parse command line options and I even doubt that this would work to implement the required behavior. Even more, POSIX Utility rules are a deliberate subset of what is used on UNIX. – schily Aug 25 '20 at 06:19
0

Even though your #! usage is non-portable because it uses more than the simple:

#!/bin/sh

or the commonly supported single argument as in:

#!/bin/sh -oneflag

The real problem is that you are using a non-portable option -o pipefail.

This is an option from ksh93 and bash that is not supported by other shells.

Background:

  • On MacOS,/bin/sh is bash that has been compiled in a specific way (e.g. to make escape sequences in echo work by default) to make it POSIX compliant. Since bash supports pipefail (see above), this works on MacOS.

  • On FreeBSD, /bin/sh is ash that does not support pipefail.

You have two possible ways to go:

  • Do not use set -o pipefail

  • Wait for two years and try again. This will most likely work, because we decided to add this option to the next POSIX standard (Issue-8) 10 months ago, see https://www.austingroupbugs.net/view.php?id=789 and since the next POSIX standard will be published in approx. one year, there is a big chance that FreeBSD will add support for set -o pipefail to ash soon.

schily
  • 19,173