16

Can someone put these tools in order of portability? Which of these is certain to be found on even the most minimal *nix systems? Is any of them 100% sure to be present? My guess is that the order is the following:

  1. awk
  2. sed
  3. sh
  4. perl

While I imagine there are systems which don't default to a bourne shell, some shell will be present as the default, will that always be at /bin/sh? Presumably not if it's not a bourne-type shell. Both awk and sed have pages explaining them on the POSIX specification so presumably they will always be present. Is that so? Can I be sure that both will be installed on any *nix? Including embedded systems?

terdon
  • 242,166
  • embedded systems often sacrifice portability to stay minimal. Many are not POSIX compliant. – jordanm Apr 04 '14 at 15:29
  • @jordanm does that mean they might conceivably not have any of these? I am assuming a system that provides at least a minimal shell interface. I know that busybox provides sh, awk and sed for example and my busybox-based NAS also has perl. – terdon Apr 04 '14 at 15:31
  • 1
    You may have them, but what they do will vary across systems, so the question does not make much sense as it is stated. – Stéphane Chazelas Apr 04 '14 at 15:35
  • @StephaneChazelas ah, could you help me narrow it down then? Isn't it possible to sort these in terms of how likely they are to be found on system X? My experience is strictly Linux and a couple of embedded NAS systems. I know you'll never get a GNU/Linux without all 4 of these but can't you give me an idea of which of them is safest to assume to be present? – terdon Apr 04 '14 at 15:41

2 Answers2

19

Which is the most portable of sed, awk, perl and sh?

sed, sh and awk are portable being specified by POSIX, perl is not as not being backed by a standard.

Can someone put these tools in order of portability?

If you stick to compliant code, there should be no order of portability for the three POSIX commands.

Which of these is certain to be found on even the most minimal *nix systems?

The three POSIX ones along with many other utilities are mandatory for an OS to be POSIX. OSes lacking some of them due to minimization, or providing incomplete/non-conforming implementations do exist though.

Actually, most (if not all) free and open source Unix like operating systems probably would not pass the conformance process should they try to, and they never try anyway.

Is any of them 100% sure to be present?

I would be surprised to find a *nix like OS lacking a Bourne syntax based shell, but anything is possible, especially with embedded systems.

My guess is that the order is the following: Some shell will be present as the default, will that always be at /bin/sh?

/bin/sh is likely to be a Bourne syntax family shell but is not guaranteed to be POSIX compliant, even en POSIX compliant systems. For example, it is /usr/xpg4/bin/sh on Solaris 10 and older while /bin/sh is the legacy original Bourne shell which is not POSIX.

jlliagre
  • 61,204
12

Take a cue from the Autotools: stick to the lowest common denominator of Bourne and POSIX shell — possibly augmented by sed — if you have to write something that must work everywhere. There may exist systems where something breaks, but you can work around such problems by rewriting.

For example, some ancient systems have problems with expansion errors in test, a.k.a. [:

 if [ $foo = bar ] ; then...

so the Autoconf practice is to rewrite it in double quotes with a single character prefix, like so:

 if [ x"$foo" = "xbar" ] ; then...

You could also use "x$foo" here. This guards against the possibility that $foo could be a valid option to test(1), and since [ is an alias for test, it could misinterpret the expression. The solution is to set up a situation where the unknown argument to [ always begins with x, which means it cannot have special meaning to [.

(Autoconf also recommends using test instead of [, but that advice comes about as a reaction to possible conflicts with M4, which also uses [ in its syntax.)

awk is POSIX, so theoretically it is available everywhere. It is even in Busybox, so you will have an awk implementation even in some very restrictive embedded Linux systems. Still, I'd be less surprised to come across a system without awk than sed. I suppose it comes down to complexity: simpler tools are more likely to survive aggressive triage.

Perl isn't part of any widespread standard, POSIX or otherwise, so you simply cannot count on it if you know nothing in advance about the target environment. Perl is not installed by default in:

  • Cygwin
  • FreeBSD and NetBSD
  • "minimal" installs for some Linuxes, including Slackware
  • many embedded Linuxes that rely primarily on Busybox for their userland

The Autoconf manual has a chapter on portable shell programming which should be helpful to you. The last section covers tools like sed, awk, and many others.

Anthony Geoghegan
  • 12,950
Warren Young
  • 72,032
  • Thanks, but I'm wondering about the order of portability of these. If I read you right, you are suggesting that sh and sed are the most portable. OK, what of the others? And is sh really that portable? What if the default shell is a C-shell derivative? Will such a system still have a symlink at /bin/sh? – terdon Apr 04 '14 at 15:39
  • Thanks for the edit, that's the thing though, both sed and awk are provided by busybox, why would you be more surprised to find sed absent? Do you have any idea if busybox alternatives also provide sed and awk? – terdon Apr 04 '14 at 15:50
  • Re: POSIX vs Bourne shell, rewrote the advice to stick to LCD of the two. Re: test quoting, I misremembered the reason for it, but found the chapter in the Autoconf manual where the reason is covered, and edited the answer accordingly. Re: sed vs awk and Busybox, I explained that: greater complexity and triage. sed does less, so its binary will be smaller, hence there is less reason to remove it than awk, which is fairly complex. (20 kiB vs 48 kiB on my system.) Re: Busybox alternatives, I'm not aware of any other one-stop-shopping alternative. – Warren Young Apr 04 '14 at 16:00
  • +1 Because I agree with your general premise here (that the lowest common denominator will be some kind of sh -- odd that terdon ranks awk and sed above this o_O?) although this is only coincidental with Autotool's purposes, since some environments, particularly small ones, may not be intended for building, period. – goldilocks Apr 05 '14 at 14:07
  • @TAFKA'goldilocks' not that odd, just simple ignorance on my part :). I'm thinking of systems whose default shell is not a bourne one. Perhaps they don't exist though. – terdon Apr 08 '14 at 17:17
  • @terdon: What does the default shell matter? All Unixy systems have /bin/sh, and that is always a Bourne-like shell, so put #!/bin/sh at the top of the script, and it will be interpreted by a Bourne type shell. /bin/sh is never a C shell or something even more incompatible. – Warren Young Apr 08 '14 at 17:42
  • 1
    @WarrenYoung that's the kind of thing I was wondering about. I have (had) no idea whether all *nix systems would actually have a bourne shell installed by default. If they all have a /bin/sh, then things are different but I did not know that all would have one. Sorry for not accepting this by the way, I was torn between the yours and jlliagre's answers both of which answered my question. I chose his because it addressed it more directly and because he has less rep :). – terdon Apr 08 '14 at 17:54