3

I have been enthusiastic to shell scripting for some months, and found that the codes I write (in bash) do not work in some machines. This is very disappointing.

I realize that I should have learned the most portable shell scripting instead. This would, of course, be a bit painful in the beginning, but I believe it will pay off eventually. I hope my shell scripts will work almost everywhere, including the Linux family and the BSD family. (But I don't care if they run on Windows.)

Questions

  1. What's the most portable shell? EDIT: Please omit this question

  2. What are some serious disadvantages to use the most portable shell? EDIT: Please omit this question

  3. What else should I pay attention to if I want to write extremely portable codes? e.g. what core-utils and which versions should I use and avoid?

Jesse Nickles
  • 185
Student
  • 453
  • Your question is a bit missleading as it leaves open whether you like to know which shell runs on the largest number of platforms ot whether you are interested in a shell that does not implement POSIX extensions. – schily Dec 01 '19 at 23:00
  • 1
    I think the question is phrased slightly wrong; AIUI the question isn't about portability of different shells, but about writing shell scripts that are portable between different shells and shell environments. – Gordon Davisson Dec 02 '19 at 00:24
  • @GordonDavisson I did mean shells, but yeah also basic-utils. I will include that in the title. Thank you! – Student Dec 02 '19 at 01:34
  • 1
    I think you're still misunderstanding my point. Looking for a portable shell means you're looking for a shell (and utilities) you can bring with you and install on whatever system you happen to be using today; but AUIU you're actually interested in writing scripts that'll work on whatever shell (and utilities) happen to already be available on the system. That's not a matter of looking for a portable shell, but of looking for portable scripting practices. – Gordon Davisson Dec 02 '19 at 02:46
  • Really basic question: do all your scripts start with a shebang: #! /bin/bash ? You might be writing bash code, and it runs fine if that is your local default shell, but on a system where the default shell is ksh, the first incompatible command will break it. – Paul_Pedant Dec 02 '19 at 11:06
  • 2
    Shell Check is nice utility and website that can help write portable shell scripts. https://www.shellcheck.net – Christopher Dec 02 '19 at 12:46
  • @Christopher Or at least syntactically correct scripts. Let's not call shellcheck a "portability compliance checker". – Kusalananda Dec 02 '19 at 12:54
  • @Christopher Shellcheck helps with things like quoting, and points out common pitfalls, syntax errors, and semantic issues. It can say nothing about portability. – Kusalananda Dec 02 '19 at 13:02
  • Let's have a peaceful day.. I appreciate both of your answers. – Student Dec 02 '19 at 13:05
  • @GordonDavisson I get you! I do mean hoping to write shell scripts that work everywhere, and for testing I need the "most portable/universal shell" to test against. I'm not too sure how to change it to make it better.. if you have some suggestion, I am happy to accept an edit. Thanks a lot again for your comment! – Student Dec 02 '19 at 13:11
  • This site does not do multiple questions per question. Questions 1 and 2 should be removed on that ground alone. But a further ground for their removal is that question 3 is the real question to be answered, as M. Davisson has explained. And the title is misleading. – JdeBP Dec 02 '19 at 13:20
  • Thank you for the comment. I just edited accordingly.. hope it does not make it worse. – Student Dec 02 '19 at 16:26
  • @Kusalananda, shellcheck does know about at least some non-POSIX features, e.g. it warns about [[ .. ]] if the hashbang shows /bin/sh. – ilkkachu Dec 03 '19 at 11:50
  • @Christopher, was that really a warranted reaction? Or did I miss something here? – ilkkachu Dec 03 '19 at 12:21
  • @ilkkachu Thanks for the question. He's a moderator who decided to openly and personally mock me and then erase the history of having done so. What's left to do in this case - complain? I will no longer associate myself with abusive people such as this one, and just using the website is association enough. The man disgusts me, and I don't come here to get spit on for no reason. – Christopher Dec 03 '19 at 17:39
  • Shell programming is mostly calling out to assorted other programs to do the heavy lifting. It isn't enough to have a "standard" shell (like 'bash --posix'), and only use it's standard commands, but also take care not to use non-standard-mandated programs or extensions on them. – vonbrand Dec 10 '19 at 16:25
  • 1
    This is a great question, basically entirely within the domain of objective non-opinion answers, and it being closed is an utter testament to communal incompetence and overconfidence and an ur-example of how StackOverflow (and over-reliance on popular consensus) gets things wrong sometimes. Of course, the most efficient way to "improve" this question (or rather, convincingly explain that it's fine as it) is to post a great answer, but StackOverflow makes it impossible to answer closed questions. I wish whatever experiences are necessary to teach the relevant lessons on all responsible. – mtraceur Oct 05 '23 at 00:30

4 Answers4

10

Your bash scripts should work on any machine that has the bash shell installed, unless you use bash features that are too new for the version of the shell on the other system, like for example, trying to use name references (declare -n), associative arrays (declare -A), or the %(...)T format of printf (or any of a number of other different things) with the default bash on macOS (which is ancient; just install a newer bash with Homebrew in that case). Your scripts may also fail if they use external tools not available on some other system, or features of tools that may not be implemented, or that are implemented differently.

The most portable shell language on POSIX systems is the sh shell. This is described here: https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html

The bash shell (and others) implement sh, with extensions. Some shells, like fish or zsh, are not even trying to be sh shells at all (but zsh provides a degree of "sh-emulation" via --emulate sh), while shells like dash does try to be more pure to the POSIX sh standard (but still with a few extensions).

In general, shell aspects that relate to syntax and built in features should be "portable" as long as the script is always executed by the same shell (and version of the shell, to some degree), even if that shell happens to be the fish shell. This is the same type of "portability" that you have with Python, Perl and Ruby scripts. There should however always be an sh shell available (usually installed as /bin/sh), which is usually bash, dash, or ksh running in some form of "compatibility mode".

The most portable utilities on a POSIX system are the POSIX utilities. These are described here: https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

GNU utilities, the ones in coreutils, implement the POSIX utilities (which, as user schily points out in comments, does not mean that they are POSIX-compliant), and then expands upon them with features that are mainly for convenience. This is also true for the corresponding utilities on other Unix systems. Also note that the GNU coreutils package on Linux does not contain all utilities that POSIX specifies and that utilities such as find, sed, awk, and others, which are all POSIX utilities, are packaged separately.

As long as you stay with utilities that are POSIX and use their POSIX behaviour (in terms of options etc.), in scripts that use POSIX sh syntax, you are likely to be "mostly portable" across most Unix systems. But also note that there are non-POSIX utilities and non-POSIX extension to POSIX utilities that are still fairly portable, because they are commonly implemented. Examples of common non-POSIX utilities are pkill, tar, and gzip. Examples of non-POSIX extensions to POSIX utilities that are common to find is the -iname predicate of find and the fact that sed often can be made to understand extended regular expressions using -E.

You'll learn what extensions in implementations of what tools, running on what Unices, behaves in what way, by testing by yourself (in virtual machines for example), and by reading questions and answers on this site. For example, How can I achieve portability with sed -i (in-place editing)?

Also note that there are instances when the POSIX standard leaves a behaviour "unspecified", which mean that the same utility on two different Unix systems, using a POSIX option, may behave differently. An example of this is found here in Why permission denied upon symbolic link update to new target with permissions OK?

The disadvantage of trying to write pure POSIX shell code is that you miss out on some really useful features. I once tried to write the POSIX equivalent tests for find that GNU's -readable predicate tests for, which wasn't easy, and trying to signal a process by name using ps and grep instead of pkill is a topic of a few too many questions on this site. Likewise, trying to "stay POSIX", you would need to stay away from really useful things like perl, and you would not be able to do much of any administrative task portably (adding users, manage backups, transfer files between systems, etc.)

In general I prefer a pragmatic approach to getting things done, while understanding that my code may have to be modified to support other systems later (if it's ever used on other systems), rather than a purist approach. This does not stop me from writing scripts that I know will have as few portability issues as possible (I wouldn't use unportable semantics everywhere). But that's my personal opinion.

Kusalananda
  • 333,661
  • What a great answer.. but it's depressing to read the last paragraph. Isn't there a way to get around it? – Student Dec 01 '19 at 22:50
  • @Student Not really. If a standard leaves a behaviour unspecified, then an implementation of the standard has to make a call about what to do in that situation. Two implementations may choose to do things differently, and if you rely an one behaviour, you may just have to be aware that on another system, the same command may fail in a particular way, and program defensively around that. – Kusalananda Dec 01 '19 at 22:54
  • 1
    The GNU utilities are not POSIX compliant because they implement intentional deviations without good reason. The reason why there are more and more "unspecified behavior" notes in the standard is mainly a result of the deviations in the GNU utilities. – schily Dec 01 '19 at 22:57
  • 2
    @schily Hi. I never said they were POSIX compliant. I said they implement the POSIX utilities. How well they adhere to the actual standard, I can't really say much about since I'm not a Linux user to any greater extent. I will however rephrase that sentence a bit to make it clear that it's not an endorsement. – Kusalananda Dec 01 '19 at 23:14
  • A rewording would be nice.I guess that most people believe that implementing a utility mentioned in POSIX means this reults in a POSIX compliant implementation. – schily Dec 01 '19 at 23:23
  • @Kusalananda I get that. But that also means there's no one single "universal" basic set of things that I can simply use without paying attention to nuances.. I'm very curious, then, that do people check compatibility, and include the information with the code? I don't think I have seen that often. How then would people know that nothing is going wrong? Is there a standard way to do it, or to make claim that which specific versions are called? – Student Dec 02 '19 at 01:33
  • 1
    @Student There is no such thing. You can't program without paying attention to details, in any language. Write a piece of code for Linux, put it up on Github, get people to use it, and eventually someone will find incompatibilities and bugs. Then you fix those issues. That's the usual workflow. You could feel safer by just testing your software, obviously, on different Unices, under different conditions, etc. – Kusalananda Dec 02 '19 at 06:35
2

The point of POSIX is to standardize these things, so

  1. bash is fine, but use the POSIX subset.
  2. You lose some efficiency
  3. Read the rest of the POSIX specs, hang around here, and other places such as Usenet
icarus
  • 17,920
2

If you want to write portable scripts on GNU/Linux systems, a good bet is to use the dash shell. This is very portable shell which provides few or no extra capabilities beyond standard. This is the standard /bin/sh for Debian and Ubuntu systems. Red Hat and CentOS use bash as their /bin/sh so they aren't good for testing portability.

MadScientist
  • 3,108
  • Which one is more standard? dash or "normal" /bin/sh? – Student Dec 02 '19 at 00:47
  • 3
    There is no "normal" /bin/sh. Every system has a shell it calls /bin/sh. Some of them few or no features beyond standard POSIX, some of them support many more features in addition to POSIX. RedHat and CentOS (and Fedora etc.) simply link /bin/sh to /bin/bash so they're the same shell. Debian and Ubuntu (and their derivatives) have /bin/bash for users but install dash as /bin/sh instead of linking it to bash. – MadScientist Dec 02 '19 at 04:26
  • 2
    You've only really mentioned one side of the shell portability issue here. Take any of the /bin/sh shell scripts on Debian or Ubuntu, and try running them on OpenBSD or Solaris. You will find that they most likely fail because even though they are written for sh, they may well use external tools in non-portable ways. In the end, the actual implementation of the shell you use (dash, bash, ksh, zsh) is less important than what utilities you use and how you use them. – Kusalananda Dec 02 '19 at 06:30
  • As a contractor, I work on various client sites, many of whom have locked-down installations for security reasons. It is not much use being a dash guru, or bringing a bunch of great dash scripts with you, if they don't have dash and won't install it for you. – Paul_Pedant Dec 02 '19 at 11:09
  • I guess I've still not made myself clear. What I'm trying to say is that dash is an implementation of POSIX standard sh with no extra non-standard stuff. So if you write your shell scripts so they run in dash, then they should run in any shell that provides compatibility with POSIX sh. That includes bash, ksh, etc. So if your script works in dash you shouldn't need your client to install any particular shell: it will work for any /bin/sh they do have installed. – MadScientist Jun 19 '20 at 19:17
2

Your question seems to be wrong.

The right question would be: "What to do, to end up in portable scripts?"

One main portability issue is programs that implement enhancements and users that use these enhancements, since the enhancements are most likely missing on a different UNIX platform.

Here is a list of frequent causes for portability problems:

  • GNU programs advertise e.g. GNU style long options like --long that are 100% non portable. Never use them in portable scripts.

  • bash implements ksh-style enhancements like [[ -z $HOME ]]. This is a feature that is not in POSIX and only ksh88, ksh93, bash, mksh implement it.

  • bash (unless compiled with the compilation options used by Mac OS and Solaris) implements a non-POSIX echo builtin that is in conflict with the UNIX behavior. Be careful, but also note that:

  • printf is also not a final solution as there are printf implementations that do not deal correctly with nul bytes in string parameters and as different printf implementations have a different feature set.

  • Signal numbers besides 1, 2, 3, 6, 9, 14, and 15 are non-portable. The related signals have different name/number mappings on different UNIX versions.

  • POSIX shell math like echo $(( a + b )) may cause portability issues as different shells come with a different set of supported features. Note that POSIX only requires a limited set of feature.

  • The POSIX shell is typically not in /bin/sh as POSIX does not deal with path names except that is requires dev/tty to be present. If you like to start a POSIX shell, call getconf PATH and use the result as the PATH for your shell, then type sh and you get a POSIX shell.

  • Even all the other utilities that people expect under /bin or /usr/bin typically are not the POSIX variants of the utilities. You need to call:

    PATH=`getconf PATH`
export PATH
sh

    to get a POSIX shell and this POSIX shell calls the POSIX variants of the utilities.

Many people recommend to use dash for tests, but this is a problem because dash is not POSIX compliant since it does not implement support for multi byte characters.

I recommend to use pbosh which is a compilation variant of bosh. pbosh implements minimal POSIX features that are switched on by default.

schily
  • 19,173
  • Thanks for a list of things to pay attention. People are right, I'm looking for a way to write portable shell scripts. More importantly, I was wishing to write the most portable shell scripts. Now I learned there's no such way -- this is already a sufficient answer since there are lots of resources out there about writing portable scripts. – Student Dec 02 '19 at 16:29