2

I wanted to write a small shell script for Linux and other UNIX systems (probably OSX). And also, why not keep things platform-independent out of principle, right?

So I went with a #!/bin/sh shebang and one part of my script is supposed to compress a string so I thought I'd just have a look which is the POSIX default utility to compress strings or streams. Unsurprisingly, neither tar or gzip were listed at https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html. But what I did not expect: the listed compress was not installed on my Debian 11 Bullseye system!

So I cannot even rely on POSIX commands being available on all so called POSIX-compliant systems? Did I make a mistake somewhere?

finefoot
  • 3,060
  • While Linux may be (almost 100%) POSIX compliant any distributor can decide what package to install by default (and so can you during installation). Why not install compress in your nice Debian system using its package manager? – Ned64 Sep 15 '21 at 23:06
  • 1
    Why are most Linux distributions not POSIX-compliant? You will find some interesting facts there. – Eduardo Trápani Sep 15 '21 at 23:13
  • Make your script larger, then do testing for presence or absence of tools, then run the commands based on what tools are there. That's how I do it, it's impossible to do anything of any complexity without doing that in my experience. POSIX is just an idea, in the end, same as any other idea, implementations vary widely in reality. Best to focus on reality is my suggestion, and don't spend time worrying that reality isn't quite the way you might want it. – Lizardx Sep 15 '21 at 23:24
  • Most (probably all) linux distros provide gzip which can decompress LZW files (as produced by the ancient compress/ncompress). Some even symlink gzip to uncompress (and sometimes to compress too). Decompression of archaic compression formats is good enough for most purposes. For those who really need to create such files, the ncompress package is available: apt-get install ncompress – cas Sep 16 '21 at 08:42
  • BTW, many Linux users don't really care much about POSIX (if they even know it exists). They (we) think of GNU userland as the standard, and POSIX is just a barely adequate, minimally-featured subset of GNU. GNU tools can easily be installed on systems that don't already have them (and IMO should be installed), in a separate directory to avoid conflicts - I was doing that on Sun and SCO boxes back in the early 90s because the included versions were crap compared to GNU....and even early 90s GNU versions were primitive compared to today's versions. I still install GNU on Freebsd today. – cas Sep 16 '21 at 09:00
  • GNU utilities may add more features in some cases, but they frequently implement aliases for POSIX features and advertize the GNU alias in order to pull users into a vendor lock in state. In addition, there are many small bugs in GNU utilities and people who frequently use these tools, now expect the incorrect behavior seen with the GNU tools. That still does not make the GNU environment a standard, since a standard is what is agreed on by many people and not what a one man GNU program likes to have – schily Sep 16 '21 at 12:28

2 Answers2

4

I've done a lot of cross platform tools, there's really two things you want to concern yourself with:

  1. The core logic/language should be posix compliant, that works fine, avoid bashisms, zsh'isms, etc. This is where POSIX compliance matters, and that's easy to achieve. This becomes particularly important in subshell code.
  2. Use that core POSIX sh logic as wrapper to test for, detect, then use, the tools you need to carry out the actions. Go from most likely to be there, to least. I don't think I've ever spent any time or effort on theoretical 'it should be there because POSIX said so' coding, I always just test and use what's there. Sure, it adds some lines of code, but if your goal is to always work on everything, then that's how you do it.

Re the 'compress' tool, I can honestly say I'd never even heard of it before this thread, no wonder from the above comments, why should I have, when it's not as good as modern tools that are always installed, or that can be installed easily?

If you get into anything complicated, the odds of your logic working anywhere at anytime declines rapidly, unless you can make it incredibly simple at the base level. Straight sh, yes, that will always work, but once you add in tools, not so much. You can't for example even count on an 'awk' on a system being basic awk, it might be nawk or gawk that has been linked to awk.

Without testing for tools, you will be sad, and you will not succeed in creating something that is truly cross platform/cross os/cross distro.

With this said, you're still going to find systems that don't work, then you have to update the tool to make that situation work by adding more tests and fallbacks, that's just how it is. I've just added a few more for new situations I'd never seen before.

This isn't hard, unless you try to maintain that a standard that GNU/Linux for example barely follows anymore at many levels (systemd? no?, ok,then, let's move on) should do what you want to believe it should do when it doesn't, that's just going to cause you wasted time and pointless frustration.

Note that if you are trying to use nontrivial tools, this gets WAY harder, because you cannot count on any OS at all at that point, you have to test for and handle each variant, each BSD for example is proud that they don't use a distribution of packages, but an OS that includes its own core tools, which means, you can never rely on output or actions being identical across platforms even if the tools have the same name in each variant/os. ps for example is significantly different on OpenBSD, FreeBSD, Linux, and even on busybox, which you might find on wrt firmware for example.

Lizardx
  • 3,058
  • 17
  • 18
  • I can tell you've never tried to support OSX, lol. Or Android. Even supporting Linux, and the two bigger BSDS, OpenBSD and FreeBSD, is not easy if you are trying to do anything non trivial. If you add in the actual Unixes, in whatever variants they exist in today, forget it. Toss in osx and I run screaming away, lol. The bright side is you can rely on sh logic, and testing for tools. Perl is awesome for cross platform, since they do a lot of the feature testing under the covers, really a great language for that. – Lizardx Sep 15 '21 at 23:52
  • 1
    It does have to be like this, because it is like this. The reality of 'standards' was captured well here: https://xkcd.com/927/ that's not a joke, that's 100% accurate. Never seen any exception over time. The more you look under the covers, the more you find human chaos, not order. The less you've looked, the more you might believe it should be x or y. Keep in mind, POSIX was created to deal with commercial Unices being splintered, Linux was the real solution which is why it took over. Same tools in 2005 linux as today in many cases, and same across distros. Not in android, lol. – Lizardx Sep 16 '21 at 00:07
  • 1
    It's a fun game to get logic working cross os/platform, that's the actual game, not trying to follow posix or other ideas, it's very empirical. Rather than worry about standards that don't exist in the real world, have fun playing the game. – Lizardx Sep 16 '21 at 00:09
2

While Linux may be (almost 100%) POSIX compliant any distributor can decide what package to install by default (and so can you during installation). Why not install compress in your nice Debian system using its package manager?

# sudo apt install ncompress

Basically, compress compresses so badly in comparison to more modern algorithms like bzip2 and xz that no-one uses it anymore. A distributor might choose not to install the program by default.

See also this announcement about Debian's POSIX compliance.

Ned64
  • 8,726
  • 2
    Of course they don't have to include it in their default installation. But I thought one of the points of having a POSIX standard is to have a basic set of utilities available on all POSIX-compliant systems. What if I want to run this script while not being connected to the internet and I don't have the ability to install the dependencies? – finefoot Sep 15 '21 at 23:09
  • @rattlesnake Yes you could argue that. Of course one might also argue: Where are your POSIX-compliantly compressed files originate if not the Internet? Also, the tool may be on the install DVD? – Ned64 Sep 15 '21 at 23:11
  • What I meant was, if you need your system to include the compress utility to work with certain old files then perhaps you received them from the Internet. You may then also download and install the ncompress package. – Ned64 Sep 15 '21 at 23:14
  • Of course, this is a website for technical questions. If you need a more political answer like one for the question "Why does Debian do xyz?" you may want to ask them elsewhere. – Ned64 Sep 15 '21 at 23:16
  • 1
    No, no, I'm sorry for putting this misleadingly. I'm not using compress (or rather uncompress) to handle legacy-compressed files. My own script is supposed to do some compression (and decompression) at some point so I was looking for the "right" POSIX-compliant utility to do that for me. A program which I can expect to be available on every possible client machine. But that didn't work out as I expected. – finefoot Sep 15 '21 at 23:17
  • 1
    In 2021, POSIX compliance of a computer system has become far less important than, say in 1996. Perhaps this could explain why distributors do not adhere to POSIX compliance so very strictly in a system installed by default installation parameters. – Ned64 Sep 15 '21 at 23:20
  • In fact, the only systems 'certified posix compliant' are unices, and osx.https://en.wikipedia.org/wiki/POSIX. Not even the BSDs are certified. And I can assure you that Apple got this certification by making certain things work for POSIX, but if you actually try to work with osx you'll quickly find that their os follows only the letter, not the spirit, of the standard. So even that compliance etc is largely meaningless in the real world. But sh is useful. – Lizardx Sep 16 '21 at 00:22
  • MacOSX is UNIX certified, that means it passed the POSIX certification including all XSI enhancements. Apple was forced to do that to avoid being sued because they did advertise to be UNIX compliant. The certification scripts ae not yet fully complete, so it may be that some features are not tested correctly and parts of the POSIX text may be read ambiugous. The problem with Linux is that the Linux people do not like to become fully PPSIX compliant without good reason. This still applies even though POSIX recently changed several definitions to permit bugs found in GNU utilities. – schily Sep 16 '21 at 12:21