21

I thought GNU was the (only) supplier of essential tools whose open-source code powers Linux distributions. However while browsing unix.se, I saw some questions that explicitly state they are dealing with GNU, such as a question on the safety of parsing GNU ls. This must imply that there are versions of ls that are not provided by GNU, right? What are these versions, who make them and where can I find them?

eyhy84394
  • 235
  • 3
    Are you expecting answers only covering Linux, or would you also welcome answers mentioning other open source Unix system, and maybe even current commercial Unix systems? I'm thinking of the BSDs, including macOS, and others. – Kusalananda Aug 26 '23 at 14:41
  • 3
    Note that your logic is faltering a bit here: for one, you talk about tools on Linux distributions; but on the other hand, you mention posts on a site about Unix-like systems in general (not just about Linux). The wider scope is already reason enough for the explicit mention. It would be different on a site specifically about, say, Ubuntu, since that's a Linux distribution with the GNU toolset. – ilkkachu Aug 26 '23 at 15:50
  • Note: also with GNU, you may have problems. Not all version of GNU handle the same parameters (some are "recent additions"). Also GNU tools on Windows may have disabled functionality (which cannot be provided by Windows). But also you may want to compile a restricted version of GNU tools (e.g. small and static for emergencies). BSD have own. Many people coded own versions. Macos may use GNU or BSD or own tools. beOS has own. If you want to be compatible, look for POSIX, and use only such subset (or an older subset). – Giacomo Catenazzi Aug 28 '23 at 12:19
  • exa is a drop-in replacement for ls with expanded capabilities – ScottishTapWater Aug 29 '23 at 09:12
  • 1
    exa own web site says it's not a drop-in replacement for ls. – Marc Wilson Aug 29 '23 at 14:42
  • The title is misleading - ls is a general utility, not specific to terminals. That said stty, which does manipulate a terminal, is part of GNU coreutils. So accidentally relevant, perhaps. – Toby Speight Aug 29 '23 at 15:55

4 Answers4

41

Taking a more direct approach, here are the major options other than the GNU utilities:

  • The BSD userland utilities:Originated with the early versions of Berkely Unix, in which they were just regular Unix tools with some extra features, by the time of SVR4 they had diverged significantly. Used today in various forms by FreeBSD, OpenBSD, NetBSD, DragonflyBSD, MINIX 3 (uses the NetBSD userland), and macOS (uses the FreeBSD userland, with some modifications), as well as by the system software on the PS3 (a complex derivative of FreeBSD and NetBSD), PS4 (a derivative of FreeBSD), and possibly the PS5 and PS Vita, and by the rather interesting Chimera Linux.
  • Busybox: Designed to be minimalistic and mostly POSIX compliant. Used by Alpine Linux by default (and thus in a huge number of Docker images, since Alpine is a popular base for them), and used as the exclusive userland by many embedded systems that run Linux, including a vast majority of networking appliances that use Linux as their OS. By sheer number of installs, this is one of the top few userlands on UNIX-like systems, mostly due to the embedded usage.
  • Toybox: Similar concept to Busybox, but even more minimalistic (and more POSIX compliant in a couple of places). Originated as a result of licensing disagreements regarding busybox. This is what Android (which is just Linux under the hood) uses, but I’m not aware of any other Linux platforms that use it by default. This is one of the other top userlands on UNIX-like systems because Android is everywhere.
  • The Heirloom project: Minimally modernized copies of the original UNIX userland, mostly based on source code released by Caldera and Sun. These are largely SVR4 derived, and are interesting in that many commands provide both a ‘classic’ version and a POSIX version. Not used by any major platform that I know of.
  • 9base: A minimally ported copy of the userland utilities provided by Plan 9 from Bell Labs. Not used by any major platform that I know of.
  • sbase and ubase: An even smaller, and in some cases more minimalistic but often more POSIX-compliant alternative to busybox. Not used by any major platform that I know of.
Austin Hemmelgarn
  • 11,867
  • You could also mention ast-open with utilities sometimes more feature-full than their GNU equivalent (can also be made builtin in ksh93 which shared the same maintainers). Some commercial Unices still use code bases from AT&T (SVR4 or sometimes earlier like HP/UX), not using the rewrites from BSDs. – Stéphane Chazelas Aug 27 '23 at 17:32
  • @StéphaneChazelas I actually did not know about ast-open, and as far as commercial Unices, I will admit I neglected to mention them, although not out of any malicious intent (I think I’ve only seen a question about a commercial UNIX variant once on this site in the past six years, and I thankfully don’t have to work with them myself, so I often forget about them). – Austin Hemmelgarn Aug 28 '23 at 00:09
  • 2
    Questions about HP/UX are rare, but we do see a few questions a month about AIX or Solaris. There are some opensource derivatives of Solaris still maintained such as Illumos (which does use ksh93, ast's shell as its sh like Solaris 11 now does). – Stéphane Chazelas Aug 28 '23 at 05:22
  • 1
    schily-tools and its star which has introduced several extensions to the tar format in use in other implementations including GNU tar, bosh, a mostly POSIX compliant shell derived from the Bourne shell, sfind (also the builtin find of bosh). – Stéphane Chazelas Aug 28 '23 at 05:26
  • 1
    There are also a lot of standalone software used as alternatives to GNU software for core POSIX utilities, even by default on some GNU system, like Ubuntu where the default sh is dash (derived from NetBSD sh, derived from ash) and the default awk is mawk. There are POSIX utilities that GNU doesn't supply or at least not for Linux generally (ps, ex, vi, login, at, crontab, lp, logger...) – Stéphane Chazelas Aug 28 '23 at 05:32
29

I thought GNU was the (only) supplier of essential tools whose open-source code powers Linux distributions.

No. Not at all. Stallman & co do like to paint that picture as if it's "GNU/Linux plus a sprinkle of non-GNU optional software", but that's really not the case.

This must imply that there are versions of ls that are not provided by GNU, right?

Hm, the mathematical implication is not there, in the strict sense, but yeah, you're right.

First of: all the non-Linux systems do have portable userlands, too, right? FreeBSD, OpenBSD, DragonflyBSD, Minix, MacOS… they all don't ship GNU tools. So, there has to be another dynasty of tools, at least one.

So, yeah, there's the BSD tools (which of course have various forks). Some are cooler than their GNU counterparts, some worse, some just different. I'm not aware of a Linux distro that completely relies on libbsd/BSD tools where others have gnulib/GNU coreutils, but at least I prefer bsdtar, prefer a different shell than GNU Bash… you get the idea.

Then, by far the most actual physical devices that run Linux will not have GNU userlands:

  • There's more than 3.6 billion active Android devices out there. Android is a Linux distribution, which uses toybox (0-BSD licensed) as collection of standard tools, and a modified Korn shell.
  • There's a couple of billion DSL/cable/fiber "router/access point" boxes that run some Linux distribution. The vast majority of them are running with Busybox (GPL), and use the sh that comes with that.

Compared to what I would guess are around 6 billion Linux devices with a non-GNU set of base tools, I'd argue that the couple million desktop/laptop Linuxes aren't really that prevalent.

But by far, the most instances of Linux kernels running are probably running in the data centers of the Hyperscaler cloud providers (Amazon, Alibaba, Microsoft, Oracle…). It's hard to put number to that, but it's going to be billions. What these are running – hard to guess. But especially for cloud computing, distros like Alpine Linux are popular, and that doesn't ship with GNU Coreutils, but with Busybox. But here we're counting virtual machines, not physical machines. As I'd argue that in a Google datacenter, a "device" is a server rack, and not a single motherboard, the number of physical devices is comparatively small.

(Personal view on this: it's not great that there's less and less percentage of strong copyleft userlands, especially because vendors really haven't stopped infringing the kernel's strong copyleft requirements, and not having an FSF-stewarded code in there isn't great for discovery of that. However, FSF's GPL enforcement has not worked, targeted and worked out like I would have wished for.)

Marcus Müller
  • 28,836
  • 8
    It's hard to object on the "far more non-GNU Linux-systems" argument based on the numbers themselves, but I can't help thinking it's not as relevant as the numbers would imply, as it's more a question of systems where the CLI tools are regularly used, rather than just the raw count of systems. I'd guess the number of Android systems or network boxes where the CLI is used interactively even once is far, far lower than those billions. It might be more representative to count the number of developers of such systems. (Same for desktop systems, but I suppose the ratios are quite different.) – ilkkachu Aug 26 '23 at 15:57
  • 3
    I think that argument would be more than fair! But while ls certainly (?) is a purely interactive command, cat, cp, stat might be the underpinning of most of these system's operation under the hood. So, really, question is whether we're asking ourselves "if I sit in front of an arbitrary Linux console that was meant for me, how much GNU do I expect?" or "if I look at how an arbitrary Linux device boots, handles log files etc., how much GNU do I expect?". – Marcus Müller Aug 26 '23 at 16:36
  • There's also the case to be made that still, the vast majority of these devices is still running a kernel which could only be compiled through GCC; Google started to build Android production kernels only in 2020, IIRC, with clang, and because I guess there's more android phones predating that than newer ones, that's not a even a close call. So, there is some GNU in most Linux devices, alright, but not necessarily in the way that its used everyday as coreutils. – Marcus Müller Aug 26 '23 at 16:39
  • 1
    re: * paint that picture as if it's "GNU/Linux plus a sprinkle of non-GNU optional software"* - that's not what stallman argued. His point was that GNU had developed a complete clone of unix userland (all the standard tools and compiler etc), lacking only a fully working kernel - GNU Hurd wasn't in a working state in 1991. Linux provided that kernel and, for most people, was good enough that Hurd was mostly irrelevant. – cas Aug 27 '23 at 00:45
  • 1
    Stallman's point was that what most people call a "Linux system" was actually a combination of the Linux kernel plus the GNU userland, which they'd spend 10 years working on - and, without which, the linux kernel wouldn't even exist (since it needed GNU gcc, ldd, libc, and other GNU development tools). A kernel is an essential piece of a unix system, but it's not the only important piece. – cas Aug 27 '23 at 00:50
  • 1
    BTW, an android phone may be running the Linux kernel, but it's not what most people would call a "unix system", it's more of an appliance to make phone calls and (most importantly!) deliver google ads and spy on the owner and allow them to run any of billions of "apps", of which 99+% are garbage. It's not until the user installs Termux or something to provide a full unix (GNU) userland that it qualifies as a unix system – cas Aug 27 '23 at 00:51
  • 1
    Well, that is a bit of an opinion, isn't it? The question was about typical core utilities on Linuxes, not specifically these typically used in a POSIX/Unix context. I do agree, from an application point of view, you usually don't interact with whatever is left of the unixoid API of that system. But then again, when people write GUI software, that's also the case when they target say Slackware Linux. They don't go in and think "oh how nice this shared memory is for taking to my window compositor! How awesome these thread synchronization mechanisms"; they are more like "what does Qt offer?" – Marcus Müller Aug 27 '23 at 01:10
  • 1
    Point being, from a practical point of view, the same programmers of the useful Android apps would not target desktop Linux in a posixy way either. It's not really fair to say one is more of a Linux than the other, just because on the latter you don't use the Unix facilities as much! – Marcus Müller Aug 27 '23 at 01:13
  • 1
    Nitpick: As can be seen from the Borg paper, we at Google do not refer to entire racks as "devices." Instead, the accepted term is "machine," and it generally refers to one motherboard, not an entire rack. For example, in this quote: "Borg [...] reduces correlated failures by spreading tasks of a job across failure domains such as machines, racks, and power domains" - the terms "machine" and "rack" are considered distinct failure domains. – Kevin Aug 27 '23 at 05:07
  • 1
    @Kevin interesting! So that's the workload perspective on that. I'd argue "failure domain" might not be optimal for definition of "single device", and "is supervised by the same rack controller" is more appropriate. But that's just me putting a counter-definition to your definition... – Marcus Müller Aug 27 '23 at 09:05
  • 1
    @MarcusMüller: Borg schedules jobs onto single machines, not onto racks. When I have been oncall for my service, I've had situations where I had to tell Borg not to schedule onto a single machine (because it appeared to be having a hardware problem), but it could still schedule onto other machines on the same rack. I'm not sure where this idea of a "rack controller" comes from - we have a per-machine "borglet" process, but I've never heard of a "rack controller," and I've been scheduling things on Borg for years. – Kevin Aug 27 '23 at 18:50
  • @Kevin I remember talking to one of the guys working on EC firmware for these racks, and well, yes, there's rack controllers :) It's a good thing that someone using the compute in a rack doesn't notice. But honestly, my memory might be betraying me there, it might only apply to specific nodes,… Not a Googler myself. – Marcus Müller Aug 27 '23 at 18:53
  • 2
    In my experience, Alpine is quite popular for containers, but for virtual machines in the hyperscaler Cloud, traditional distros like Ubuntu and Red Hat seems to still be the most popular. The main reason for this is that Canonical and Red Hat are really the only providers of commercial Linux support which has the scale and reach to support large enterprises, and if you're running something on VM, saving a few hundred megabytes is basically just a rounding error. – Lie Ryan Aug 28 '23 at 02:01
16

GNU Coreutils in general are not the original version of most of those tools. Typically they are supplied by the operating system and since most flavors of Linux are GNU/Linux you are most familiar with GNU Coreutils. However prior to Linux even being a thing there were BSD tools and various flavors of UNIX tools that mostly still exist today on the various flavors of unix still in use. Also some versions of Linux are not GNU and may use something like Busybox (such as Alpine Linux).

Regardless of who makes them it's important they follow the POSIX standard(s), which is not always the case especially in regards to GNU utilities.

You can find OpenBSD, FreeBSD, UNIX, GNU, Busybox, etc. versions of pretty much any command.

Many of the UNICES in use originated in at least some way from UNIX version 6 so they may share some identical utilities, or they may have modified those utilities uniquely along the way.

jesse_b
  • 37,005
5

This must imply that there are versions of ls that are not provided by GNU, right? What are these versions, who make them and where can I find them?

Yes. The original implementation of ls under the old AT&T Unix license.

As for modern implementations of ls, all the ones distributed by the various BSDs (which, unlike Linux, were actually derived from the original AT&T Unix).

You may think that BSD is a very minority OS with very few users but they exist. And also there is a distro of BSD that has a very large number of users: Apple's Mac OS.

It is worth remembering what the GNU project is. GNU is GNU's Not Unix. It is a project to make clean copies of Unix tools (such as ls) but is definitely not Unix. As such you can think of the GNU ls as a clone/copy of Unix's original ls. In reality features of GNU tools have inspired other developers to update their tools with the GNU features. Over the years there have been a lot of cross-pollination between GNU and the rest of the Unix world.

slebetman
  • 687
  • 1
    You'll more likely find original AT&T code in HP/UX or SysV derived commercial Unices rather than BSD where most of it was rewritten for license reason (around the same time as a lot of the GNU project was written). On the FLOSS side, you'll find some in derivatives of OpenSolaris such as Illumos or bits in ast-open, ksh93, heirloom toolchest, schily-tools. In any case, saying GNU ls is a clone of BSD ls is far too much of a shortcut. – Stéphane Chazelas Aug 28 '23 at 05:13
  • @StéphaneChazelas Reworded the last paragraph. While a lot of its parts have been replaced BSD is still a descendent of Unix rather than a copy. – slebetman Aug 28 '23 at 05:22