Why do different Linux distros have different package formats (and package managers)?

Question

This question is a follow-up to this one: How to install package versions not available with apt-get?

At the linked question, I learned about the Linux package "ecosystemm," including how different Linux distros use different package formats, and some actually have their users compile+install new packages directly from source code!

As a follow up to that, I'd like to ask/learn why this difference in package formats exists.

The accepted answer to the linked question says:

all the child branches [of Linux] are defined mainly by the package management system in use.

Some of the things I'd like to learn are:
Why are package formats so fundamental that they drive the creation of entire Linux distros?
Why have different package formats in the first place? I naively imagine that packages include some combination of executables, shared/static libs, headers, etc., i.e. some "collection of files" that often go into specific directories defined by the Filesystem Hierarchy Standard -- is it not the case that one package format universally solves that problem and/or will have demonstrated itself, over time, as the most superior? I.e. what are some of each distros' rationales to use .deb vs. .rpm vs. .zip, etc.?
Why are there sometimes different package-managers for the same package format? E.g., per my understanding, both Debian and Ubuntu use .deb files for their packages, but Debian uses dpkg as its package manager, while Ubuntu uses apt (and apt-get?!).

Answer 1

Package management is a very complex problem and there have been multiple attempts to solve it. Each attempt tries to improve on the previous, but none is perfect.

Each package management system uses its own format because the package management system and the format were designed together. It would be theoretically possible to have one unified format that all of the systems could use, but then other aspects of package management such as dependency graph generation, patch management, config file contents and placement, and even version compatibility between packages would also have to be unified to make packages interchangeable.

Since each package management system takes a different approach to dependency management, and each distro takes a different approach to package version selection and patch management, there is no way to unify these things.

These choices are what makes a linux distribution. To try to unify them would be to try to merge all the distros into one distro. This is why the package management system is the core of the distro -- that's where the decisions are that make the distro what is is. There's really not a better way to explain this than https://xkcd.com/927/

There have been several attempts to make a portable package format. Appimage is one of them, and it works by embedding (almost) all the dependencies into the distribution package, which makes them very large and use more memory than necessary.

An alternative is snap, which again separates the dependencies from the OS and tries to handle them itself, so that one snap may bring in other snaps that it depends on, again, duplicating OS components where versions differ.

The ultimate portable base package is the tar ball of source code files that have to be manually patched, configured, compiled, and installed. This doesn't include any dependencies, so it's also up to the installer to resolve the dependencies before compiling. This is made worse, because there's no standard way of listing dependencies, and some packages don't even bother, because there are so many assumptions about what things are available as standard on a system, all the way down to what is in libc.

There just isn't a better way to solve this. Every solution has problems. Creating a solution to fix the problems just creates different problems, and many of the solutions are mutually exclusive.

Answer 2

The simple answer is that the primary reason for the existence of a Linux distribution is to, well, distribute a coherent set of packages. And since you don't want to get stuck on the same versions for all eternity, you also need a way to upgrade. And since you don't want to have software installed you don't need, you need a way to select which packages you want. And since you cannot predict the future, you will need a way to change this selection later, i.e., install or remove packages. And since reuse is one of the basic tenets of software engineering, packages depend on other packages, so you need a way to handle those dependencies.

All this leads to the creation of a package management system.

So, if someone creates a new Linux distribution, that probably means that they think there is something wrong with every existing Linux distribution. And since package management is so fundamental to a Linux distribution, it stands to reason that there is a high likelihood that one of the things they disagree with, is the packagement management system.

This doesn't really answer the question "why do we have multiple package management systems", but rather it pushes it back to "why do we have multiple Linux distributions".

Now, you don't question why there are multiple Linux distributions. But then, you also shouldn't question why there are multiple package managers.

And, by the way, nothing here is specific to Linux. Most of the BSDs also have their own package managers. The commercial Unices have package managers.

Heck, it's not even specific to Unix. Windows alone has half a dozen different ways of installing software, some of which are arguably package managers: Microsoft created the NuGet package manager, which however is only for .NET, so the community created the Chocolatey package manager, then Microsoft created the WinGet package manager, but also there is the Microsoft Store which allows you to install and remove apps, and there is the Microsoft Installer, … Also, just for the Microsoft Installer alone, there are about a dozen different tools to create MSI packages, not just from third parties, but even Microsoft itself has multiple tools.

But there is another reason: package management, and especially dependency management is an extremely hard problem to solve, but it looks deceptively simple (and this question is a good example of that).

The fact it is extremely hard to solve means that almost everybody has first-hand experience with it going wrong. The fact that it looks deceptively simple means that almost everybody who has first-hand experience with it going wrong thinks "this is stupid, I can do better".

And, bam, now you have N+1 package management systems.

Answer 3

Compilation

Let's start here as all distributions share this in common "in the background." Each piece of software was compiled at one point or another before being chosen to be part of a distribution. As you have correctly surmised a completed compiled program can contain one or all of the following:

• A Library: These are commonly stored as Shared Object Files Or Archives. See What is the difference between .a and .so file?
• A header file. See Header Files in the GCC Documentation
• The binary, a.k.a. the executable program

Compilation to Packaging

Again, as you've noted, each distribution has packages, and each distribution has a Package Manager. I believe that where confusion might be arising is this: The Packaging Format, and the Package Contents are not the Package Management System. The Package Management System of any distribution all contain 3 distinct parts:

1. The Package Format, i.e., how the items in the Compilation Section are stored, or arranged
2. The Packaging Tool - The executable/binary that packages the compiled items according to #1. This creates the package.
3. The Package Manager - The executable/binary that repeatedly calls the tool in #2 to manipulate packages. Manipulation refers to actions, which can include:
   • Installing/Reinstalling
   • Removal
   • Updating config files
   • Updating all installed packages

Why Were Package Managers Created

In the days before package managers, it was perfectly acceptable to install packages using the packaging tool. The issue that arises when installing a package that way is this:

1. I install on Debian and start package foo.deb with dpkg -i foo.deb
2. I attempt to run foo.deb with the command foo
3. The program abruptly closes with the error message: Program foo requires bar.so; File not found, exiting
4. Well now I'm stuck, I really need to execute foo, so I repeat steps 1 - 3 for bar.deb
5. I try again and receive another error that Program bar requires static library baz.a and the program exits.

Packaging to Package Manager

Well now I'm six steps in and really angry because I really want to use foo. What if there was a program that did all that for me. That's the Package Manager's main job. The Package Manager draws a dependency graph in memory that computes that foo requires parts of bar and bar requires parts of baz. It keeps computing, working backwards until it finds a package with no dependencies, and then works forward to see if any of the needed packages are already installed. If the package is already installed with the needed configurations enabled, it's marked off the graph and the next package is searched. If the next package searched needs an update to meet the requirements, the package repo is queried and the updated version is added. Whoa, now we have to start the graph over again, or redraw a new branch, because the update we just added has requirements too.

This process repeats over and over again every time you install or update packages.

Differences in Naming

Technically there is no difference between package managers. Each manager performs the functions discussed previously admirably on their installed systems. Seeing that the FH Standard you referred to is quite flexible after the requirements are met each distribution decided to tweak something package related and they do that in their Packaging Manager. In the days when this concept came out it was mainly for brand recognition. The R in .rpm is for Red Hat and the 3-letter .deb extension is short for Debian.

Why Compiling Your own Packages Is Dangerous

In your linked question, you were looking for an updated libuv, which isn't available in the Ubuntu Repo for jammy. Because of the unavailability, compiling the new version was the only option, although it wasn't the best option:

• When compiling software, the Package Management System, cannot manage what you compiled because the Package Manager never installed the package after you compiled it. In short you now have the added headache of managing unmanaged software, aside from hosting it, and you lost the power provided by the Package Manager, albeit for 1 package.
• Make sure the compiled package is stored inside of the customer's website. Per the point above, it's always in the best interest of the host OS not to pollute it with unmanaged software.
• That means any calls to said program are local. Don't mistakenly refer to the hosts /bin when you meant to refer to the customer's /bin