45

Scenario: In version controlled system configuration based on Puppet, Chef etc., it is required to reproduce a certain system state. This is done by explicitly specifying system package versions.

Recently we ran into a problem where certain package versions were missing in the Debian repositories. One example: The "patch" package was required in version 2.7.5-1+deb9u1, but only 2.7.5-1+deb9u2 was available. Another, even more severe example: "linux-headers-4.9.0-9-common" is required (due to the associated kernel being installed) and only "linux-headers-4.9.0-11-common" is available.

This makes it impossible to reproduce a certain state of a system.

The above packages are just examples (which I in fact encountered). I am interested in understanding and solving the general problem.

What is the idea behind these updates, 'vanishing' packages and package versions?

Where can I get previous versions (not really old versions, but versions that are a couple of weeks old) of Debian packages? It should be possible to automate the installation process in general way.

Braiam
  • 35,991
Flo
  • 593
  • 1
    Somewhat depends on the software used to configure the repository. Reprepro, iirc, only allows a single version of each package – muru Sep 30 '19 at 10:35
  • 2
    stable remains consistent, at least until the next point release. stable-updates, testing, and unstable only contain the latest version of any given package. For anything else, you'll have to look on http://archive.debian.org/ (or https://snapshot.debian.org/ as mentioned in SK's answer) – cas Sep 30 '19 at 10:46
  • 5
    Is there a reason you aren't running your own repo on which you can control replacement policy and pin versions (which will have the advantage of making future automatic installs local)? – Eric Towers Oct 01 '19 at 02:00
  • 2
    The new linux pkg name is an exception: in general, Debian stable's packages go by the same package name and only change version number. linux-image-amd64 never changes name and always depends on the latest linux-image-4.9.0-*. The new linux-image-4.9.0-* pkg name marks the incompatible kernel ABI changes required to backport some bugfixes and allows for dealing with the necessary recompilation of custom built modules (dkms, etc). Similarly for linux-headers-*. – ignis Oct 01 '19 at 20:22
  • 1
    What is the idea behind these updates apt-get changelog packagename – ignis Oct 01 '19 at 20:34
  • I even found the case where a dependency package was no longer available. Turned out it was removed for being broken and we thought we were using the system package but actually it used a bundled one, so it was just a matter of removing it from the list of packages that needed to be installed. Still, it was "fun" to see that a previously working recipe now failed, due to a Debian package no longer existing. – Ángel Oct 03 '19 at 00:13
  • 1
    If you want to be able to replicate a scenario forward in time you need a local copy of all the dependencies so you can control them. – Thorbjørn Ravn Andersen Oct 03 '19 at 05:17
  • 1
    If you rely on specific packages, you should keep them yourself. apt-mirror might be something to look into – Robert Riedl Oct 03 '19 at 08:49

3 Answers3

74

Being able to reproduce a specific setup, down to the exact version, is your requirement, not Debian’s.

Debian only supports a single version of each binary package in any given release; the counterpart of that is that great care is taken to ensure that package updates in any given release don’t introduce regressions, and when such care isn’t possible, to document that fact. Keeping multiple versions of a given package would only increase the support burden and the test requirements: for example, package maintainers would have to test updated packages against all available versions of the libraries they use, instead of only the currently-supported versions... Packages are only updated in a stable release when really necessary, i.e. to fix a serious bug (including security issues). In the kernel’s case, this sometimes means that the kernel ABI changes, and the package name changes as a result of that (to force rebuilds of dependent packages); there are meta-packages which you can pull in instead of hard-coding the ABI (linux-image-amd64, linux-headers-amd64, etc.).

There is however a workaround for your situation: every published source and binary package is archived on snapshot.debian.org. When you create a versioned setup, you can pick the corresponding snapshot (for example, one of the September 2019 snapshots) and use that as your repository URL:

deb https://snapshot.debian.org/archive/debian/20190930T084755Z/ buster main

If you end up relying on this, please use a caching mirror of some sort, for example Apt-Cacher NG. This will not only reduce the load on the snapshot server, it will ensure that you have a local copy of all the packages you need.

(The situation with regards to source packages is slightly more complex, and the archives do carry multiple versions of some source packages in a given release, because of licensing dependencies. But that’s not relevant here. Strictly speaking, Debian does provide multiple versions of some binaries in supported releases: the current version in the current point release, along with any updates in the security repositories and update repositories; the latter are folded in at the next point release. So maintaining a reproducible, version-controlled system configuration is feasible without resorting to snapshots, as long as you update it every time a point release is made.)

Stephen Kitt
  • 434,908
  • Note that sometimes some prior versions are available - apt-cache madison packagename will show all versions that apt can see via configured repositories. – ivanivan Sep 30 '19 at 14:10
  • 6
    (Please don't overload the snapshot / archive servers by using them unnecessarily, instead of a nearby mirror. So leave your normal mirror in your sources.list with higher priority than snapshot, so packages still available from it it can be fetched that way.) – Peter Cordes Oct 01 '19 at 16:35
  • 3
    Just to verify that I have understood correctly: In practice, it is not about the versions I specify when installing packages (because older versions may not be available), but rather the state of the package repository(ies) I'am using. So, if I want a 'fresh' Debian 9.8 I need the package repository in that very state (e.g. a snapshot or a repository that I created myself), and then, of course, the correct linux-header-* package keeps being available. If I want to migrate to Debian 9.9, I get the package repository to the associated state and run apt-get dist-upgrade. Is this correct? – Flo Oct 02 '19 at 06:02
  • 2
    Yes, that’s correct. – Stephen Kitt Oct 02 '19 at 06:11
  • 3
    @Flo note, that point releases (like debian X.9, or X.8) are just meant for downloadable iso's, so new installations don't download tons of packages. In the repositories, there isn't a distinction between any of the point release, you always get the latest package. – Braiam Oct 02 '19 at 17:08
19

Don't rely on servers not under your control to reproduce a specific system state. Even thought the Debian servers are pretty reliable, you never know what could happen in the future. This is especially relevant with other repositories, you might use.

You should maintain your own mirror to get reproduce able system states. This way you can even have a production state for your normal systems and several testing states for new configurations.

The repository management tool aptly is able to create mirrors of repositories. You can choose the packages to mirror, create snapshots of repository contents at a specific points in time and combine several mirrors or snapshots into one repository. This way you can have completely reproduce able system states.

Charleston
  • 191
8

While Stephen Kitt's answer certainly is one possible solution, I think it would be safer for you to keep your own copies of the needed packages.

When recording a system setup, make sure to save copies of the .deb-files from /var/cache/apt/archives/. You can also use apt-get download.

When restoring a system setup, you have to be very strict with apt to avoid triggering potentially dangerous automatic actions.

It will probably be easier to use dpkg directly to install exactly what you want.

Stig Hemmer
  • 795
  • 6
    An even better approach IMO would be to use a local APT cache. That avoids the problems in your third paragraph, and also avoids having to harvest /var/cache/apt. – Stephen Kitt Oct 01 '19 at 13:03
  • 4
    There are two variants of this - one is to use something like apt-mirror to clone a whole repo, the other is to download only specific packages and dependencies. Then snapshot the dir - e.g. with btrfs as pkgs-20190501, then publish the snapshot dir as a repo. At build time, put the versioned repo url (e.g. http://debmirror/pkgs-20190501/...) in sources.list, then run apt-get update, apt-get install $pkgs, etc. – bain Oct 01 '19 at 15:27