4

I did an update of CentOS 6 last night which was rather large. By the time I decided to leave it to finish, the process was only a few packages away from completing the download. So, I scheduled a shutdown for +60 minutes and left.

When I checked in on the system this morning it had shutdown. I'm having some troubles (I expected something), only, while I was looking at logs, I noticed the end of the yum log seems like the update took longer then I expected.

When I noticed this, I reran yum update. Everything is up to date. Another question is,

Did it finish cleanly. How can I check?

xtian
  • 583
  • 1
    I had a similar problem. Please read my question and answer(s) in: http://unix.stackexchange.com/questions/245687/is-it-safe-to-interrupt-an-upgrade-process-from-fedora-22-to-23-using-dnf – polemon Aug 09 '16 at 01:21

3 Answers3

3

Occasionally it will break your machine. It's been a known problem:

and is supposed to have been fixed by a change to systemd. According to

CentOS 6 isn't running systemd. So the problem may exist in your machine.

Community
  • 1
Thomas Dickey
  • 76,765
2

Ideally, updates are idempotent (if you run the same update multiple times, you get the same result as if you'd run it once) and resilient (if an update is interrupted by a reboot, you can resume the udpate after the reboot). But updates are not atomic (there are half-updated system states).

In practice, idempotence really works. If yum update is interrupted (by an error, by pressing Ctrl+C, by a scheduled reboot, by a power failure, …), just run yum update again after the system restarts, and wait for it to complete.

In practice, resilience usually works. The package manager takes care to first unpack the files of the new version under a temporary name, then replace the old files with the new one. If the update is interrupted during the longest phase (the unpacking, which requires a lot of disk writes), the system state hasn't change meaningfully, so no harm is done. If the update is interrupted during the replacement phase, things get more complicated. The software that is being updated may be unusable due to a version mismatch between its different files.

Most critical packages can be updated on a file-by-file basis, or have configuration steps that are performed in the appropriate order to ensure that the system remains operational. For example rpm consists of a single binary, so you either get the old one or the new one. Same thing with the kernel. However, with thousands of packages, hundreds of thousands of files, and a large potential for write reordering at the filesystem level, the combinatorics is very, very high, and it's impossible to test everything. So it's possible to end up with a system that doesn't boot cleanly if it was interrupted in the middle of an update. But the probability is very small (barring bugs, of course).

Gilles 'SO- stop being evil'
  • 829,060
  • It seems like it would be useful for the package manager to have an option which pauses the update process after unpacking and notifies the user that the next replacing existing files with temp files stage should not be interrupted, - that way you know when you need to babysit the upgrade a bit more, and conversely you can just download the package archives on a more adhoc basis, e.g. when bandwidth is available. – the_velour_fog Aug 09 '16 at 00:38
  • @the_velour_fog That wouldn't really work in practice: that stage happens for each package, and lasts for a very short time. A typical update might concern hundreds of packages, which would mean hundreds of notifications. Because of dependencies, and of disk space considerations, it isn't possible to group all unpacking and then all replacements. – Gilles 'SO- stop being evil' Aug 09 '16 at 00:48
1

On CentOS the file /var/log/yum.log shall have the time the last package was upgraded. For example on my CentOS (mine is currently running CentOS 7, but the file is in the same place):

# tail -n 3 /var/log/yum.log
Jun 28 17:10:04 Updated: 1:mariadb-libs-5.5.47-1.el7_2.x86_64
Jun 28 17:10:05 Updated: graphite2-1.3.6-1.el7_2.x86_64
Jun 28 17:10:05 Updated: zsh-5.0.2-14.el7_2.2.x86_64

(Yes, I did not run yum update for some time)

You can then compare this against syslog messages about the shutdown. Syslog rotates the logs, so you may have something like messages-20160807 instead of plain messages, check that too. For example:

# egrep -i 'stopped|shut' /var/log/messages*
/var/log/messages-20160731:Jul 24 23:55:55 orion systemd: Stopped target Multi-User System.
/var/log/messages-20160731:Jul 24 23:55:55 orion systemd: Stopped Resets System Activity Logs.
/var/log/messages-20160731:Jul 24 23:55:57 orion systemd: Stopped Permit User Sessions.
/var/log/messages-20160731:Jul 24 23:55:57 orion systemd: Stopped target Remote File Systems.
/var/log/messages-20160731:Jul 24 23:56:33 orion systemd[1]: Listening on Delayed Shutdown Socket.
/var/log/messages-20160731:Jul 24 23:56:33 orion systemd[1]: Starting Delayed Shutdown Socket.

(CentOS 6 will not have systemd messages, but something relevant will be printed.)

Depending on the configuration in /etc/rsyslog.conf//etc/syslog.conf shutdown may not be logged, but something certainly logs when the system goes down. Therefore searching for stopped|shut is a good heuristic to check when things started going down.

Comparing the timestamps in both logs you can get a good notion of whether yum finished its work before the system went down.

grochmal
  • 8,657
  • That was good, and I think I'm OK. The last yum activity was to erase a package at 0221h and the last messages log item with the 'stopped|shut' keywords 0257h; +30 minutes. – xtian Aug 20 '16 at 22:26
  • @xtian - 30 minutes is plenty of time, you're definitely alright. – grochmal Aug 20 '16 at 23:26