9

I'm having an issue here where I try to automate a setup with Ansible.

Some of the steps require interaction with apt, but occasionally I get an error because unattended-upgrade kicked off and locked apt. This will make the playbook stop.

I've tried many ways around this, the most successful being the repetition of a failed apt command.

But this does not scale, is also not 100% reliable and feels bad.

I've opted to issue an apt -y purge unattended-upgrades right at the beginning of the playbook. I also tried apt -y remove unattended-upgrades, but that one seems to return while it is still at work. Purging appears to shut down unattended upgrades as before it exits, which is what I want.

But it turns out that even that call to apt -y purge unattended-upgrades can fail due to locking. So I changed it to while [[ $(dpkg -l | grep -P "unattended-upgrades" | wc -c) -ne 0 ]]; do apt -y purge unattended-upgrades; done, but also that fails occasionally (I can't figure out why)

I need one command which, when executed, will terminate and bury unattended upgrades immediately, regardless if it is running or not, and make the guarantee that it won't start anymore as soon as that command returns, until I explicitly apt install it again. It is ok if that command takes a minute to finish it's job.

Also, the system doesn't have Python installed, so Ansible is only issuing raw commands, until I manage to install Python which should be after a successful call to apt -y update

I am in a state where I can trigger unattended upgrades easily, since this is a VM, and as soon as I issue a date -s command to correct the stale date, unattended-upgrade kicks in. After starting the VM, I have a couple of minutes until date corrects itself automatically which then starts unattended-upgrades.

This is what I'm doing now:

- name: Disable autoupdate (part 1 of 2)
  raw: sed -i /Update/s/"1"/"0"/ /etc/apt/apt.conf.d/10periodic && sync

- name: Disable autoupdate (part 2 of 2)
  raw: echo 'APT::Periodic::Unattended-Upgrade "0";' >> /etc/apt/apt.conf.d/10periodic && sync

- name: Terminate any active autoupdate
  raw: ps -A | grep unattended-upgrades | awk '{print $1}' | xargs -r kill -15 $1

- name: Terminate any active dpkg
  raw: ps -A | grep dpkg | awk '{print $1}' | xargs -r kill -15 $1

- name: Allow dpkg to recover
  raw: dpkg --configure -a

- name: Purge autoupdate
  raw: apt -y purge unattended-upgrades    

- name: Update apt cache
  raw: apt -y update

- name: If needed, install Python
  raw: test -e /usr/bin/python || apt -y install python

Terminating dpkg is what creeps me out. All that is run on a fresh install of Ubuntu Server 18.04.1

Here is the solution created by using the accepted answer:

https://stackoverflow.com/a/51919678/277267

Daniel F
  • 867
  • 1
    I never allow unattended-upgrades near a production server. However, I do advise doing all updates before running other ansible operations. – Rui F Ribeiro Aug 19 '18 at 13:28
  • It's not much of a choice to have it disabled if Ubuntu Server ships it in an enabled state (at least it's disabled on the VPS). -- I'm only installing Python to get Ansible running, then I create some users and before I really start installing stuff, I do an apt dist-upgrade (with the exception of git because I need that one early and sometimes it isn't installed). I hope that's OK. – Daniel F Aug 19 '18 at 13:34
  • aptitude purge unattended-upgrades or dpkg --purge unattended-upgrades – Rui F Ribeiro Aug 19 '18 at 13:38
  • 1
    Is there a benefit of using dpkg --purge unattended-upgrades instead of apt -y purge unattended-upgrades (which I'm doing now)? – Daniel F Aug 19 '18 at 13:42
  • 4
    btw you're alledgedly supposed to use apt-get in scripts, so that the interactive apt program has full freedom to change behaviour to make it more friendly. – sourcejedi Aug 19 '18 at 13:44
  • you are not taking advantage of ansible as you are using it, if you use bash instead of the internal directives, better use bash scripts. and sourcejedi has a poing. apt is for command line. about the commands, pretty much the same – Rui F Ribeiro Aug 19 '18 at 13:46
  • I first need to install a Python interpreter on the server in order to be able to use Ansible modules (Ubuntu Server 18.04 LTS doesn't have it installed). I can't use Ansible's shell or apt at that point. – Daniel F Aug 19 '18 at 13:50
  • All recent Debian derivates have installed python by default for ages. – Rui F Ribeiro Aug 19 '18 at 13:54
  • This is very odd. I installed it three days ago into a VM, a freshly downloaded ubuntu-18.04.1-live-server-amd64.iso and it has no /usr/bin/python. It asks me to install it if I type python in the shell. – Daniel F Aug 19 '18 at 13:58
  • 1
    @RuiFRibeiro Error due to Ansible defaulting to python, ubuntu1804 only has python3. https://linuxconfig.org/install-python-2-on-ubuntu-18-04-bionic-beaver-linux / https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#managed-node-requirements – sourcejedi Aug 19 '18 at 13:58
  • 1
    @RuiFRibeiro not sure how much real Ansible makes a difference in this situation. https://github.com/ansible/ansible/issues/25414 – sourcejedi Aug 19 '18 at 14:02
  • @sourcejedi IMO the place of unattended-upgrades is not in servers. I prefer a playbook that I control. – Rui F Ribeiro Aug 19 '18 at 14:04
  • 1
    The problem is that unattended-upgrades is enabled by default. I think it's just the download task of the upgrade, but that seems to be enough to create the lock. Also I had this same problem with Ansible's apt module, there I was using retry stuff which was making me instane so that I opted for a real solution. ATM my first tests show that sourcejedi's wait command is the solution. – Daniel F Aug 19 '18 at 14:17

1 Answers1

13

Apparently unattended-upgrades is run from one of the systemd units apt-daily.service / apt-daily-upgrade.service. (These in turn are triggered by systemd .timer units, with the same names).

You could try to wait for the systemd units as follows:

systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

This is independent of whether you want to send SIGTERM to dpkg or apt-get or something to try and get them to finish quicker. kill only transmits a signal; it does not wait for anything. In principle you always need some way to wait, before you can use the resource that is freed up.

sourcejedi
  • 50,249
  • 1
    What does that command do? Wait until dpkg finishes if it has been started due to an unattended upgrade? I'd prefer doing that instead of sending a SIGTERM to it, which is what I'm doing now. – Daniel F Aug 19 '18 at 13:43
  • It assumes systemd (and new enough apt). It waits for the apt timer-triggered services to complete, if they are running. Apparently those are what run the scheduled unattended-upgrades jobs. Maybe it's a more general approach, since apt-daily might be running a daily apt-get update even if the unattended-upgrades package is not installed. I don't know anything about killing the unattended-upgrades process beforehand - no idea whether that does anything useful, or is any less brutal than killing dpkg. – sourcejedi Aug 19 '18 at 13:49
  • 1
    So first disabling autoupdate via /etc/apt/apt.conf.d/10periodic modification and then issuing your command and waiting for it to finish should leave the system in a state where only I am triggering dpkg (through whatever means)? I'll give it a try. – Daniel F Aug 19 '18 at 13:54
  • 2
    I just want to link here my complete solution which is based on your answer https://stackoverflow.com/a/51919678/277267 – Daniel F Aug 19 '18 at 16:30