How to disable `apt-daily.service` on Ubuntu cloud VM image?

Question

The Ubuntu 16.04 server VM image apparently starts the "apt-daily.service" every 12 hours or so; this service performs various APT-related tasks like refreshing the list of available packages, performing unattended upgrades if needed, etc.

When starting from a VM "snapshot", the service is triggered immediately, as (I presume) systemd realizes quickly that the timer should have gone off long ago.

However, a running APT prevents other apt processes from running as it holds a lock on /var/lib/dpkg. The error message indicating this looks like this:

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

I need to disable this automated APT task until Ansible has completed the machine setup (which typically involves installing packages); see https://github.com/gc3-uzh-ch/elasticluster/issues/304 for more info and context.

I have tried various options to disable the "unattended upgrades" feature through a "user data" script for cloud-init, but all of them have failed so far.

1. Disable the systemd task

systemd task apt-daily.service is triggered by apt-daily.timer. I have tried to disable one or the other, or both, with various cobinations of the following commands; still, the apt-daily.service is started moments after the VM becomes ready to accept SSH connections::

    #!/bin/bash

    systemctl stop apt-daily.timer
    systemctl disable apt-daily.timer
    systemctl mask apt-daily.service
    systemctl daemon-reload

2. Disable config option APT::Periodic::Enable

Script /usr/lib/apt/apt.systemd.daily reads a few APT configuration variables; the setting APT::Periodic::Enable disables the functionality altogether (lines 331--337). I have tried disabling it with the following script::

    #!/bin/bash

    # cannot use /etc/apt/apt.conf.d/10periodic as suggested in
    # /usr/lib/apt/apt.systemd.daily, as Ubuntu distributes the
    # unattended upgrades stuff with priority 20 and 50 ...
    # so override everything with a 99xxx file
    cat > /etc/apt/apt.conf.d/99elasticluster <<__EOF
    APT::Periodic::Enable "0";
    // undo what's in 20auto-upgrade
    APT::Periodic::Update-Package-Lists "0";
    APT::Periodic::Unattended-Upgrade "0";
    __EOF

However, despite APT::Periodic::Enable having value 0 from the command line (see below), the unattended-upgrades program is still run...

    ubuntu@test:~$ apt-config shell AutoAptEnable APT::Periodic::Enable
    AutoAptEnable='0'

3. Remove /usr/lib/apt/apt.systemd.daily altogether

The following cloud-init script removes the unattended upgrades script altogether::

    #!/bin/bash

    mv /usr/lib/apt/apt.systemd.daily /usr/lib/apt/apt.systemd.daily.DISABLED

Still, the task runs and I can see it in the process table! although the file does not exist if probed from the command line::

ubuntu@test:~$ ls /usr/lib/apt/apt.systemd.daily
ls: cannot access '/usr/lib/apt/apt.systemd.daily': No such file or directory

It looks as though the cloud-init script (together with the SSH command-line) and the root systemd process execute in separate filesystems and process spaces...

Questions

Is there something obvious I am missing? Or is there some namespace magic going on which I am not aware of?

Most importantly: how can I disable the apt-daily.service through a cloud-init script?

Answer 1

Yes, there was something obvious that I was missing.

Systemd is all about concurrent start of services, so the cloud-init script is run at the same time the apt-daily.service is triggered. By the time cloud-init gets to execute the user-specified payload, apt-get update is already running. So the attempts 2. and 3. failed not because of some namespace magic, but because they altered the system too late for apt.systemd.daily to pick the changes up.

This also means that there is basically no way of preventing apt.systemd.daily from running -- one can only kill it after it's started.

This "user data" script takes this route::

#!/bin/bash

systemctl stop apt-daily.service
systemctl kill --kill-who=all apt-daily.service

# wait until `apt-get updated` has been killed
while ! (systemctl list-units --all apt-daily.service | egrep -q '(dead|failed)')
do
  sleep 1;
done

# now proceed with own APT tasks
apt install -y python

There is still a time window during which SSH logins are possible yet apt-get will not run, but I cannot imagine another solution that can works on the stock Ubuntu 16.04 cloud image.

Answer 2

Note: Unfortunately part of the solution below doesn't work on Ubuntu 16.04 systems (such as that of the questioner) because the suggested systemd-run invocation only works on Ubuntu 18.04 and above (see the comments for details). I'll leave the answer here because this question is still a popular hit regardless of which Ubuntu version you are using...

On Ubuntu 18.04 (and up) there may be up to two services involved in boot time apt updating/upgrading. The first apt-daily.service refreshes the list of packages. However there can be a second apt-daily-upgrade.service which actually installs security critical packages. An answer to the "Terminate and disable/remove unattended upgrade before command returns" question gives an excellent example of how to wait for both of these to finish (copied here for convenience):

systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true

(note this has to be run as root). If you are trying to disable these services on future boots you will need to mask BOTH services:

systemctl mask apt-daily.service apt-daily-upgrade.service

Alternatively you can systemctl disable both services AND their associated timers (i.e. apt-daily.timer and apt-daily-upgrade.timer).

Note the masking/disabling techniques in this answer only prevent the update/upgrade on future boots - they won't stop them if they are already running in the current boot.

Answer 3

You can disable this via the "bootcmd" cloud-init module. This runs before network is brought up, which is required before apt update can get a chance to run.

#cloud-config
bootcmd:
    - echo 'APT::Periodic::Enable "0";' > /etc/apt/apt.conf.d/10cloudinit-disable
    - apt-get -y purge update-notifier-common ubuntu-release-upgrader-core landscape-common unattended-upgrades
    - echo "Removed APT and Ubuntu 18.04 garbage early" | systemd-cat

Once you ssh into the instance, you should also wait for the final phases of cloud-init to finish, since it moves apt sources / lists around.

# Wait for cloud-init to finish moving apt sources.list around... 
# a good source of random failures
# Note this is NOT a replacement for also disabling apt updates via bootcmd
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
    echo 'Waiting for cloud-init to finish...'
    sleep 3
done

This is also helpful to see how early the bootcmd runs:

# Show microseconds in systemd journal
journalctl -r -o short-precise

You can verify this worked as follows:

apt-config dump | grep Periodic

# Verify nothing was updated until we run apt update ourselves.
cd /var/lib/apt/lists
sudo du -sh .   # small size
ls -ltr         # old timestamps

Answer 4

Based on Anon's solution, I created this script, which after being run and then rebooting, solves the problem for me:

#!/bin/sh

systemctl mask apt-daily.service apt-daily-upgrade.service
systemctl disable apt-daily.service apt-daily-upgrade.service
systemctl disable apt-daily.timer apt-daily-upgrade.timer

Answer 5

Woudn't ist be easier to mask the unit

systemctl mask apt-daily.service

?

Answer 6

This cloud-init works.

#cloud-config
apt:
  conf: |
    APT {
      Periodic {
        Update-Package-Lists "0";
      };
    };
    Unattended-Upgrade {
      Package-Blacklist {
        "*";
      };
    };
runcmd:
  - [ systemctl, stop, apt-daily.timer, apt-daily-upgrade.timer ]
  - [ systemctl, disable, apt-daily.timer, apt-daily-upgrade.timer ]

Answer 7

The excellent answers on here have been working for me for ages but suddenly I started getting the dreaded

{"changed": false, "msg": "Failed to lock apt for exclusive operation"}

from Ansible again

I'm running an Ubuntu 20.04 AWS image in EC2, the issue turned out to be that since 16.04 the AWS SSM-agent is included by default as a snap package and snap runs on it's own update schedule. SSM-agent was trying to update at the exact time I was trying to do an apt update. There are various options to change the update schedule for snap or put it on hold for up to 60 days but it seems like it ignores whatever you say for the first update and will still cause instance provisioning to fail.

The only thing that worked for me was to run:

snap disable amazon-ssm-agent

as the first command in the instance user-data

and:

snap enable amazon-ssm-agent

as the last

It took me a long time to work this out

[ update ]

disabling snap will fail if the snap update is already running, this worked for me

i=0
while ! snap disable amazon-ssm-agent; do
  echo 'attempting to disable snap...' $i
  sleep 2
  ((i=i+1))
  if [ $i -gt 10 ]; then
    echo unable to disable snap..
    exit 1
  fi
done

what a mess

Answer 8

This waits for 1sec in a whil loop and checks if the lock is released.

while : ; do
                sleep 1
                echo $( ps aux | grep -c lock_is_held ) processes are using apt.
                ps aux | grep -i apt
                [[ $( ps aux | grep -c lock_is_held ) > 2 ]] || break
        done
        echo Apt released

Answer 9

If the purpose is to provision the machine without incurring in a locking error, the simplest and most stable solution is to run the provisioner remote command (or the apt-related commands) while locking the apt-daily lock file. No service masking/disabling/waiting etc.etc.

In Ubuntu server, the package update programs (apt-daily, unattended-upgrades, cloud-init), go through apt-daily, so the solution can be centered on it.

Apt-daily uses flock(2) locks, which, differently from dpkg/apt (which use fnctl), can be managed via a commandline tool, with the convienience that it supports waiting on the lock.

Therefore if one, for example, uses Chef, it's as simple as running, on the nodes:

$ sudo flock /var/lib/apt/daily_lock chef-client

and:

• either chef-client will grab the lock first, and any apt-daily-based service will wait;
• or any apt-daily-based service will lock it first, then chef-clien will wait, and finally run once apt-daily completes.

If the provisioner is not invoked on the nodes, it's possible to modify the provisioner configuration (eg. Ansible playbook) to run:

$ sudo flock /var/lib/apt/daily_lock apt update
$ sudo flock /var/lib/apt/daily_lock apt upgrade

(or any variation, like sh -c '...' and so on)

Source: https://saveriomiroddi.github.io/Handling-the-apt-lock-on-ubuntu-server-installations.