Restart=always does not retry restarting after the required service is failed

Question

I have a kubelet.service which requires docker.service. The kubelet.service is like:

[Unit]
Description=kubelet
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet ...
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

When the docker.service is restarted, the kubelet.service will restart automatically. However, when I reboot the server, if docker.service failed when booting, then kubelet.service will not restart automatically!

# journalctl -u kubelet.service 
-- Logs begin at Fri 2018-05-25 09:35:00 CST, end at Fri 2018-05-25 09:53:13 CST. --
May 25 09:35:03 debian systemd[1]: Dependency failed for kubelet.
May 25 09:35:03 debian systemd[1]: kubelet.service: Job kubelet.service/start failed with result 'dependency'.

# journalctl -u docker
-- Logs begin at Fri 2018-05-25 09:35:00 CST, end at Fri 2018-05-25 09:53:46 CST. --
May 25 09:35:03 debian systemd[1]: Starting Docker Application Container Engine...
May 25 09:35:03 debian dockerd[1905]: invalid value "" for flag --mtu: strconv.ParseInt: parsing "": invalid syntax
May 25 09:35:03 debian dockerd[1905]: See '/usr/bin/dockerd --help'.
May 25 09:35:03 debian systemd[1]: docker.service: Main process exited, code=exited, status=125/n/a
May 25 09:35:03 debian systemd[1]: Failed to start Docker Application Container Engine.
May 25 09:35:03 debian systemd[1]: docker.service: Unit entered failed state.
May 25 09:35:03 debian systemd[1]: docker.service: Failed with result 'exit-code'.
May 25 09:35:12 debian systemd[1]: docker.service: Service hold-off time over, scheduling restart.
May 25 09:35:12 debian systemd[1]: Stopped Docker Application Container Engine.
May 25 09:35:12 debian systemd[1]: Starting Docker Application Container Engine...

As you can see, kubelet stopped at 09:35:03 and it just never restarts even after docker started normally at 09:35:12

score 1 · Answer 1 · answered May 25 '18 at 02:07

1

You are hitting your limit in restart attempts. Review StartLimitBurst. Similar question here.

answered May 25 '18 at 02:07

B. Miller

146

1

Thanks for the answer. I have set RestartSec=10 already... – Haoyuan Ge May 25 '18 at 02:12
There is still a limit on the amount of attempts. This is the StartLimitBurst. The link explains this in detail. – B. Miller May 25 '18 at 02:20
There would be an entry in the log if the restart limit were being reached. There is no such entry in the log in the question. – JdeBP May 25 '18 at 07:17

Hauke Laging · Answer 2 · 2018-05-25T10:16:42.617

0

Restart does not refer to failed dependencies but to the process belonging to this unit.

man systemd.service:

Restart=
Configures whether the service shall be restarted when the service process exits, is killed, or a timeout is reached. The service process may be the main service process, but it may also be one of the processes specified with ExecStartPre=, ExecStartPost=, ExecStop=, ExecStopPost=, or ExecReload=. When the death of the process is a result of systemd operation (e.g. service stop or restart), the service will not be restarted. Timeouts include missing the watchdog "keep-alive ping" deadline and a service start, reload, and stop operation timeouts.

edited May 25 '18 at 10:16

answered May 25 '18 at 02:01

Hauke Laging

90,279

I think when Restart=always, it should always be restarted. But the journal shows systemd only tries to start it once... – Haoyuan Ge May 25 '18 at 02:13
This is the right answer, but it could be a lot less terse. As can be seen, the questioner hasn't understood the answer as it currently stands. – JdeBP May 25 '18 at 07:20

Restart=always does not retry restarting after the required service is failed

2 Answers2