Starting systemd service inside systemd service causes deadlock

Question

I have a systemd service (let's call it first.service) that calls a bash script.

In that bash script I first stop and then start another service with systemctl (e.g. systemctl start another.service.

I have noticed that when I execute my script in a shell- everything works correctly and another.service is being stopped and later started.

When I call systemctl restart first.service systemd stops the "another.service" properly, but hangs on starting it.

When I check result of ps it says that both systemctl calls are working, i.e. systemctl restart first.service and systemctl start another.service.

I use systemd 230

Is this known behaviour? How can I fix this?

Is there any better way to handle (re)starting systemd services from within the one service?

EDIT: My first.service file:

[Unit]
Description=First service
#Before=local-fs.target apache2.service rsyslog.service

[Service]
Type=oneshot
ExecStart=/usr/bin/test.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

My another.service file:

[Unit]
Description=Test service

[Service]
Type=oneshot
ExecStart=/bin/bash -c "while ( true  ) ; do { date ; echo 'It works!' ; sleep 10 ; } done"
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

and my bash script:

#!/bin/bash
echo stopping
systemctl stop another.service
echo "result of stop = $?"
echo starting
systemctl start another.service
echo "result of start = $?"
echo DONE

After hanging I get the following output from ps:

[piotr@/etc/systemd/system] $ ps -aux | grep systemctl
root      7801  0.0  0.0  27696  1336 pts/21   S+   16:06   0:00 systemctl restart first
root      7807  0.0  0.0  27696  1320 ?        S    16:06   0:00 systemctl start another.service
piotr     7915  0.0  0.0  15752   968 pts/22   S+   16:06   0:00 grep --color=auto systemctl

EDIT2: I'm posting systemctl status from both services in the exact moment when in another console I called systemctl restart first.

I have also added checking return value in my bash script. Updated above.

first.service:

● first.service - First service
Loaded: loaded (/etc/systemd/system/first.service; disabled; vendor preset: enabled)
Active: activating (start) since Wed 2017-04-19 16:34:43 CEST; 46s ago
Main PID: 12761 (test.sh)
CGroup: /system.slice/first.service
├─12761 /bin/bash /usr/bin/test.sh
└─12766 systemctl start another.service

Apr 19 16:34:43 piotr-ideapad systemd[1]: Starting First service...
Apr 19 16:34:43 piotr-ideapad test.sh[12761]: stopping
Apr 19 16:34:43 piotr-ideapad test.sh[12761]: result of stop = 0
Apr 19 16:34:43 piotr-ideapad test.sh[12761]: starting

another.service:

● another.service - Test service
Loaded: loaded (/etc/systemd/system/another.service; disabled; vendor preset: enabled)
Active: activating (start) since Wed 2017-04-19 16:34:43 CEST; 1min 40s ago
Main PID: 12767 (bash)
CGroup: /system.slice/another.service
├─12767 /bin/bash -c while ( true   ) ; do { date ; echo 'It works!' ; sleep 10 ;  } done
└─13066 sleep 10

Apr 19 16:35:43 piotr-ideapad bash[12767]: Mi 19. Apr 16:35:43 CEST 2017
Apr 19 16:35:43 piotr-ideapad bash[12767]: It works!
Apr 19 16:35:53 piotr-ideapad bash[12767]: Mi 19. Apr 16:35:53 CEST 2017
Apr 19 16:35:53 piotr-ideapad bash[12767]: It works!
Apr 19 16:36:03 piotr-ideapad bash[12767]: Mi 19. Apr 16:36:03 CEST 2017
Apr 19 16:36:03 piotr-ideapad bash[12767]: It works!
Apr 19 16:36:13 piotr-ideapad bash[12767]: Mi 19. Apr 16:36:13 CEST 2017
Apr 19 16:36:13 piotr-ideapad bash[12767]: It works!
Apr 19 16:36:23 piotr-ideapad bash[12767]: Mi 19. Apr 16:36:23 CEST 2017
Apr 19 16:36:23 piotr-ideapad bash[12767]: It works!

EDIT3: After exchanging some comments I'll try to rephrase the problem.

Say that none of my processes listed above is enabled or started. When I call systemctl status on them - they are loaded but inactive.
Then I call systemctl start first and this terminal command does not finish.
If I call systemctl status on those services I get that status of Active: for both of them is activating (start) but my execution of systemctl start first in the terminal still did not exit and it hangs indefinitely.

I think that both of those processes are queued and systemctl start another in my bash script waits for the systemctl start first to finish before it finishes by itself and here we get a deadlock.

Answer 1

What appears to be a deadlock is actually the Type=oneshot directive working as advertised. The "hang" can be triggered by running your another.service directly:

systemctl start another

Once cancelled, one can review journalctl to find it wasn't "hung" at all, but running an infinite loop as expected. In man systemd.service, we find the documentation for Type=oneshot:

Behavior of oneshot is similar to simple; however, it is expected that the process has to exit before systemd starts follow-up units.

In other words, you have written an infinite loop and then instructed systemd to wait till it finishes before moving on.

You either need to use a different Type=, like Type=simple, or quit using an infinite loop for your main process.