Preventing multiple python scripts from crashing with single shell script?

Question

I would like to create a shell script to prevent a python script from crashing in a Raspberry Pi. This python script is auto started on reboot with cron, however, sometimes it can crash while running.

Currently my shell script reads:

#!/bin/sh

COMMAND='python home/pi/projects/mypythonscript.py'
LOGFILE=restart.txt

writelog() {
  now=`date`
  echo "$now $*" >> $LOGFILE
}

writelog "Starting"
while true ; do
  $COMMAND
  writelog "Exited with status $?"
  writelog "Restarting"
done

I wanted to ask if I can create two command lines, in order to make sure two independent scripts can be prevented from crashing with this shell script? Or do I have to make a new shell script file for the second python script?

I am thinking of modifying it this way:

#!/bin/sh

COMMAND1='python home/pi/projects/mypythonscript1.py'
COMMAND2='python home/pi/projects/mypythonscript2.py'
LOGFILE=restart.txt

writelog() {
  now=`date`
  echo "$now $*" >> $LOGFILE
}

writelog "Starting"
while true ; do
  $COMMAND1
  $COMMAND2
  writelog "Exited with status $?"
  writelog "Restarting"
  done

Would this modification work? I appreciate any advice as I am still quite new to the linux (debian) platform.

Your sigle-script approach will start COMMAND1, wait for it to terminate, then start COMMAND2, and also wait for it to terminate, before it repeats that cycle. Are you sure this is intended? — Murphy, Feb 15 '18 at 13:41
Thanks for highlighting this. This is not what I intended. I would like that either of the scripts be terminated (not both), and the cycle to be repeated. That way, both python scripts would keep running. So it seems that I might have to create two separate shell scripts for each? Or is there another way I can modify this shell script? — Craver2000, Feb 15 '18 at 13:47
The exact implementation depends on how the two processes depend on each other. If one crashes or otherwise terminates, should the other be terminated and restarted, too? If it can just run along, then probably two independent scripts would be the most straight-forward solution. In short, try to nail down your complete requirements first, then you'll probably get a better idea how to handle restarting. Make a list, check it for being complete and consistent, then come back and update your question if you still need help with that. — Murphy, Feb 15 '18 at 14:40
Thanks. I've looked at the tour as suggested. As to your previous post, I agree that having two independent scripts might be needed. — Craver2000, Feb 15 '18 at 16:36

rusty shackleford · Answer 1 · 2018-02-15T14:10:45.017

1

If you are using systemd you could create a service for each command, then get systemd to restart it if it crashes.

Something like:

[Unit]
Description='description of script'

[Service]
ExecStart=/path/too/script
Restart=always

[Install]
WantedBy=multi-user.target

This can then be put into /etc/systemd/system, after running systemctl daemon-reload you will be able to start the service. It will also start after a reboot.

edited Feb 15 '18 at 14:10

answered Feb 15 '18 at 14:05

rusty shackleford

2,415

Thanks for the suggestion. I remember testing out a couple of approach back then and found the shell script method the most reliable. I might also have tried the systemd approach but I can't remember now. But I'll give this approach a try. Thanks! – Craver2000 Feb 16 '18 at 02:32
If I have two scripts that I want to ensure that they would always be running, do I just stack one service on top of the other? like:[Service] ExecStart=/path/too/script1 Restart=always [Service] ExecStart=/path/too/script2 Restart=always – Craver2000 Feb 17 '18 at 08:23
No you would create a separate service for each script, this will also allow you to stop and start them independently. – rusty shackleford Feb 19 '18 at 08:46

G-Man Says 'Reinstate Monica' · Answer 2 · 2018-02-25T06:51:32.727

If you want to handle an arbitrary number of asynchronous processes from one control script, you can use this approach:

#!/bin/sh
COMMAND1=(python home/pi/projects/mypythonscript1.py)
COMMAND2=(python home/pi/projects/mypythonscript2.py)
    ︙
rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
rm -f COMMAND2_failed; ("${COMMAND2[@]}"; touch COMMAND2_failed)&
       ︙
while true
do
        if [ -e COMMAND1_failed ]
        then
                # Restart Command1
                rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
        fi
        if [ -e COMMAND2_failed ]
        then
                # Restart Command2
                rm -f COMMAND2_failed; ("${COMMAND2[@]}"; touch COMMAND2_failed)&
        fi
         ︙
        sleep 60
done

COMMAND1=(python home/pi/projects/mypythonscript1.py) defines COMMAND1 to be an array. This allows the command to include words that contain spaces; for example, something like COMMAND1=(python "home/pi/projects/my python script1.py").
"${COMMAND1[@]}" is analogous to your use of $COMMAND1; it runs your first command, quoting its constituent words.
(command_A; command_B) essentially creates an ad-hoc two-line script:
```
command_A
command_B
```
so ("${COMMAND1[@]}"; touch COMMAND1_failed) says “run command #1, wait for it to finish, and then touch (create) a file called COMMAND1_failed. This entire mini-script runs in the background, so the main script can go on and run command #2, etc.
Realistically, you should probably use full pathnames for the failed files, and maybe put their names into variables.
After starting all the background jobs, the main script enters an infinite (forever) loop. If the COMMAND1_failed file exists, that means that command #1 and needs to be restarted. Etc.
Adjust the sleep as desired. Too low a value, and the control script might run so often that it affects the performance of the system. Too high a value and the asynchronous tasks will not be restarted promptly, so the responsiveness of the system would suffer in that way.

The above starts all the processes, and then enters an infinite loop. Here is a trivial optimization:

#!/bin/sh
    ︙
touch COMMAND1_failed
       ︙
while true
do
        if [ -e COMMAND1_failed ]
        then
                # (Re)start Command1
                rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
        fi
         ︙
        sleep 60
done

which starts the background processes on the first iteration of the loop, after creating all the necessary COMMANDn_failed files to make it look like the processes need to be (re)started. The only difference is that it lists the commands to start the processes once instead of twice.

Preventing multiple python scripts from crashing with single shell script?

2 Answers2