2

I have a python script that periodically generates a png image, and puts it on a server that I can access through a browser. The image displays the progress of a job I am running on the server. I'm using matplotlib to generate the images.

The python script goes to sleep after generating each image. After an hour, it wakes up, generates a new image, and then goes back to sleep.

I start the script over an ssh session. When I log off, the script gets killed. I've read the other threads covering this problem, but the proposed solutions do not work. The script is killed after I log off, even if I run it with

nohup script.py &> job-monitor.log &

or if I run it like this

script.py
[Ctrl-Z]
disown -h %1
bg 1

The strange thing is, when I log off and log back in, the script is no longer running, and job-monitor.log is empty.

What could be happening, and how do I get the script to continue running after I log off?

Additional information: I think the problem has to do with the plotting library I'm using, matplotlib. I think it may require an X session to produce a png, even though the graphic is just being saved to disk.

apdnu
  • 172
  • Take a look at this thread: http://unix.stackexchange.com/questions/4004/how-can-i-close-a-terminal-without-killing-the-command-running-in-it/4006#4006 – slm Sep 22 '13 at 13:24
  • Also if you search through these Q's on the site you'll find additional examples and guidance: http://unix.stackexchange.com/search?tab=votes&q=nohup – slm Sep 22 '13 at 13:24
  • Is your job-monitor.log not empty when you do a nohup and then empty after you logout and back in? Does script.py finish normally (with a filled log file) if you let it run long enough without login out? Apart from finding the reason for why this happens, you can try tmux or screen – Anthon Sep 22 '13 at 13:30
  • 1
    @slm: This question is not a duplicate of How can I close a terminal without killing the command running in it?. The accepted answer to that question gives two methods that I explicitly stated in my question do not work in this case. Users with the same problem as I have should not be redirected to an answer that doesn't solve their problem. – apdnu Sep 22 '13 at 17:58
  • Based on the information you've provided that Q&A is the most related and has all the information regarding the topic, from what I know of Unix & Linux. I understand your frustration, but we need more info to go on in order to assist you more than what's here. – slm Sep 22 '13 at 18:11
  • Just because a Q is labeled as a duplicate doesn't mean it definitely is, if more information is provided or other avenues are presented then the Q can definitely be reopened, this is a mechanism within the site to basically communicate this to the OP and to others. – slm Sep 22 '13 at 18:13
  • @slm: It may be the most related question, but it addresses a different problem, which has a different solution. I've edited the question to make this even more explicit, but I thought that when I asked the question initially, I made it pretty clear that the solutions presented elsewhere don't work for this question. – apdnu Sep 22 '13 at 18:29
  • I've found a workaround for matplotlib that I could post here, if the question were re-opened. The question is more general than just using matplotlib, however, and other users might like to know how to keep an X server running after logging off. Either way, the answers that have been provided here and in the other thread don't address the problem. – apdnu Sep 22 '13 at 18:33
  • Not strictly a duplicate, but this provides a solution: How can I run Firefox on Linux headlessly (i.e. without requiring libgtk-x11-2.0.so.0)? – Gilles 'SO- stop being evil' Sep 22 '13 at 20:50
  • @Thucydides411 - apologies, one of our other users knows the issue you're experiencing. I and 2 others have already voted to reopen this, give it some time to work around so a couple of others will reopen. Need 5 to reopen. – slm Sep 22 '13 at 21:47
  • I am pretty sure I did not mark this as a duplicate, but indicated that more information (now provided) needed to be given. The system does not seem to make a distinction between the argument of the first close vote and the rest of the reason of the rest of the voters. – Anthon Sep 23 '13 at 03:00

4 Answers4

3

If the problem is with matplotlib (that is, your script never provides an answer if you stay connected, or it works because ssh forwards your Xwindow connection), you have to put in your matplotlibrc file:

backend : AGG

This way, the script doesn't need Xwindow to work.

Tobby
  • 46
  • Importing matplotlib first, and invoking matplotlib.use('Agg') before importing matplotlib.pyplot also works. – apdnu Sep 23 '13 at 14:54
0

Instead of IO/redirecting use tmux or screen command for session monitoring.

PersianGulf
  • 10,850
0

May I humbly suggest that you throw away the time.sleep() part from your script and use either cron or at for periodic and/or delayed invocations?

This simplifies your code, safely removes dependencies on ssh session detachment, and separates the concerns of image generation and scheduling.

Regarding the script dying: I suspect that the script after nohup might not have write access to the log file, either because it lacks write access to the directory where you start it, or because an existing log file is owned by e.g. root and thus inaccessible.

I don't know if disowning a job and then trying to call bg against it should work. I always do the opposite: foo & disown %1; it works exactly in circumstances like yours.

9000
  • 1,639
-1

Try redirecting STDIN as well. It's possible the script is reading from stdin, so when you close the shell, stdin goes away, and it dies.

script.py </dev/null &>/dev/null &
disown
phemmer
  • 71,831