How to prevent a program from dying when its ssh session dies?

Question

I have a really old console application that I want to make a bit more resilient. The program is used this way:

the user uses a custom terminal emulator to connect to a remote machine through ssh
the user starts a shell script
the shell script might start a long-running progress database process.

Obviously, sometimes users simply lose the ssh connection to the machine, and in this case, the ssh session is terminated, the shell script running inside is terminated, and finally the progress database process is terminated as well. In like one case out of a thousand this causes corruption in the database, so I'd like to prevent it from happening.

What I tried so far:

starting a screen or tmux session before starting the shell script - this doesn't work because the application needs the TERM variable to be set to at386 (and it bypasses termcap/terminfo completely... ugh...)
nohup/disown the progress process - this doesn't work because the shell script and the progress process are continuously communication with each other in obscure ways it seems

Any other idea of how to make sure that the progress process doesn't terminate when the ssh session is killed?

Since you tried screen and nohup, we probably need more information about your specific programs. — Falmarri, May 23 '11 at 20:47
"needs the TERM variable to be set to at386" redo, or delete your application. — innocent-world, Aug 21 '13 at 22:16

score 5 · Accepted Answer · answered May 23 '11 at 21:08

5

Add term at386 to your .screenrc in order to override TERM. If this doesn't help, try dtach instead of Screen (which doesn't do any terminal emulation itself).

answered May 23 '11 at 21:08

Adam Byrtek

1,339

score 4 · Answer 2 · answered May 24 '11 at 15:22

4

In many cases, the simplest solution is to use nohup and put the non-interactive process into the background using & and redirect standard output and standard error to files.

answered May 24 '11 at 15:22

RedGrittyBrick

2,109

score 2 · Answer 3 · answered May 23 '11 at 21:22

2

In other words, you're looking for something like screen, but that passes control sequences to the terminal rather than doing its own translation? Try dtach. It provides the same terminal detachment feature as screen, but without the multiple windows or the terminal emulation.

answered May 23 '11 at 21:22

Gilles 'SO- stop being evil'

829,060

Thanks - this works well enough, though (what a surprise) the program that I'm running has no method to redraw itself, so at every re-attach I lose the contents of the screen buffer. Any solution to that? – Zizzencs May 24 '11 at 08:36

score 0 · Answer 4 · answered May 23 '11 at 20:46

Perhaps try a remote X desktop via vnc or nx. You could tunnel vnc over ssh for security or use nx which runs via ssh by default (and is much faster). The app could then run in a Terminal on the X desktop and if the connection is lost, the desktop would keep going allowing reconnection at any time. My preference is No Machine NX which is free for up to two simultaneous users and installs very easily.

An alternative, if available, is to use a remote console via ALOM, ILOM, DRAC, IP-kvm or IP-serial console, etc.. That way the app would be running on the console and could not be interrupted by a connection failure.

How to prevent a program from dying when its ssh session dies?

4 Answers4