Keep processes running after SSH session disconnects

Question

I sometimes have long-running processes that I want to kick off before going home, so I create a SSH session to the server to start the process, but then I want to close my laptop and go home.

Later, after dinner, I want to check on the process that I started before leaving work. How can I do that with SSH?

My understanding is that if you break your SSH connection you will also break your login session on the server, therefore killing the long-running process.

score 341 · Accepted Answer · edited Mar 12 '24 at 23:53

341

Use nohup to make your process ignore the hangup signal:

$ nohup long-running-process &
$ exit

To monitor progress, check nohup's output:

$ tail -f nohup.out

edited Mar 12 '24 at 23:53

Paradox

744

answered Aug 14 '10 at 05:00

Warren Young

72,032

35

This is it. No need to install screen or tmux. This is actually the good old way to do it. Sure 'screen' or 'tmux' are both wonderous apps and should be used when needed, but as simple as running a process in the background from which you can log out, go for the above. – reiche Aug 14 '10 at 11:40
5

is nohup like disown? – cjac Aug 15 '10 at 09:12
47

They have similar purposes, but they differ in many ways. nohup intercepts SIGHUP so that when the shell that ran it quits and sends SIGHUP to all its still-running children, long-running-process doesn't die. disown simply removes the specified job from Bash's child list, so it won't try to send SIGHUP at all. nohup is a program separate from the shell, so it works with all shells, whereas disown is a Bash builtin. nohup accepts the command to run, whereas disown only works after the job is started and you've backgrounded it so you can get back to the shell. – Warren Young Aug 15 '10 at 14:00
Hey that's cool, I didn't know you could do that. Very useful in cases where you can't use screen for whatever reason. – Sandy Aug 16 '10 at 13:11
3

Awesome quick review for those who are unfamiliar with nohup and disown - as well as some easy ways to deal with terminal sessions
http://www.serverwatch.com/tutorials/article.php/3935306/Detach-Processes-With-Disown-and-Nohup.htm
– Dan Aug 12 '14 at 21:58
Does this work on Cygwin for windows? – William Aug 30 '15 at 22:40
37

tail -f nohup.out to check what's going on when you're back. – Ricardo Stuven Nov 03 '16 at 15:15
1

You saved a human today! – Junaid Apr 13 '17 at 10:32
2

this simply doesn't work in my case. I run the process with nohup, log out and when i log back in, 3 seconds later. The process is dead. – john-jones Nov 01 '17 at 09:30
@HermannIngjaldsson - Maybe the process ended because it ended (IE, it hit a fatal error), and it has nothing to do with your logging out? – ArtOfWarfare Nov 08 '17 at 21:58
no the process is implementing a sleep 1000 command. – john-jones Nov 08 '17 at 22:00
how to stop the process which is started using nohup? i tried disown but it didin't work – Abhay Mar 26 '18 at 04:17
@AbhayGawade: Find its PID and kill it. pgrep, pkill, ps -eaf | grep, etc. If you need to reattach and shut it down gracefully, then you probably want to be using screen/tmux instead. – Warren Young Mar 26 '18 at 04:42
1

what is the & ? – Andrew Apr 03 '18 at 20:55
2

@Andrew: It immediately puts long-running-process in the background so you can log out without having to do that via a separate step. – Warren Young Apr 03 '18 at 21:46
1

setsid + nohup if running in AWS EC2 / cloud provider: Aws implements its own signaling and sends kill to process when session is closed, with setsid you avoid this. – NicoKowe Nov 29 '21 at 08:52
To use with a loop, see https://stackoverflow.com/questions/3099092/why-cant-i-use-unix-nohup-with-bash-for-loop – enharmonic Dec 09 '22 at 06:29
What is the & ? See What are the shell's control and redirection operators? See also  Difference between nohup, disown and &. – G-Man Says 'Reinstate Monica' Mar 12 '24 at 22:12

Sandy · Answer 2 · 2010-08-14T01:14:04.717

181

You want to be using GNU Screen. It is super awesome!

ssh me@myserver.com
screen               #start a screen session
run-a-long-process

CTRL+a , d to detatch from your screen session

exit                 #disconnect from the server, while run-a-long-process continues

When you come back to your laptop:

ssh me@myserver.com
screen -r            #resume the screen session

Then check out the progress of your long-running process!

screen is a very comprehensive tool, and can do a lot more than what I've described. While in a screen session, try ctrl+a,? to learn a few common commands. Probably the most common are:

CTRL+a , c to create a new window
CTRL+a , n to switch to the next window in your screen session
CTRL+a , p to switch to the previous window in your screen session
if you log in from a bunch of different systems, you may have accidentally left yourself attached to an active screen session on a different computer. for that reason, I always resume with screen -d -r to ensure that if another shell is attached to my screen session, it will be detached before I resume it on my current system.

edited Aug 14 '10 at 01:14

answered Aug 14 '10 at 01:04

Sandy

3,262
2
22
14

12

tmux is a modern analogue of screen. – Andriy Volkov Aug 14 '10 at 02:21
I don't know about modern, but default key prefix of tmux (Ctrl + b) helps if you're used to bash/emacs keys. – sajith Aug 14 '10 at 04:15
2

screen is pretty cool if you want a shared terminal for some reason. Create a screen screen -S name and let your other friend connect to it with screen -x name. – Patrick Oct 06 '10 at 09:23
3

tmux with tmuxinator is a great combination for fancy setups, while I prefer screen as a quick and simple solution. – earthmeLon Jun 19 '14 at 15:31
ctrl + a, K to kill the screen session. Answer with y to confirm the Really kill this window question. – User Feb 27 '17 at 13:36
Exactly. GNU screen still comes preinstalled on some more frugal systems, so it's nice to know its basics, but for daily work I prefer Tmux by far. Although I will change its prefix to Ctrl+a as about the first thing I always do. – 0xC0000022L Nov 04 '23 at 20:41

score 76 · Answer 3 · answered Oct 30 '11 at 16:19

76

If you haven't planned ahead and setup screen, etc. just do the following:

If your process is running in the background: goto #3, else: Ctrl-Z to suspend foreground process. This will report the job # of the suspended process, for example:
```
[1]+  Stopped                 processName
```
Send processName to the background with bg %1 (using whatever the job # is following the %). This will resume processName in the background.
Disown processName with disown %1 or disown PID. Use the -h flag if you want to maintain ownership until you terminate your current shell.

answered Oct 30 '11 at 16:19

Mike Covington

1,188

6

Thanks for this! I had a job running and thought I wasn't going to be able to keep it going because I hadn't used & when I started it. This seems to work great! – Matt Mar 25 '12 at 15:42
5

This is amazing! I never knew Linux could let you transfer control of active processes like this. I was afraid my battery would die while SSH'ing into my server in the middle of a long running process, and this was the answer for me! – Pluto May 20 '14 at 21:05
1

Does a disowned process dies if it needs stderr or stdout? Compared to nohup. – it3xl Sep 26 '21 at 17:12

score 17 · Answer 4 · answered Aug 14 '10 at 07:00

What you want to use is screen or even better a user-friendly wrapper around screen called byobu.

Screen allows you to run multiple virtual terminal sessions in the same ssh session. A tutorial and help pages are available.

byobu is a wrapper that allows to easily open new screens with a simple function key instead of key combination from ctrl-a. It also shows a status line with all the open virtual terminals which can be named.

Another nice feature is the fact that all your screen can stay up while your ssh connection is disconnected. You just connect again via ssh and call byobu and everything is like before.

At last some screenshots of byobu.

I launched a process on my server within Byobu Terminal without really knowing what it was (I just searched Terminal and clicked on Byobu because I'd never seen it before). I kept using Byobu because it had a colorful status bar at the bottom and I thought it was cool. Today I found myself wanting to access that terminal session remotely through SSH. All I had to do was type "byobu" and it showed that terminal session right away. Really happy I stumbled across byobu! — nick, Mar 10 '15 at 20:13

score 16 · Answer 5 · answered Oct 14 '19 at 05:35

Tmux is a good option available to run your long-running processes in the background.

I have to keep running the long-running processes on a google cloud platform's VM instance/server (with OS: Ubuntu 16.0). Where I have to start SSH terminal and from the terminal, I have to keep the terminal connected to run the process. Now up to this, all is good. But wait if the connection with my SSH terminal is terminated then the long-running processes stop immediately and hence I have to re-run them once again once the ssh terminal is restarted or from a new ssh terminal.

I find tmux is the good solution to avoid termination of processes that we want to run even after the terminal is closed.

Terminal Multiplexer (tmux) to start the tmux session:

Start the ssh terminal
Type tmux. It will open a window in the same terminal.
Run the command to start long-running processes in the tmux session.

Now even if SSH terminal is closed/terminated suddenly tmux session will keep running the started lon-running processes on the instance/server.

If the connection terminated then how to reconnect it to see the processes running in the tmux session in the background:

reconnect or open new ssh terminal. To see this process(which is kept running in the background) type: tmux attach command.

Want to terminate tmux session:

Stop the process. Then type exit in tmux-terminal-window.

(Note that: If we use tmux detach command: it will exit from tmux session window/terminal without terminating/stopping the tmux sessions)

For more details please refer following article:

score 13 · Answer 6 · edited Jun 08 '17 at 08:47

13

It might be worth noting that

ssh -t lala screen -rxU moo will attach to the moo session on host lala

ssh -t lala screen -S moo will create the moo session on host lala

and

ssh -t lala screen -S moo quux will create the moo session on host lala and run the program quux, quitting the session on completion.

edited Jun 08 '17 at 08:47

Community

1

answered Aug 15 '10 at 09:11

cjac

386

score 11 · Answer 7 · answered Nov 13 '15 at 15:16

Old question, strange still nobody advised tmux, which acts as a wrapper for n consoles and keep them open until needed. This allows more control, beside a number of functions tmux has. It's easy to manage it, you just execute tmux, which brings you in its shell, start your looong job, then press ctrl+b followed by d (detach) (ctrl+b is the "ok google" of tmux, and d is the command to close without exit from the shell). This actually works if you just close, for example, putty. After dinner, when you connect again, you can reopen tmux with tmux attach to see your screen exactly as you left. Something I love is splitting pane: ctrl+b and then press ". To change from one pane to another, ctrl+b and then press the up/down arrow.

Victor Hugo Arango A. · Answer 8 · 2016-12-22T14:59:54.103

You can find a good guide here: Keep Your SSH Session Running when You Disconnect

sudo apt-get install screen

Now you can start a new screen session by just typing screen at the command line. You’ll be shown some information about screen. Hit enter, and you’ll be at a normal prompt.

To disconnect (but leave the session running) Hit Ctrl + A and then Ctrl + D in immediate succession. You will see the message [detached]

To reconnect to an already running session

screen -r

To reconnect to an existing session, or create a new one if none exists

screen -D -r

To create a new window inside of a running screen session Hit Ctrl + A and then C in immediate succession. You will see a new prompt.

To switch from one screen window to another Hit Ctrl + A and then Ctrl + A in immediate succession.

To list open screen windows Hit Ctrl + A and then W in immediate succession

You can use <kbd>Ctrl</kbd> to show the names as keyboard keys. — , Dec 22 '16 at 07:41

score 9 · Answer 9 · edited Dec 19 '21 at 14:55

In 2021 we can also suggest for linux distributions with systemd the use of systemd-run as a way to detach a process and keep it running after closing ssh login.

One can launch a command , give it some name like 'background_cmd_service' and turn it into a systemd service. Later inspect status:

systemd-run --unit=background_cmd_service --remain-after-exit command
# later on
journalctl -b -u background_cmd_service.service
systemctl status background_cmd_service.service

When launching the service as regular user, one might need to enable lingering (cf. enable-linger loginctl option). This is true even with nohup since systemd is cleaning up all services running within the session once the user logs out.

For scripts, or other executables, you can provide the path to the executable like this:

systemd-run --unit=background_cmd_service --remain-after-exit --working-directory=/full/path/ command

You can also stop your service or clear a failed service using the standard systemctl options like:

systemctl stop background_cmd_service.service

systemctl reset-failed background_cmd_service.service

Many of the pre-2016, but highly voted answers on stack exchange suggesting nohup, disown, command &, and (command &) don't work since 2016, due to a systemd default change in how it treats user processes after user logout - Details

enable-linger needs root access or some sort of authentication - is there any way to do this without root? — Elenchus, Feb 09 '22 at 13:49
@Elenchus You can run "loginctl disable-linger|enable-linger" with a regular no root user to set lingering state for the current user running the command. At least this is the default setup on Debian/11. Changing login lingering for other users requires root access. — kaliko, Feb 10 '22 at 15:52
I get a message saying 'Authentication is required to run programs as a non-logged-in user' and then it gives me a list of a few users to authenticate as - the names I recognise are from our IT department, so I thought it might be a root thing. I guess they've changed the default behaviour - their ethos seems to be to disable everything until enough people complain about it. Thanks for the info, anyway — Elenchus, Feb 11 '22 at 02:37

Rob_before_edits · Answer 10 · 2011-10-30T05:25:30.730

I use NX NoMachine, which is free for me because it's only me. Essentially, it runs an X session on the server which you can connect to and disconnect from over and over. The X session keeps running when you're not connected. Connections can be made from anywhere. You can choose between floating windows or a single window containing a whole desktop (eg a complete Gnome desktop). The client (which you would run on your laptop) can be run on Linux, MacOS, Solaris or Microsoft Windows. In the latter case if you choose floating windows they appear individually on the Windows Taskbar.

I use my Windows XP laptop (which I need for certain Windows-specific hardware I have) as a front end for my two Linux servers using NX Nomachine. I can even print to the printer attached to my Windows laptop from Linux.

score 3 · Answer 11 · answered Feb 09 '21 at 14:12

If you want to run the process in a non-interactive ssh session (so tmux and screen are out) you can do:

ssh SERVER 'nohup bash -c "echo start; sleep 10; echo finish"'

The process will continue if the ssh session is interrupted but you cannot reconnect to see its output.

ssh SERVER 'nohup bash -c "(echo start; sleep 10; echo finish)|tee -p /tmp/log"'

This will show you the output during the ssh session and if it disconnects tee -p will continue to write to the /tmp/log file.

Note: since the session is not interactive you can't use & to put the job into the background.

Keep processes running after SSH session disconnects

11 Answers11

Linked

Related