(bash) Script A, wait for script B, but not its child process

Question

So I have scriptA which does:

ssh server1 -- scriptB &
ssh server2 -- scriptB &
ssh server3 -- scriptB &
wait
otherstuffhappens

ScriptB does:

rsync -av /important/stuff/. remoteserver:/remote/dir/.
rsync -av /not/so/important/stuff/. remoteserver:/remote/dir/. &
exit

My desired result is scriptA will wait for all the instances of scriptB to finish before moving on, which it currently does, however it's also waiting for the background rsyncs of the not so important stuff. These are larger files that I don't want to wait on.

I've read through Difference between nohup, disown and & and tried different combinations, but I'm not getting the result I'm looking for.

At this point I'm pretty stumped. Any help would be appreciated!

Answer 1

The problem here is that sshd waits for end-of-file on the pipe it is reading the command's stdout (not the stderr one for some reason, at least with the version I'm testing on) from. And the background job inherits a fd to that pipe.

So, to work around that, redirect the output of that background rsync command to some file or /dev/null if you don't care for it. You should also redirect stderr, because even if sshd is not waiting for the corresponding pipe, after sshd exits, the pipe will be broken so rsync would be killed if it tries to write on stderr.

So:

rsync ... > /dev/null 2>&1 &

Compare:

$ time ssh localhost 'sleep 2 &'
ssh localhost 'sleep 2 &'  0.05s user 0.00s system 2% cpu 2.365 total
$ time ssh localhost 'sleep 2 > /dev/null &'
ssh localhost 'sleep 2 > /dev/null &'  0.04s user 0.00s system 12% cpu 0.349 total

And:

$ ssh localhost '(sleep 1; ls /x; echo "$?" > out) > /dev/null &'; sleep 2; cat out
141  # ls by killed with SIGPIPE upon writing the error message
$ ssh localhost '(sleep 1; ls /x; echo "$?" > out) > /dev/null 2>&1 &'; sleep 2; cat out
2    # ls exited normally after writing the error on /dev/null instead
     # of a broken pipe

Answer 2

Write a script which is the parent of both, then you can easily control both. Alternatively establish a communications channel between the two channels.

To specifically ignore certain background jobs you can capture the PID ($! is the last backgrounded job PID) and wait $pid to wait only for that job to complete.

Answer 3

The -f flag to ssh fixes the problem. Tested on:

#!/bin/sh -e
echo $$_script__begin
( echo sleeping; sleep 2; echo slept )&
echo $$_script__end

When I run it with ssh localhost ./script, it waits until slept shows up. With the -f flag, it exits at echo $$_script__end and slept later shows up in the background after the ssh command has returned.

Answer 4

This is a known problem of OpenSSH server, that is described and discussed in upstream bugzilla #2071. In the bug, there are proposed several workarounds either in the OpenSSH side, but also for the script.

If you want to wait for the output of the scripts, you should add a wait before exit of the scriptB too.

If you don't care about the output, use some variation of nohup and IO redirection to /dev/null, which will resolve the problem same way.

Answer 5

You can try this.$! is default shell variable that contains the process ID of the most recently executed background pipeline/process.

command1 &
lpid1=$!

command2 &
lpid2=$!

command3 &
lpid=$!

wait $lpid1  # waits for only the process with PID lpid1  to complete. 

You need to utilize this as per your script by using export variable etc

Answer 6

B just needs to wait for it's own background processes:

rsync -av /important/stuff/. remoteserver:/remote/dir/.
rsync -av /not/so/important/stuff/. remoteserver:/remote/dir/. &
wait
exit