6

I have a remote server on which I am deploying code. I am currently using scp to upload code, but scp is very slow and I'd like to switch to rsync (or something else which is more efficient). The bottleneck is in making the ssh connection, so I'd like to minimize the number of connections I make to the server.

Locally, my build process outputs a directory out containing bin, lib and etc subdirectories. I am pushing a subset of the output files to matching positions on the remote server.

My scp commands look like this:

scp out/bin/$BINFILES remote:/usr/bin
scp out/lib/$LIBFILES remote:/usr/lib
scp out/etc/$ETCFILES remote:/etc

This makes three connections to the remote server, and copies the files in full even if they haven't been modified. This is what I did with rsync:

rsync -e ssh -avz out/bin/$BINFILES remote:/usr/bin
rsync -e ssh -avz out/lib/$LIBFILES remote:/usr/lib
rsync -e ssh -avz out/etc/$ETCFILES remote:/etc

which is faster (compression and elimination of duplicates), but still makes three connections to the remote server.

Is there a way to achieve these three copies using a single rsync command? I'm open to e.g. putting all {src,dest} pairs in a temporary file before copying, if that will work.

I'm also open to trying a different tool, but it should be available for OS X (my current development platform), and it should preferably support transfer optimizations like rsync does.

nneonneo
  • 1,088

1 Answers1

12

The reason multiple successive rsync commands are slow is that establishing an SSH connection takes a little while. You can solve this problem by using an established SSH channel. Set up master connection sharing, then open a master connection, use your rsync commands unchanged (note that -e ssh has been the default since a decade or so) and finally close the SSH connection:

ssh -Nf remote
rsync -avz out/bin/$BINFILES remote:/usr/bin
rsync -avz out/lib/$LIBFILES remote:/usr/lib
rsync -avz out/etc/$ETCFILES remote:/etc
ssh -O exit remote

If you can't use SSH connection sharing (e.g. because one side is running Dropbear or a very old version of OpenSSH), you can establish a tunnel, but this is more cumbersome.

Alternatively, with a bit of work, you may be able to refactor your rsync command into one, by playing with rsync filters and symbolic links. If you can move the bin and lib directories, it's pretty easy:

mkdir out/usr
mv out/bin out/lib out/usr/
rsync -avz \
      --include='/bin/***' --include='/lib/***' --include='/etc/***' \
      --exclude='*' out/ remote:/

If you can't move them, but there aren't any absolute symbolic links in the trees that you're copying, you can use an absolute symlink and tell rsync to treat it as a directory:

ln -s "$PWD" out/usr
rsync -avz --copy-unsafe-links \
      --include='/bin/***' --include='/lib/***' --include='/etc/***' \
      --exclude='*' out/ remote:/
Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • I knew that establishing SSH connections was the bottleneck, but I didn't know about connection sharing...nifty! Thanks a bunch! – nneonneo Jun 04 '14 at 04:34