5

I'd like to transfer a directory between two servers, but compress the directory on the remote host before transfer, and then uncompress to another host. I'm sure it's possible to pipe everything all the way through and do it in a one liner.

I realise it would be better of course if I could transfer between the hosts directly but that would involve transferring keys and what not, and I love Unix one line power tools. I'm sure people can come up with a few different ways to do this. I'm looking for the shortest syntax and most bandwidth conservative.

To start off I have 

ssh -n REMOTEHOST 'tar zcvf - DIRTOCOPY' | localZip.tar.gz
0xC0000022L
  • 16,593
barrymac
  • 1,115
  • Why not replace the local cat with a (cd dest; tar xvzf -)? – jw013 Aug 16 '12 at 14:16
  • Can we put another remote ssh before those brackets? ssh -n REMOTEHOST1 'tar zcvf - DIRTOCOPY' | ssh -n REMOTEHOST2 (cd dest; tar xvzf -) – barrymac Aug 16 '12 at 14:19
  • why not to use rsync? it can compress on the fly.. – rush Aug 16 '12 at 14:21
  • @lk yep that's one shortener, will edit – barrymac Aug 16 '12 at 14:25
  • @rush Rsync isn't as bandwidth efficient as a local tar before putting it on the wire especially for the remote1 to remote 2 scenario. Correct me if I'm wrong though – barrymac Aug 16 '12 at 14:26
  • 2
    Something like ssh -n host1 'tar cvzf - dir' | ssh -n host2 'cd dest; tar xvzf -'? – jw013 Aug 16 '12 at 14:35
  • 4
    @barrymac from man rsync about -z option : Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection. – rush Aug 16 '12 at 14:42
  • @rush very good, didn't realise rsync was that smart! – barrymac Aug 16 '12 at 15:20

3 Answers3

8

Similar to what jw013 suggested in the comments with separate compression/decompression steps, i.e. combine two ssh commands with a pipe:

compress=gzip
decompress=gunzip

ssh remote1 "cd srcdir; tar cf - dir | $compress" |
ssh remote2 "cd destdir; $decompress | tar xvf -"

Note that the compression/decompression is configurable without depending on the version of tar.

Update

You could also add checksum verification into the pipe:

compress=gzip
decompress=gunzip

ckprg=md5sum
cksum=/tmp/cksum

ssh remote1 "cd srcdir; tar cf - dir | $compress | tee <($ckprg > $cksum)" |
ssh remote2 "cd destdir; tee <($ckprg > $cksum) | $decompress | tar xvf -"

ssh remote1 cat $cksum
ssh remote2 cat $cksum
Thor
  • 17,182
2

Your transfer would be faster if you could establish a direct connection between the two hosts. But lacking that, the simplest way is to use cp. First mount the remote filesystems using sshfs

mkdir ~/net ~/net/sourcehost ~/net/destinationhost
sshfs sourcehost: ~/net/sourcehost
sshfs destinationhost: ~/net/destinationhost
cp -Rp ~/net/sourcehost/path/to/source ~/net/destinationhost/path/to/destination

Be sure to activate compression in your ~/.ssh/config:

Host sourcehost
HostName sourcehost.example.com
Compression yes
CompressionLevel 9

Host destinationhost
HostName destinationhost.example.com
Compression yes
CompressionLevel 9
Gilles 'SO- stop being evil'
  • 829,060
  • well that doesn't use compression, and I specified that I actually do want to do it via the local host. It's also a lot more hassle to set up sshfs than a one liner I can pull out of the tool box for a one off situation – barrymac Aug 17 '12 at 09:12
  • 1
    @barrymac Compression yes ensures that it uses compression. SSHFS requires one line of permanent setup (mkdir), one line of per-session setup per host (mount), and the actual copying is trivial (cp) and doesn't even require you to remember that the files you're copying are remote. – Gilles 'SO- stop being evil' Aug 17 '12 at 09:25
  • Sorry didn't spot that compression flag, and I was thinking of the hassle of installing it and what not. Would be interesting to benchmark that method. It's not quite the one liner I was thinking of but it's handy alright – barrymac Aug 17 '12 at 09:50
1

Your proposed answer:

ssh -n REMOTEHOST 'tar zcvf - DIRTOCOPY' | localZip.tar.gz

did not work for me - the pipe to a file failed.

I did this instead and it worked:

ssh -n REMOTEHOST 'tar zcvf - DIRTOCOPY' | cat - > localZip.tar.gz

Pipe it to 'cat' via standard input and redirect the output to the file.

another solution would be to remove the "| cat -" and just send the SSH output directly to the tarball:

ssh -n REMOTEHOST 'tar zcvf - DIRTOCOPY' > localZip.tar.gz
Community
  • 1
Mistere
  • 21