60

I always use either rsync or scp in order to copy files from/to a remote machine. Recently, I discovered in the manual of scp (man scp) the flag -C

 -C      Compression enable.  Passes the -C flag to
         ssh(1) to enable compression.

Before I discovered this flag, I used to zip before and then scp.

Is it as efficient to just use the -C than zipping and unzipping? When is using one or another process make the transfer faster?

Remi.b
  • 1,175
  • 5
    This is totally irrelevant to the question, but zip is a very "windows"-ish file format. You will almost never see or need it when operating a linux machine with native linux software. tar is used for rolling up directories into one file while preserving permissions and names and such, while gzip, bzip2, xz, etc are used to compress files. tars are often compressed, making tar.gz and tar.xz common formats for archives on linux. I've seen people roll their own scp workalike with commands like tar cvz directory | ssh machine 'cd somewhere; tar xz'. – Score_Under Jul 27 '15 at 17:52
  • 3
    @Score_Under - The zip format is also used by Java to package .jar files, so zip is still used extensively on many Linux servers. – Johnny Jul 27 '15 at 23:59
  • 1
    Instead of using the option on each file transfer, you can put Compression yes in your .ssh/config file. – Barmar Jul 29 '15 at 18:54
  • If you really want speed, you may be able to avoid SSH: https://unix.stackexchange.com/questions/227951/what-is-the-fastest-way-to-send-massive-amounts-of-data-between-two-computers – rogerdpack Sep 27 '18 at 22:01

3 Answers3

44

The -C flag enables a gzip compression of an SSH stream.

It's an equivalent of Accept-Encoding: gzip in HTTP.

How the flag performs depends on a kind of data you transfer:

  • When transferring a single large file, the performance would be near the same to zipping the file before the transfer (neglecting efficiency of zip vs. gzip algorithm).

    But using -C is a less effort for you as an user.

  • When transferring a lots of small files, the performance will be inferior to zipping the files before transfer.

    A reason behind that is, that before each file transfer, there's an interactive communication between the SCP server and the client (for exchanging file metadata, like timestamp and permissions). So both sides have to wait a bit for the other side to respond (compression won't help while waiting). That's a wasted time for each transferred file. How much time is wasted depends on a latency of the connection. In the end, the transfer can be magnitudes slower.

    When you transfer a single zipped file, that communication happens only once.

Martin Prikryl
  • 2,330
37

It's never really going to make any big difference, but zipping the file before copying it ought to be a little bit less efficient since using a container format such as zip that can encapsulate multiple files (like tar) is unnecessary and it is not possible to stream zip input and output (so you need a temporary file).

Using gzip on the other hand, instead of zip ought to be exactly the same since it's what ssh -C does under the hood... except that gzipping yourself is more work than just using ssh -C.

Celada
  • 44,132
  • 1
    Ok, I'll check out what gzip is. Does your answer mean that scp -rC is probably the most efficient solution I have? – Remi.b Jul 27 '15 at 17:43
  • 3
    Your answer does not consider that -C compresses an interactive protocol stream. You consider the data only. So your conclusions are wrong. See my answer – Martin Prikryl Jul 28 '15 at 06:32
  • @Celada Zip can write to a pipeline since the member directory is placed at the end. However, as you said, unzipping requires seeking to extract more than one member and so cannot read from a pipeline. – jrw32982 Jul 28 '15 at 22:53
13

It enables the gzip compression in ssh (under the scp).

On slow connections this will speed things up, on any reasonably fast connection ( 100Mbit or faster) the compression is very likely to slow things down.

It will be more or less efficient than zip based on whether gzip (specifically gzip -6) would be more or less efficient than your chosen zip compression level

Wayne Walker
  • 1,043
  • 2
    In my specific case, I have a relatively good connection (I am on campus) but the folders I have to copy are very large (~100GB over 442 .bin and .txt files). So you would suggest to just use scp -r and no -C flag and no zip, gzip not tar? – Remi.b Jul 27 '15 at 17:46
  • 3
    @Remi.b: You probably have to benchmark it both ways and see. The question is, whether the CPU is fast enough to compress the data at a faster rate than it could be sent over the network uncompressed. So the answer will depend on your particular machine and network. – Nate Eldredge Jul 28 '15 at 00:47
  • SSH itself seems to take some CPU I notice, sometimes maxing out at far below your bandwidth max. Not sure what to do there... – rogerdpack Aug 29 '18 at 16:38
  • 1
    The performance depends on the data as well. Copying a file that is essentially all zeros will be highly compressed. I have a 500Mb link between two remote servers, and just copied a 50G file (VMWare VMDK) containing all zeros over this link at ~128-130MB/s (likely some scp compression buffer limit), taking only about 6-7 minutes. Without compression this was going to take 1:45 hrs. Your mileage will vary depending on the complexity of the data and how well it can be compressed. – Topher Sep 02 '18 at 13:29