What is the fastest way to send massive amounts of data between two computers?

Question

This is a situation I am frequently in:

• I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on other machines as well, I would prefer the answer to work on any "reasonable" Linux machine)
• I have a backup server with several terabytes of hard-drive space (exact specs here, see disclaimer above)

I want to transfer 320GB of data from the source server to the target server (specifically, the data from /dev/sda).

1. The two computers are physically next to each other, so I can run cables between them.
2. I'm on a LAN, and I'm using a new-ish router, which means my network speeds should "ideally" be 1000Mbit, right?
3. Security is not an issue. I am on a local network, and I trust all machines on the network, including the router.
4. (optional) I don't necessarily need a signed checksum of the data, but basic error checking (such as dropped packets, or the drive becoming unreadable) should be detected rather than just disappear into the output.

---

I searched for this question online, and have tested several commands. The one that appears the most often is this:

ssh user@192.168.1.100 'dd bs=16M if=/dev/sda | gzip' > backup_sda.gz

This command has proven too slow (it ran for an hour, only got about 80GB through the data). It took about 1 minute and 22 seconds for the 1GB test packet, and ended up being twice as fast when not compressed. The results may also have been skewed by the fact that the transferred file is less than the amount of RAM on the source system.

Moreover (and this was tested on 1GB test pieces), I'm getting issues if I use the gzip command and dd; the resulting file has a different checksum when extracted on the target, than it does if piped directly. I'm still trying to figure out why this is happening.

Answer 1

Since the servers are physically next to each other, and you mentioned in the comments you have physical access to them, the fastest way would be to take the hard-drive out of the first computer, place it into the second, and transfer the files over the SATA connection.

Answer 2

netcat is great for situations like this where security is not an issue:

# on destination machine, create listener on port 9999
nc -l 9999 > /path/to/outfile

# on source machine, send to destination:9999
nc destination_host_or_ip 9999 < /dev/sda
# or dd if=/dev/sda | nc destination_host_or_ip 9999

Note, if you are using dd from GNU coreutils, you can send SIGUSR1 to the process and it will emit progress to stderr. For BSD dd, use SIGINFO.

pv is even more helpful in reporting progress during the copy:

# on destination
nc -l 9999 | pv > /path/to/outfile

# on source
pv /dev/sda | nc destination_host_or_ip 9999
# or dd if=/dev/sda | pv | nc destination_host_or_ip 9999

Answer 3

1. Do use fast compression.

   • Whatever your transfer medium - especially for network or usb - you'll be working with data bursts for reads, caches, and writes, and these will not exactly be in sync.
   • Besides the disk firmware, disk caches, and kernel/ram caches, if you can also employ the systems' CPUs in some way to concentrate the amount of data exchanged per burst then you should do so.
   • Any compression algorithm at all will automatically handle sparse runs of input as fast as possible, but there are very few that will handle the rest at network throughputs.
   • lz4 is your best option here:

     LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

2. Preferably do not unnecessarily seek.

   • This can be difficult to gauge.

   • If there is a lot of free space on the device from which you copy, and the device has not been recently zeroed, but all of the source file-system(s) should be copied, then it is probably worth your while to first do something like:

     </dev/zero tee >empty empty1 empty2; sync; rm empty*
     

   • But that depends on what level you should be reading the source. It is usually desirable to read the device from start to finish from its /dev/some_disk device file, because reading at the file-system level will generally involve seeking back-and-forth and around the disk non-sequentially. And so your read command should be something like:

     </dev/source_device lz4 | ...
     

   • However, if your source file-system should not be transferred entire, then reading at the file-system level is fairly unavoidable, and so you should ball up your input contents into a stream. pax is generally the best and most simple solution in that case, but you might also consider mksquashfs as well.

     pax -r /source/tree[12] | lz4 | ...
     mksquashfs /source/tree[12] /dev/fd/1 -comp lz4 | ...
     

3. Do not encrypt with ssh.

   • Adding encryption overhead to a trusted medium is unnecessary, and can be severely detrimental to the speed of sustained transfers in that the data read needs reading twice.
   • The PRNG needs the read data, or at least some of it, to sustain randomness.
   • And of course you need to transfer the data as well.
   • You also need to transfer the encryption overhead itself - which means more work for less data transferred per burst.

   • And so rather you should use netcat (or, as I prefer, the nmap project's more capable ncat) for a simple network copy, as has elsewhere been suggested:

     ###  on tgt machine...
     nc -l 9999 > out.lz4
     ###  then on src machine...
     ... lz4 | nc tgt.local 9999
     

Answer 4

There are several limitations that could be limiting the transfer speed.

1. There is inherent network overhead on a 1Gbps pipe. Usually, this reduces ACTUAL throughput to 900Mbps or less. Then you have to remember that this is bidirectional traffic and you should expect significantly less than 900Mbps down.

2. Even though you're using a "new-ish router" are you certain that the router supports 1Gbps? Not all new routers support 1Gbps. Also, unless it is an enterprise-grade router, you likely are going to lose additional transmit bandwidth to the router being inefficient. Though based on what I found below, it looks like you're getting above 100Mbps.

3. There could be network congestion from other devices sharing your network. Have you tried using a directly attached cable as you said you were able to do?

4. What amount of your disk IO are you using? Likely, you're being limited, not by the network, but by the disk drive. Most 7200rpm HDDs will only get around 40MB/s. Are you using raid at all? Are you using SSDs? What are you using on the remote end?

I suggest using rsync if this is expected to be re-run for backups. You could also scp, ftp(s), or http using a downloader like filezilla on the other end as it will parallelize ssh/http/https/ftp connections. This can increase the bandwidth as the other solutions are over a single pipe. A single pipe/thread is still limited by the fact that it is single-threaded, which means that it could even be CPU bound.

With rsync, you take out a large amount of the complexity of your solution as well as allows compression, permission preservation, and allow partial transfers. There are several other reasons, but it is generally the preferred backup method (or runs the backup systems) of large enterprises. Commvault actually uses rsync underneath their software as the delivery mechanism for backups.

Based on your given example of 80GB/h, you're getting around 177Mbps (22.2MB/s). I feel you could easily double this with rsync on a dedicated ethernet line between the two boxes as I've managed to get this in my own tests with rsync over gigabit.

Answer 5

We deal with this regularly.

The two main methods we tend to use are:

1. SATA/eSATA/sneakernet
2. Direct NFS mount, then local cp or rsync

The first is dependent on whether the drive can be physically relocated. This is not always the case.

The second works surprisingly well. Generally we max out a 1gbps connection rather easily with direct NFS mounts. You won't get anywhere close to this with scp, dd over ssh, or anything similar (you'll often get a max rate suspiciously close to 100mpbs). Even on very fast multicore processors you will hit a bottleneck on the max crypto throughput of one of the cores on the slowest of the two machines, which is depressingly slow compared to full-bore cp or rsync on an unencrypted network mount. Occasionally you'll hit an iops wall for a little while and be stuck at around ~53MB/s instead of the more typical ~110MB/s, but that is usually short lived unless the source or destination is actually a single drive, then you might wind up being limited by the sustained rate of the drive itself (which varies enough for random reasons you won't know until you actually try it) -- meh.

NFS can be a little annoying to set up if its on an unfamiliar distro, but generally speaking it has been the fastest way to fill the pipes as fully as possible. The last time I did this over 10gbps I never actually found out if it maxed out the connection, because the transfer was over before I came back from grabbing some coffee -- so there may be some natural limit you hit there. If you have a few network devices between the source and destination you can encounter some slight delays or hiccups from the network slinky effect, but generally this will work across the office (sans other traffic mucking it up) or from one end of the datacenter to the other (unless you have some sort of filtering/inspection occurring internally, in which case all bets are off).

EDIT

I noticed some chatter about compression... do not compress the connection. It will slow you down the same way a crypto layer will. The bottleneck will always be a single core if you compress the connection (and you won't even be getting particularly good utilization of that core's bus). The the slowest thing you can do in your situation is to use an encrypted, compressed channel between two computers sitting next to each other on a 1gbps or higher connection.

FUTURE PROOFING

This advice stands as of mid-2015. This will almost certainly not be the case for too many more years. So take everything with a grain of salt, and if you face this task regularly then try a variety of methods on actual loads instead of imagining you will get anything close to theoretical optimums, or even observed compression/crypto throughput rates typical for things like web traffic, much of which is textual (protip: bulk transfers usually consist chiefly of images, audio, video, database files, binary code, office file formats, etc. which are already compressed in their own way and benefit very little from being run through yet another compression routine, the compression block size of which is almost guaranteed to not align with your already compressed binary data...).

I imagine that in the future concepts like SCTP will be taken to a more interesting place, where bonded connections (or internally bonded-by-spectrum channelized fiber connections) are typical, and each channel can receive a stream independent of the others, and each stream can be compressed/encrypted in parallel, etc. etc. That would be wonderful! But that's not the case today in 2015, and though fantasizing and theorizing is nice, most of us don't have custom storage clusters running in a cryo-chamber feeding data directly to the innards of a Blue Gene/Q generating answers for Watson. That's just not reality. Nor do we have time to analyze our data payload exhaustively to figure out whether compression is a good idea or not -- the transfer itself would be over before we finished our analysis, regardless how bad the chosen method turned out to be.

But...

Times change and my recommendation against compression and encryption will not stand. I really would love for this advice to be overturned in the typical case very soon. It would make my life easier.

Answer 6

A nifty tool that I've used in the past is bbcp. As seen here: https://www.slac.stanford.edu/~abh/bbcp/ .

See also http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm

I've had very fast transfer speeds with this tool.

Answer 7

If you get a first pass somehow (over the wire/sneakernet/whatever), you can look into rsync with certain options that can greatly speed up subsequent transfers. A very good way to go would be:

rsync -varzP sourceFiles destination

Options are: verbose, archive mode, recursive, compress, Partial progress

Answer 8

Added on insistence of the original poster in comments to zackse’s answer, although I’m not sure it is the fastest in typical circumstances.

bash has a special redirection syntax:
For output:     > /dev/tcp/ IP / port
For input:      < /dev/tcp/ IP / port
IP ban be either dotted-decimal IP or a hostname; port ban be either a decimal number or a port name from /etc/services.

There is no actual /dev/tcp/ directory. It’s a special syntactic kludge that commands bash to create a TCP socket, connect it to the destination specified, and then do the same thing as a usual file redirection does (namely, replace the respective standard stream with the socket using dup2(2)).

Hence, one can stream data from dd or tar at the source machine directly via TCP. Or, conversely, to stream data to tar or something alike directly via TCP. In any case, one superfluous netcat is eliminated.

Notes about netcat

There is an inconsistency in syntax between classical netcat and GNU netcat. I’ll use the classical syntax I’m accustomed to. Replace -lp with -l for GNU netcat.

Also, I’m unsure whether does GNU netcat accept -q switch.

Transferring a disk image

(Along the lines of zackse’s answer.)
On destination:

nc -lp 9999 >disk_image

On source:

dd if=/dev/sda >/dev/tcp/destination/9999
 

Creating a tar.gz archive, with tar

On destination:

nc -lp 9999 >backup.tgz

On source:

tar cz files or directories to be transferred >/dev/tcp/destination/9999

Replace .tgz with .tbz and cz with cj to get a bzip2-compressed archive.

Transferring with immediate expansion to file system

Also with tar.
On destination:

cd backups
tar x </dev/tcp/destination/9999

On source:

tar c files or directories to be transferred |nc -q 1 -lp 9999

It will work without -q 1, but netcat will stuck when data ended. See tar(1) for explanation of the syntax and caveats of tar. If there are many files with high redundancy (low entropy), then compression (e. g. cz and xz instead of c and x) can be tried, but if files are typical and the network is fast enough, it would only slow the process. See mikeserv’s answer for details about compression.

Alternative style (the destination listens port)

On destination:

cd backups
nc -lp 9999 |tar x

On source:

tar c files or directories to be transferred >/dev/tcp/destination/9999

Answer 9

Try the suggestions regarding direct connections and avoiding encrypted protocols such as ssh. Then if you still want to eke out every bit of performance give this site a read: https://fasterdata.es.net/host-tuning/linux/ for some advice on optimizing your TCP windows.

Answer 10

I would use this script I wrote that needs the socat package.

On the source machine:

tarnet -d wherefilesaretosend pass=none 12345 .

On the target machine:

tarnet -d wherefilesaretogo pass=none sourceip/12345

If the vbuf package (Debian, Ubuntu) is there then the file sender will show a data progress. The file receiver will show what files are received. The pass= option can be used where the data might be exposed (slower).

Edit:

Use the -n option to disable compression, if CPU is a bottle neck.

Answer 11

If budget is not the main concern, you could try connecting the drives with a Intel Xeon E5 12 core "drive connector". This connector is usually so powerful, that you can even run your current server software on it. From both servers!

This might look like a fun answer, but you should really consider why you are moving the data between servers and if a big one with shared memory and storage might make more sense.

Not sure about current specs, but the slow transfer might be limited by the disk speeds, not the network?

Answer 12

If you only care about backups, and not about a byte for byte copy of the hard drive, then I would recommend backupPC. http://backuppc.sourceforge.net/faq/BackupPC.html It's a bit of a pain to setup but it transfers very quickly.

My initial transfer time for about 500G of data was around 3 hours. Subsequent backups happen in about 20 seconds.

If your not interested in backups, but are trying to sync things then rsync or unison would better fit your needs.

A byte for byte copy of a hard disk is usually a horrid idea for backup purposes (no incrementals, no space saving, drive can't be in use, you have to backup the "empty space", and you have to back up garbage (like a 16 G swap file or 200G of core dumps or some such). Using rsync (or backuppc or others) you can create "snapshots" in time so you can go to "what your file system looked like 30 mins ago" with very little overhead.

That said, if your really want to transfer a byte for byte copy then your problem is going to lie in the transfer and not in the getting data from the drive. With out 400G of RAM a 320G file transfer is going to take a very ling time. Using protocols that are not encrypted are an option, but no matter what, your just going to have to sit there and wait for several hours (over the network).

Answer 13

Regardless of program, I have usually found that "pulling" files over a network is faster than "pushing". That is, logging into the destination computer and doing a read is faster than logging into the source computer and doing a write.

Also, if you are going to use an intermediate drive, consider this: Get an external drive (either as a package, or a separate drive plugged into a docking station) which uses eSATA rather than USB. Then on each of the two computers either install a card with an eSATA port, or get a simple adapter cable which brings one of the internal SATA ports to an external eSATA connector. Then plug the drive into the source computer, power up the drive, and wait for it to auto-mount (you could mount manaully, but if you are doing this repeatedly you might as well put it into your fstab file). Then copy; you will be writing at the same speed as to an internal drive. Then unmount the drive, power down, plug into the other computer, power up, wait for an auto-mount, and read.

Answer 14

How about an ethernet crossover cable? Instead of relying on wireless speeds you're capped at the wired speed of your NIC.

Here's a similar question with some examples of that kind of solution.

Apparently just a typical ethernet cable will suffice nowadays. Obviously the better your NIC the faster the transfer.

To summarize, if any network setup is necessary, it should be limited to simply setting static IPs for your server and backup computer with a subnet mask 255.255.255.0

Good luck!

Edit:

@Khrystoph touched on this in his answer

Answer 15

I'm going to recommend that you look at NIC-teaming. This involves using multiple network connections running in parallel. Assuming that you really need more than 1Gb transfer, and that 10Gb is cost prohibitive, 2Gbs provided by NIC-teaming would be a minor cost, and your computers may already have the extra ports.

Answer 16

FWIW, I've always used this:

tar -cpf - <source path> | ssh user@destserver "cd /; tar xf -"

Thing about this method is that it will maintain file/folder permissions between machines (assuming the same users/groups exist on both) (Also I typically do this to copy virtual disk images since I can use a -S parameter to handle sparse files.)

Just tested this between two busy servers and managed ~14GB in 216s (about 64MB/s) - might would do better between dedicated machines and/or compression... YMMV

$ date; tar -cpf - Installers | ssh elvis "cd /home/elvis/tst; tar xf -"; date
Wed Sep  9 15:23:37 EDT 2015
Wed Sep  9 15:27:13 EDT 2015

$ du -s Installers
14211072   Installers

Answer 17

Unless you want to do filesystem forensics, use a dump/restore program for your filesystem to avoid copying the free space that the FS isn't using. Depending on the what filesystem you have, this will typically preserve all metadata, including ctime. inode numbers may change, though, again depending on what filesystem (xfs, ext4, ufs...).

The restore target can be a file on the target system.

If you want a full-disk image with the partition table, you can dd the first 1M of the disk to get the partition table / bootloaders / stuff, but then xfsdump the partitions.

I can't tell from your info-dump what kind of filesystem you actually have. If it's BSD ufs, then I think that has a dump/restore program. If it's ZFS, well IDK, there might be something.

Generally full-copying disks around is too slow for anything except recovery situations. You can't do incremental backups that way, either.

Answer 18

You could also setup the systems to have a shared storage!

I am considering that these are next to each other, and you are likely to do this again & again ....

Answer 19

Several folks recommend that you skip ssh because encryption will slow you down. Modern CPUs may actually be fast enough at 1Gb, but OpenSSH has problems with its internal windowing implementation that can drastically slow you down.

If you want to do this with ssh, take a look at HPN SSH. It solves the windowing problems and adds multithreaded encryption. Unfortunately you'll need to rebuild ssh on both the client & server.

Answer 20

OK I have attempted to answer this question for two computers with "very large pipes" (10Gbe) that are "close" to each other.

The problem you run into here is: most compression will bottleneck at the cpu, since the pipes are so large.

performance to transfer 10GB file (6 Gb network connection [linode], uncompressible data):

$  time bbcp 10G root@$dest_ip:/dev/null
0m16.5s 

iperf:

server: $ iperf3 -s -F /dev/null
client:
$ time iperf3 -c $dest_ip -F 10G -t 20 # -t needs to be greater than time to transfer complete file
0m13.44s
(30% cpu)

netcat (1.187 openbsd):

server: $ nc -l 1234 > /dev/null
client: $ time nc $dest_ip 1234 -q 0 < 10G 
0m13.311s
(58% cpu)

scp:

$ time /usr/local/bin/scp 10G root@$dest_ip:/dev/null
1m31.616s
scp with hpn ssh patch (scp -- hpn patch on client only, so not a good test possibly): 
1m32.707s

socat:

server:
$ socat -u TCP-LISTEN:9876,reuseaddr OPEN:/dev/null,creat,trunc
client:
$ time socat -u FILE:10G TCP:$dest_ip:9876
0m15.989s

And two boxes on 10 Gbe, slightly older versions of netcat (CentOs 6.7), 10GB file:

nc: 0m18.706s (100% cpu, v1.84, no -q option
iperf3: 0m10.013s (100% cpu, but can go up to at least 20Gbe with 100% cpu so not sure it matters)
socat: 0m10.293s (88% cpu, possibly maxed out)

So on one instance netcat used less cpu, on the other socat, so YMMV.

With netcat, if it doesn't have a "-N -q 0" option it can transfer truncated files, be careful...other options like "-w 10" can also result in truncated files.

What is happening in almost all of these cases is the cpu is being maxed out, not the network. scp maxes out at about 230 MB/s, pegging one core at 100% utilization.

Iperf3 unfortunately creates corrupted files. Some versions of netcat seem to not transfer the entire file, very weird. Especially older versions of it.

Various incantations of "gzip as a pipe to netcat" or "mbuffer" also seemed to max out the cpu with the gzip or mbuffer, so didn't result in faster transfer with such large pipes. lz4 might help. In addition, some of the gzip pipe stuff I attempted resulted in corrupted transfers for very large (> 4 GB) files so be careful out there :)

Another thing that might work especially for higher latency (?) is to tune tcp settings. Here is a guide that mention suggested values:

http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm and https://fasterdata.es.net/host-tuning/linux/ (from another answer) possibly IRQ settings: https://fasterdata.es.net/host-tuning/100g-tuning/

suggestions from linode, add to /etc/sysctl.conf:

net.core.rmem_max = 268435456 
net.core.wmem_max = 268435456 
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_no_metrics_save = 1
net.core.default_qdisc = fq 

Additionally, they would like you to run:

 /sbin/ifconfig eth0 txqueuelen 10000 

worth double checking after tweaking to make sure changes don't cause harm too.

Also may be worth tuning the window size: https://iperf.fr/iperf-doc.php#tuningtcp

With slow(er) connections compression can definitely help though. If you have big pipes, very fast compression might help with readily compressible data, haven't tried it.

The standard answer for "syncing hard drives" is to rsync the files, that avoids transfer where possible.

Another option: use "parallel scp" (somehow or other), then it will use more cores...