118

This is a situation I am frequently in:

  • I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on other machines as well, I would prefer the answer to work on any "reasonable" Linux machine)
  • I have a backup server with several terabytes of hard-drive space (exact specs here, see disclaimer above)

I want to transfer 320GB of data from the source server to the target server (specifically, the data from /dev/sda).

  1. The two computers are physically next to each other, so I can run cables between them.
  2. I'm on a LAN, and I'm using a new-ish router, which means my network speeds should "ideally" be 1000Mbit, right?
  3. Security is not an issue. I am on a local network, and I trust all machines on the network, including the router.
  4. (optional) I don't necessarily need a signed checksum of the data, but basic error checking (such as dropped packets, or the drive becoming unreadable) should be detected rather than just disappear into the output.

I searched for this question online, and have tested several commands. The one that appears the most often is this:

ssh user@192.168.1.100 'dd bs=16M if=/dev/sda | gzip' > backup_sda.gz

This command has proven too slow (it ran for an hour, only got about 80GB through the data). It took about 1 minute and 22 seconds for the 1GB test packet, and ended up being twice as fast when not compressed. The results may also have been skewed by the fact that the transferred file is less than the amount of RAM on the source system.

Moreover (and this was tested on 1GB test pieces), I'm getting issues if I use the gzip command and dd; the resulting file has a different checksum when extracted on the target, than it does if piped directly. I'm still trying to figure out why this is happening.

IQAndreas
  • 10,345
  • 55
  • @gwillie Believe me, it was an option I considered, especially after ssh was being too slow, I just didn't have enough free space on the external harddrive. – IQAndreas Sep 07 '15 at 04:45
  • 4
    Do you want to transfer /dev/sda as an image or just the files. Why is rsync no option? Is /dev/sda mounted while you dded? – Jodka Lemon Sep 07 '15 at 05:48
  • 15
    Your performance data (1GB/80sec, 80GB/1h) match perfectly what we should expect on 100MBit. Check your hardware. ... and gerrit is right, 320GB may be large, but "massive amount of data" raises wrong expectations. – blafasel Sep 07 '15 at 13:09
  • @blafasel 100mbps is also very close to what I usually observe as the processor's max throughput on an encrypted connection (at least ssh/scp). Generally it doesn't matter how fast over 100mpbs your connection is if you're over ssh: the processor's throughput limits you to something very close to 100mpbs due to crypto overhead (and this is not a trivially parallelizable load). – zxq9 Sep 07 '15 at 13:48
  • 1
    So gzip makes transfer slower, not faster. Maybe the data is too random to be effectively compressed-- did you check the compression ratio on those 80 GB? But clearly the compression speed can't keep up with the maximum capacity of the channel, so you're effectively throttling it. The problem will get worse if you set up a faster channel-- just leave out the compression. Maybe you can also leave out the secure connection (i.e. ssh), since you trust the network. – alexis Sep 07 '15 at 16:14
  • You need to find and fix your data corruption issue. Who cares how fast you can transfer it, if it isn't intact? First try to narrow it down to a machine, e.g., if you gzip it on one machine and ungzip it on the same machine, does the checksum match? Repeat on the other machine. If only one of the machines gives different checksums, you now know which is broken. If in both, then ask another question about it---possibly you're doing something wrong, or possibly they're both broken. Once you know which machines are broken, proceed to hardware troubleshooting (e.g., memtest86). – derobert Sep 07 '15 at 17:56
  • 1
    NFS protocol has smaller overhead compared to ssh for example, I would use NFS if using eSATA between the two machines is not an option. – jet Sep 07 '15 at 19:08
  • High Performance SSH/SCP is what you are looking for if you wish to do SSH/SCP data transfer in your given scenario. You can disable encryption with it, but still have the integrity checking, and authentication. If your data has a large amount of entropy, then skip on the compression, if it doesn't, use something like lz4 rather than gzip. Either the encryption is killing your transfer speed, there is a poor connection, or the router can't route at 1Gbps (if you are routing rather than just switching), ideally you would want this switched. – Phizes Sep 08 '15 at 01:42
  • Like alexis said, gzip is slow. Leave it out; you are only hurting yourself by trying to use it. Also, make sure your ssh session doesn't try to compress the data stream: ssh -o Compression=no. In between these two, and with a fast cipher (on modern hardware 128-bit AES, on ancient hardware RC4, or since you trust both the systems and the link even none), the overhead of the SSH pipe should be minimal. – user Sep 08 '15 at 08:15
  • I've had good success with this approach: http://intermediatesql.com/linux/scrap-the-scp-how-to-copy-data-fast-using-pigz-and-nc/ – Freiheit Sep 08 '15 at 14:13
  • 1
    @blafasel In this particular case, I am transferring only 320GB, but I want methods that theoretically should work even if I transfer larger amounts, such as a 4TB image, hence "massive amounts of data". – IQAndreas Sep 08 '15 at 14:30
  • My observations on ssh connections differ from those zxq9 has. I often reach the theoretical maximum of my network hardware - even 10Gb - without hitting any cpu limits. I suppose you are reaching your bandwidth limit which gets the real bottleneck. Go with Phizes and deactivate compression and maybe encryption - just as a test. I expect your cpu to idle while you will reach the same througput. – blafasel Sep 08 '15 at 14:39
  • @blafasel "Your performance data match perfectly what we should expect on 100MBit" - Except I'm on a LAN which supports 1000MBit, and all of the hardware is fairly recent. The netcat suggestion by zackse ended up being much faster than SSH. (I'll write a blog post with the exact speed comparisons in a few days) – IQAndreas Sep 08 '15 at 16:49
  • 8
    "Never underestimate the bandwidth of a freight train full of disks." .. Are you asking about throughput, latency, or some mix of the two? – keshlam Sep 08 '15 at 21:19
  • If you're doing this very frequently, is it perhaps worth it to put the data in some external database and only pull the data you need with a query? – corsiKa Sep 08 '15 at 22:16
  • @keshlam (data) bandwidth refers to bits/second, not latency. – Nick T Sep 08 '15 at 22:31
  • @nickt: I know that, obviously. It isn't clear, however, which of the two the OP was asking about. – keshlam Sep 08 '15 at 22:46
  • 2
    @keshlam Irrelevant. I want the to get exactly 320GB from point A to point B in the fastest time possible. It doesn't matter if it's high latency (transferring data to an external harddrive and then plugging the drive in the other computer) or low throughput (sending data over the LAN). By "fastest", I mean that all I care about is how many seconds until the process is completed. – IQAndreas Sep 09 '15 at 00:11
  • 2
    It does matter whether you're including setup time or not. If latency is included in the total, the optimal answer is probably different from when it isn't. If you're counting time before you can start reading and processing the data, that's different from counting time until the last bit arrives, and again the optimal solution may shift. – keshlam Sep 09 '15 at 00:24
  • 2
    Sneakernet was the first thing that came to mind while reading the absolutist nature of the question, though I didn't know it's called that. Related is RFC2549 – phresnel Sep 09 '15 at 07:07
  • 2
    Another thing to consider: right now for some reason there are gobs and gobs of super cheap (20 USD) mellanox connectx 10G NICs on ebay. If you have a free PCIe slot on both ends, then a pair of 10G NICs and a direct attach cable would significantly increase your available network bandwidth. – alex.forencich Sep 09 '15 at 08:50
  • Setup windows clustered storage and have both computers share the volume? – Andy Sep 09 '15 at 21:49
  • 8
    A friend of mine always said: "Never underestimate the bandwidth of a pile of hard drives on a truck". – AMADANON Inc. Sep 10 '15 at 02:04
  • One big detail that's missing in your question: what are you using the backup for? A disk/parition image (such as a dd image) is great for cloning machines, swapping drives, and 100% backups. An archive (such as atarfile, or evenrsync`) is great if you'll need to retrieve portions of your backup but not the entire thing. – STW Sep 10 '15 at 13:29
  • @STW In this particular case, I'm doing "deleted file recovery", so I need the dd image, but I occasionally run into the same type of problem when transferring very large tar images, or even other files. – IQAndreas Sep 10 '15 at 15:09
  • Have you considered hardware copying? http://www.newegg.com/Product/Product.aspx?Item=N82E16817707370&cm_re=hard_drive_duplicator--17-707-370--Product – Martin York Sep 11 '15 at 00:48
  • The specific command in the OP does the compression on the target machine, AFTER it has been transferred over the network. This would be better: dd bs=16M if=/dev/sda | gzip | ssh user@192.168.1.100 'cat - > backup_sda.gz'. Caveats about ssh and compression speed still apply. If you have multiple cores to devote to the task, try pigz instead of gzip. – Dan Pritts Sep 14 '15 at 14:08
  • Why is this closed as opinion-based? There is a very clear metric that can potentially rank the best answer. The question explicitly asks for an answer based on a quantitative measure. – Sparhawk Sep 28 '18 at 03:06
  • 1
    The super-fastest way is: On source server buffer -s 8192 -b 2048 -i /dev/sda|nc target.ip 64738. On the target server: nc -p 64738 -l|buffer -s 8192 -b 2048 -o /dev/targethd. It parallelizes everything (including disk & network io), doesn't compress anything (you don't need it here). All the other answers, including the ones in the comments, have at most half the speed than this, mainly for their single-thread executions and internal locks. – peterh Sep 28 '18 at 04:10
  • Here the answers saying any compression/encryption are inherently bad, they cause an unacceptable speed decrease in your case. The dd-based answers are still sub-optimal, because dd is singlethreaded (disk block read, wait, network write, wait, goto 1, this will be the cycle), but they are better. But the only command-line tool which can parallelize the network communication with the disk i/o, is the buffer. Doing this optimally with any single-threaded tool is impossible. – peterh Sep 28 '18 at 04:15
  • Btw, if you compress the image on the target, then you don't need quick network communication, because the compression speed will be the weak point. Btw, you likely don't need to compress & archive the data in your free blocks, so you would make it better if you would tar the data on the source and would use again a buffer-based solution on the target. | The answer saying the physical move of harddisk is also bad, despite its 100+ score,because SATA hard disks have around 60 MB/sec read speed, while gigabit ethernet can pass ideally 120MB/sec (you need also good switch and NICs for that!). – peterh Sep 28 '18 at 04:21
  • I'm losing my mind, I can read at 1.5GB/s, transfer (iperf, tcp) at 1.1GB/s, and write at 1.8GB/s; but put the three together and I get 0.3GB/s. where is the bottle neck? dd if=/dev/xvda bs=10M status=progress | nc -q0 dest.localdomain 1234 nc -l -p 1234 </dev/null | dd of=./xvda.img bs=10M status=progress. I know: iperf over ssh ~1.25GB/s, ddzero-to-ddnull shell pipe ~0.4GB/s, ddzero-to-ddnull over ssh ~0.1GB/s, ddzero-to-ddnull over local nc-to-nc ~1GB/s, too slow: sha512sum ~ 0.25GB/s, md5sum ~ 0.5GB/s, gzip ~0.025GB/s, zstdmt ~0.6GB/s, gpg ~0.1GB/s, gpg nocompress ~0.19GB/s, – ThorSummoner Feb 19 '21 at 20:20

20 Answers20

141

Since the servers are physically next to each other, and you mentioned in the comments you have physical access to them, the fastest way would be to take the hard-drive out of the first computer, place it into the second, and transfer the files over the SATA connection.

  • 16
    +1: Transferring via physical seems to be the fastest route, even if it means getting a big external hard drive from somewhere. It's about £40, and you've probably spent that much in time already, – deworde Sep 07 '15 at 10:06
  • I agree. It seems that one of the "spare" 1 TB units could be disconnected from Gallant Gorilla and the 298 GB unit 0 from Dramatic Dingo could be moved into the empty slot. However, this would probably require shutting at least one server down for the duration, which may be undesirable. And it still won't be super-fast with spindle drives, but you might cut half the time off a network transfer. – MichaelS Sep 07 '15 at 10:23
  • @MichaelS SATA drives are quite easy to hotswap, so that would be an option - SCSI is a bit trickier, though. – Luaan Sep 07 '15 at 10:41
  • From Newegg, it looks like the RAID controllers do support hot-swapping. However, the fdisk page on the specs says the 300 GB drive is SDA. I'm not terribly familiar with Unix servers, but my assumption was this is the system drive. If so, the system wouldn't work while that drive is in the other machine. – MichaelS Sep 07 '15 at 11:12
  • The eSATA connector has a design-life of 5,000 matings; the ordinary SATA connector is only specified for 50. -- wiki – A.D. Sep 07 '15 at 17:46
  • This is not ideal for "frequent" transfers. Tape jockeys are a thing of the past. –  Sep 07 '15 at 22:05
  • 3
    I completely disagree with this idea if one is getting full speed across a gigabit network. Testing over NFS/SMB over a Zyxel Gigabit switch between an HP Gen 7 microserver, and a Pentium G630 machine gives me ~100MB/s transfer. (Until I leave the outer edge of the drive platters.) So I think it'd realistically be done in under 3 hours. Unless you are using SSD's or extremely high performance drives/storage, I don't think 2 copies can produce a throughput of 100MB/s, that would require each copy operation to be 200MB/s just to break even. – Phizes Sep 08 '15 at 01:52
  • 3
    @Phizes: obviously you don't copy to a temporary. That was deword's bad idea, not what everyone else is talking about. The point of connecting the source drive to the target machine is to go SATA->SATA with dd (or a filesystem tree copy). – Peter Cordes Sep 09 '15 at 21:12
  • 10
    "Never underestimate the bandwidth of a truck full of hard drives. One hell of a latency though" – Kevin Sep 11 '15 at 04:31
  • @PeterCordes I think that there might be a few comments missing between when I commented, and you responded. Apart from that I see I only covered a temporary drive, furthering my point more in accordance to the exact answer provided above, over how much of the 320Gb platter will a SATA HDD be able to exceed a gigabit of bandwidth? I can't find details for those exact drives, but (I think) similar drives at the time for the 320's start in the mid 80MB/s, and quickly drop below even mid 50MB/s. – Phizes Sep 12 '15 at 21:19
  • @Phizes: you weren't replying to anyone specifically, so I assumed you were commenting on BlueRaja's answer. He's saying get both SATA drives connected to one computer's SATA controllers. Good point about 320GB drives being slow, though. It's a good answer for the general case, and future-proof, because current 7200RPM drives (like my Toshiba 3TB) read at ~190MB/s. You can lz4 on the fly to get more throughput from the same 1Gb network bandwidth, gigE is a bottleneck for modern drives. With RAID or SSDs, even 2 ports aggregated (with a fancy switch supporting LACP) is a 200MB/s bottleneck. – Peter Cordes Sep 12 '15 at 21:28
  • 1
    @PeterCordes: You're bringing in more technical detail than is actually necessary. Think about it. You have to go through SATA regardless of how you transfer the files, so SATA -> SATA will be at least as fast as any other method since either a) SATA is the bottleneck and they're both equally slow or b) SATA is not the bottleneck and is therefore faster. – Kevin Sep 13 '15 at 03:36
  • 3
    @Kevin: yes, my point was that a direct copy between disks in the same computer is at least as fast as any other possible method. I brought up real-life bandwidth numbers to acknowledge Phize's point that going over gigE is fine for the OPs old drive, but a bottleneck for new drives. (One case where both drives in one computer is not the best option is when having separate computers using their RAM to cache the metadata of the source and dest is important, e.g. for rsync of billions of files.) – Peter Cordes Sep 13 '15 at 03:59
  • Instead of moving the source drive on the target machine, bring a drive from the target machine in the source machine and transfer there. You may find the use of a cheap SATA dock/rack (to put in a normal 5.25 bay normally used for CD/DVD/BR drives) very convenient. This keeps the source system online, and (technically off-topic) in the extremely common case of backups you don't even need the data to be processed by anything after that (no need to attach on a second machine). – tne Jan 13 '16 at 11:36
72

netcat is great for situations like this where security is not an issue:

# on destination machine, create listener on port 9999
nc -l 9999 > /path/to/outfile

# on source machine, send to destination:9999
nc destination_host_or_ip 9999 < /dev/sda
# or dd if=/dev/sda | nc destination_host_or_ip 9999

Note, if you are using dd from GNU coreutils, you can send SIGUSR1 to the process and it will emit progress to stderr. For BSD dd, use SIGINFO.

pv is even more helpful in reporting progress during the copy:

# on destination
nc -l 9999 | pv > /path/to/outfile

# on source
pv /dev/sda | nc destination_host_or_ip 9999
# or dd if=/dev/sda | pv | nc destination_host_or_ip 9999
zackse
  • 1,523
  • 9
  • 10
  • 2
    For the second example, is dd even required, or can pv/nc treat /dev/sda just fine on their own? (I have noticed some commands "throw up" when trying to read special files like that one, or files with 0x00 bytes) – IQAndreas Sep 07 '15 at 04:07
  • 1
    In basic testing, both netcat and pv could read the block device without issue. You could also use shell redirection: pv < /dev/sda | nc destination 9999. – zackse Sep 07 '15 at 04:14
  • Also, the pv manpage gives an example of addressing a block device as the FILE command-line argument. See the DESCRIPTION section, or search for /dev/sda: http://www.ivarch.com/programs/quickref/pv.shtml – zackse Sep 07 '15 at 04:23
  • 1
    Can we get some compression in the pipeline? – user1794469 Sep 07 '15 at 05:21
  • 5
    @user1794469 Will the compression help? I'm thinking the network is not where the bottleneck is. – IQAndreas Sep 07 '15 at 06:40
  • 1
    To speed things up a little bit more you could run a direct cable between the two Gigabit Ethernet cards and not go through a switch, then configure the ports to use static IP addresses (with a routing entry). – Ned64 Sep 07 '15 at 09:19
  • 1
    @user1794469 You can compress easily with e.g. lzop which is very fast and does not use much CPU. Check your CPU-load to avoid being limited by CPU instead of network. – jofel Sep 07 '15 at 16:23
  • 17
    Don’t forget that in bash one can use > /dev/tcp/IP/port and < /dev/tcp/IP/port redirections instead of piping to and from netcat respectively. – Incnis Mrsi Sep 07 '15 at 16:41
  • 6
    Good answer. Gigabit Ethernet is often faster than hard drive speed, so compression is useless. To transfer several files consider tar cv sourcedir | pv | nc dest_host_or_ip 9999 and cd destdir ; nc -l 9999 | pv | tar xv. Many variations are possible, you might e.g. want to keep a .tar.gz at destination side rather than copies. If you copy directory to directory, for extra safety you can perform a rsync afterwards, e.g. from dest rsync --inplace -avP user@192.168.1.100:/path/to/source/. /path/to/destination/. it will guarantee that all files are indeed exact copies. – Stéphane Gourichon Sep 08 '15 at 09:08
  • 3
    Instead of using IPv4 you can achieve a better throughput by using IPv6 because it has a bigger payload. You don't even configure it, if the machines are IPv6 capable they probably already have an IPv6 link-local address – David Costa Sep 08 '15 at 13:07
  • 1
    @IncnisMrsi Could you add that solution as an answer? I'd like you to expand on it, and see the full code for it. – IQAndreas Sep 09 '15 at 00:15
  • 1
  • Regarding pv: NO, NO, and NO!!! The question was about fastest, and pv will hurt you badly! Check out my answer to this question: http://askubuntu.com/questions/215505/how-do-you-monitor-the-progress-of-dd (search in the page for string "triad"). pv is good for those who like to watch the blinky lights, otherwise use dd or nc. And dd will still allow a bit of blinkiness, using SIGUSR1. – Mike S Sep 10 '15 at 13:24
  • +1 for netcat. Given this specific scenario (looking to transfer full drive capacity across a secure LAN with speed as the top priority) it really is the best tool for the job. Combine it with dd or tar and minimal compression for maximum throughput--and consider tuning your kernel and network (Jumbo Frames, etc) if you really need to eek out speed. – STW Sep 10 '15 at 13:27
36
  1. Do use fast compression.

    • Whatever your transfer medium - especially for network or usb - you'll be working with data bursts for reads, caches, and writes, and these will not exactly be in sync.
    • Besides the disk firmware, disk caches, and kernel/ram caches, if you can also employ the systems' CPUs in some way to concentrate the amount of data exchanged per burst then you should do so.
    • Any compression algorithm at all will automatically handle sparse runs of input as fast as possible, but there are very few that will handle the rest at network throughputs.
    • lz4 is your best option here:

      LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

  2. Preferably do not unnecessarily seek.

    • This can be difficult to gauge.
    • If there is a lot of free space on the device from which you copy, and the device has not been recently zeroed, but all of the source file-system(s) should be copied, then it is probably worth your while to first do something like:

      </dev/zero tee >empty empty1 empty2; sync; rm empty*
      
    • But that depends on what level you should be reading the source. It is usually desirable to read the device from start to finish from its /dev/some_disk device file, because reading at the file-system level will generally involve seeking back-and-forth and around the disk non-sequentially. And so your read command should be something like:

      </dev/source_device lz4 | ...
      
    • However, if your source file-system should not be transferred entire, then reading at the file-system level is fairly unavoidable, and so you should ball up your input contents into a stream. pax is generally the best and most simple solution in that case, but you might also consider mksquashfs as well.

      pax -r /source/tree[12] | lz4 | ...
      mksquashfs /source/tree[12] /dev/fd/1 -comp lz4 | ...
      
  3. Do not encrypt with ssh.

    • Adding encryption overhead to a trusted medium is unnecessary, and can be severely detrimental to the speed of sustained transfers in that the data read needs reading twice.
    • The PRNG needs the read data, or at least some of it, to sustain randomness.
    • And of course you need to transfer the data as well.
    • You also need to transfer the encryption overhead itself - which means more work for less data transferred per burst.
    • And so rather you should use netcat (or, as I prefer, the nmap project's more capable ncat) for a simple network copy, as has elsewhere been suggested:

      ###  on tgt machine...
      nc -l 9999 > out.lz4
      ###  then on src machine...
      ... lz4 | nc tgt.local 9999
      
mikeserv
  • 58,310
  • 1
    Fantastic answer. One minor grammatical point -- "lessen the amount of data which needs exchanging per burst" -- I think you're using compression to increase the information density as the 'bursts' are fixed-width and therefore the amount of exchanged data remains constant though the information transferred per burst may vary. – Software Engineer Sep 09 '15 at 08:39
  • @EngineerDollery - yes, that was dumb. I think it is better, – mikeserv Sep 10 '15 at 08:17
  • @IQAndreas - I would seriously consider this answer. Personally I use pigz, and the speed increase is amazing. The parallelism is a huge win; CPUs are much faster than any other part of the data pipeline so I doubt parallel compression will slow you down (gzip is not parallelizable). You may find this fast enough that there's no incentive to juggle hard drives; I wouldn't be surprised if this one is overall faster (including disk swap time). You could benchmark with and without compression. In any case, either BlueRaja's diskswap answer or this one should be your accepted answer. – Mike S Sep 10 '15 at 13:38
  • Fast compression is an excellent advice. It should be noted, though, that it helps only if the data is reasonably compressible, which means, e.g., that it must not already be in a compressed format. – Walter Tross Sep 10 '15 at 21:50
  • @WalterTross - it will help if any input is compressible, no matter the ratio, so long as the compression job outperforms the transfer job. On a modern four-core system an lz4 job should easily pace even wide-open GIGe, and USB 2.0 doesn't stand a chance. Besides, lz4 was designed only to work when it should - it is partly so fast because it knows when compression should be attempted and when it shouldn't. And if it is a device-file being transferred, then even precompressed input may compress somewhat anyway if there is any fragmentation in the source filesystem. – mikeserv Sep 10 '15 at 22:13
  • What is seek? Why is it bad? You added commands to avoid it but no explanation why seeking should be avoided. – A.L Sep 11 '15 at 17:32
  • @A.L. - I guess that's a fair criticism, though I did mention that fs reads typically involve seeking around the disk, but reading the disk's descriptor will instead typically involve reading the device from start to finish. I could, and maybe should, embellish that a little, but any truly satisfactory explanation will involve many caveats relating to the type of device we're talking about, and so is probably better explained in a more targeted question. Your own comment here would make for a very good question, as I think. – mikeserv Sep 11 '15 at 17:39
  • Not sure I understand the point about the PRNG needing "read data". At worst it sounds like potentially increased memory usage, but nothing indicates this would actually be impeding transfer speed. Also, CSPRNGs don't need to be fed anything continuously (which is their magic sauce), and using a pure RNG seems like a bad idea in this scenario (and you did say PRNG specifically), so the apparent premise of that sentence is quite confusing to me. – tne Jan 13 '16 at 12:14
  • @tne - ok. sorry. – mikeserv Jan 13 '16 at 12:41
  • @mikeserv, good to see you back – iruvar Oct 18 '16 at 10:34
  • @iruvar - just stopping by. ive been homeless since my house burnt down last month. i dont get much time to play with computers lately – mikeserv Oct 19 '16 at 07:09
25

There are several limitations that could be limiting the transfer speed.

  1. There is inherent network overhead on a 1Gbps pipe. Usually, this reduces ACTUAL throughput to 900Mbps or less. Then you have to remember that this is bidirectional traffic and you should expect significantly less than 900Mbps down.

  2. Even though you're using a "new-ish router" are you certain that the router supports 1Gbps? Not all new routers support 1Gbps. Also, unless it is an enterprise-grade router, you likely are going to lose additional transmit bandwidth to the router being inefficient. Though based on what I found below, it looks like you're getting above 100Mbps.

  3. There could be network congestion from other devices sharing your network. Have you tried using a directly attached cable as you said you were able to do?

  4. What amount of your disk IO are you using? Likely, you're being limited, not by the network, but by the disk drive. Most 7200rpm HDDs will only get around 40MB/s. Are you using raid at all? Are you using SSDs? What are you using on the remote end?

I suggest using rsync if this is expected to be re-run for backups. You could also scp, ftp(s), or http using a downloader like filezilla on the other end as it will parallelize ssh/http/https/ftp connections. This can increase the bandwidth as the other solutions are over a single pipe. A single pipe/thread is still limited by the fact that it is single-threaded, which means that it could even be CPU bound.

With rsync, you take out a large amount of the complexity of your solution as well as allows compression, permission preservation, and allow partial transfers. There are several other reasons, but it is generally the preferred backup method (or runs the backup systems) of large enterprises. Commvault actually uses rsync underneath their software as the delivery mechanism for backups.

Based on your given example of 80GB/h, you're getting around 177Mbps (22.2MB/s). I feel you could easily double this with rsync on a dedicated ethernet line between the two boxes as I've managed to get this in my own tests with rsync over gigabit.

muru
  • 72,889
Khrystoph
  • 459
  • 12
    +1 for rsync. It might not be faster the first time you run it, but it certainly will be for all the subsequent times. – Skrrp Sep 07 '15 at 08:23
  • 4
    > Most 7200rpm HDDs will only get around 40MB/s. IME you're more likely to see over 100MB/s sequential with a modern drive (and this includes ~5k drives). Though, this might be an older disk. – Bob Sep 07 '15 at 15:38
  • 2
    @Bob: Those modern still can read only 5400 circular tracks per minute. These disks are still fast because each track contains more than a megabyte. That does mean they're also quite big disks, A small 320 GB disk can't hold too many kilobytes per track, which necessarily limits their speed. – MSalters Sep 07 '15 at 17:07
  • 1
    40MB/s is definitely very pessimistic for sequential read for any drive made in the last decade. Current 7200RPM drives can exceed 100MB/s as Bob says. – hobbs Sep 10 '15 at 18:00
  • @MSalters It's a whole lot more than "kilobytes per track", and you can read one cylinder per rotation (1 track * number of heads, which is probably 1 track * 2 * number of platters). And you replaced 7200 with 5400 along the way :) – hobbs Sep 10 '15 at 18:14
  • 3
    Gigabit Ethernet is 1000 mbps full duplex. You get 1000mbps (or, as you say, around 900mbps in reality) each direction. Second... hard drives now routinely get 100MB/sec. 40MB/sec is slow, unless this is a decade old drive. – derobert Sep 10 '15 at 20:54
  • 1
    Modern HDDs are definitely in the 100-140MB/s range, rather than 40MB/s. And we've been in that territory for at least 7 years (since the Samsung F1, at the very least) – Jon Story Sep 15 '15 at 08:12
18

We deal with this regularly.

The two main methods we tend to use are:

  1. SATA/eSATA/sneakernet
  2. Direct NFS mount, then local cp or rsync

The first is dependent on whether the drive can be physically relocated. This is not always the case.

The second works surprisingly well. Generally we max out a 1gbps connection rather easily with direct NFS mounts. You won't get anywhere close to this with scp, dd over ssh, or anything similar (you'll often get a max rate suspiciously close to 100mpbs). Even on very fast multicore processors you will hit a bottleneck on the max crypto throughput of one of the cores on the slowest of the two machines, which is depressingly slow compared to full-bore cp or rsync on an unencrypted network mount. Occasionally you'll hit an iops wall for a little while and be stuck at around ~53MB/s instead of the more typical ~110MB/s, but that is usually short lived unless the source or destination is actually a single drive, then you might wind up being limited by the sustained rate of the drive itself (which varies enough for random reasons you won't know until you actually try it) -- meh.

NFS can be a little annoying to set up if its on an unfamiliar distro, but generally speaking it has been the fastest way to fill the pipes as fully as possible. The last time I did this over 10gbps I never actually found out if it maxed out the connection, because the transfer was over before I came back from grabbing some coffee -- so there may be some natural limit you hit there. If you have a few network devices between the source and destination you can encounter some slight delays or hiccups from the network slinky effect, but generally this will work across the office (sans other traffic mucking it up) or from one end of the datacenter to the other (unless you have some sort of filtering/inspection occurring internally, in which case all bets are off).

EDIT

I noticed some chatter about compression... do not compress the connection. It will slow you down the same way a crypto layer will. The bottleneck will always be a single core if you compress the connection (and you won't even be getting particularly good utilization of that core's bus). The the slowest thing you can do in your situation is to use an encrypted, compressed channel between two computers sitting next to each other on a 1gbps or higher connection.

FUTURE PROOFING

This advice stands as of mid-2015. This will almost certainly not be the case for too many more years. So take everything with a grain of salt, and if you face this task regularly then try a variety of methods on actual loads instead of imagining you will get anything close to theoretical optimums, or even observed compression/crypto throughput rates typical for things like web traffic, much of which is textual (protip: bulk transfers usually consist chiefly of images, audio, video, database files, binary code, office file formats, etc. which are already compressed in their own way and benefit very little from being run through yet another compression routine, the compression block size of which is almost guaranteed to not align with your already compressed binary data...).

I imagine that in the future concepts like SCTP will be taken to a more interesting place, where bonded connections (or internally bonded-by-spectrum channelized fiber connections) are typical, and each channel can receive a stream independent of the others, and each stream can be compressed/encrypted in parallel, etc. etc. That would be wonderful! But that's not the case today in 2015, and though fantasizing and theorizing is nice, most of us don't have custom storage clusters running in a cryo-chamber feeding data directly to the innards of a Blue Gene/Q generating answers for Watson. That's just not reality. Nor do we have time to analyze our data payload exhaustively to figure out whether compression is a good idea or not -- the transfer itself would be over before we finished our analysis, regardless how bad the chosen method turned out to be.

But...

Times change and my recommendation against compression and encryption will not stand. I really would love for this advice to be overturned in the typical case very soon. It would make my life easier.

zxq9
  • 308
  • 1
    @jofel Only when the network speed is slower than the processor's compression throughput -- which is never true for 1gpbs or higher connections. In the typical case, though, the network is the bottleneck, and compression does effectively speed things up -- but this is not the case the OP describes. – zxq9 Sep 08 '15 at 00:42
  • 2
    lz4 is fast enough to not bottleneck gigE, but depending on what you want to do with the copy, you might need it uncompressed. lzop is pretty fast, too. On my i5-2500k Sandybridge (3.8GHz), lz4 < /dev/raid0 | pv -a > /dev/null goes at ~180MB/s input, ~105MB/s output, just right for gigE. Decompressing on the receive side is even easier on the CPU. – Peter Cordes Sep 09 '15 at 21:34
  • @PeterCordes lz4 can indeed fit 1gbps connections roughly the same way other compression routines fit 100mpbs connections -- given the right circumstances, anyway. The basic problem is that compression itself is a bottleneck relative to a direct connection between systems, and usually a bottleneck between two systems on the same local network (typically 1gbps or 10gbps). The underlying lesson is that no setting is perfect for all environments and rules of thumb are not absolute. Fortunately, running a test transfer to figure out what works best in any immediate scenario is trivial. – zxq9 Sep 10 '15 at 04:39
  • 1
    Also, 3.8GHz is a fair bit faster than most server processors run (or many business-grade systems of any flavor, at least that I'm used to seeing). It is more common to see much higher core counts with much lower clock speeds in data centers. Parallelization of transfer loads has not been an issue for a long time, so we are stuck with the max speed of a single core in most cases -- but I expect this will change now that clockspeeds are generally maxed out but network speeds still have a long way to go before hitting their maximums. – zxq9 Sep 10 '15 at 04:44
  • I'm of course doing this on my desktop, with turbo. Good point on finding the right CPU vs. ratio tradeoff. http://lz4.info/ has a bar-graph comparing various compressors for 1gbit compress-transfer-decompress. (Look at the 2nd tab). lz4 has adjustable ratio vs. usage, but the default is already -1. I looked, but didn't see a parallel lz4 like pigz / pbzip2. It's not like the compressor uses a big dictionary to get a high ratio, so blocking the input into 128k chunks and doing each with a different thread should give a speedup. (128k = half L2 cache size for Intel CPUs). – Peter Cordes Sep 10 '15 at 06:12
  • Just found https://github.com/t-mat/lz4mt, link and other discussion on https://groups.google.com/forum/#!topic/lz4c/SPVAC62_6WU – Peter Cordes Sep 10 '15 at 06:15
  • @PeterCordes Interesting stuff there. :-) This is sort of a niche case for general computing at the moment (if you really need it, then you probably have a $10m contract with IBM, don't use connections measured in "giga"s, and have transfer mechanisms that bypass the main processor entirely; if you don't, its a nice-to-have or an issue of curiosity). Within a few years, maybe a decade at the very longest, I imagine it will be the focus of significant open source effort and eventually be taken for granted. – zxq9 Sep 10 '15 at 06:32
  • 2
    I completely disagree with your comments regarding compression. It depends completely on the compressability of the data. If you could get a 99.9% compression ratio, it would be foolish not to do so -- why transfer 100GB when you can get away with transferring 100MB? I'm not suggesting that this level of compression is the case for this question, just showing that this has to be considered on a case by case basis and that there are no absolute rules. – Software Engineer Sep 10 '15 at 10:31
  • 1
    @EngineerDollery This doesn't play out in bulk transfer at all in the real world. I do this almost every day and have tested a variety of methods and settings. In the general case large bulk transfers of unknown data (anything you don't have time to run compression tuning tests on -- which means in practice almost everything in any data center, corporate infrastructure, small business server, or home network) are much faster across a 1gbps or higher connection. Go try it. Text is typically best-case for compression. Text comprises a tiny fraction of a typical bulk transfer payload. – zxq9 Sep 10 '15 at 13:45
  • 1
    Anyway, disagree, agree -- it doesn't matter. What does matter is what is faster. Unless we are transferring 50TB of plain text files (HTML and source files would be awesome!) between two servers on a 1gbps connection, compression will generally slow down the transfer overall (certain segments may enjoy a massive boost, though). Most large loads are mostly binary executables/bytecode, images, audio, video, and database files. These are already compressed and typically don't benefit much from going through another compression routine -- but it eats all kinds of processor time. – zxq9 Sep 10 '15 at 13:51
  • @zxq9 - the best case scenario is that the OP transfers only one file - a device file. In that case the bulk payload could very well be many gigabytes worth of text files - as those are the kind most often edited on a disk and each time it is done the file itself is moved while their data remains. As I suggested elsewhere, it might be best to zero free-space in some cases, but maybe not. But even if the bulk of the payload is pre-compressed media, *fast, parallel* compression should still be used at least to handle sparseness and any other possible gains offered. – mikeserv Sep 10 '15 at 17:34
  • Not too long ago I learned to my surprise that Samba is nearly as fast as NFS when copying from Linux to Linux, easily saturating GigE and getting 110MB/s net throughput copying large files. When the client is Windows it seems to be a lot slower, however. – hobbs Sep 10 '15 at 18:22
  • 1
    @hobbs Indeed, Samba has an overhead comparable to NFS. The problem you will get with Windows is Window's own really horrible caching behavior; it doesn't matter if its the server or the client, transfer speeds slow to a crawl after a few GB. It is optimized for pulling a single file repeatedly (i.e. opening the same dozen or so documents to edit over and over), not large transfers. If you write a network transfer (simple write/read over TCP) utility on Windows you can achieve comparable rates to Linux. Why that's not standard is beyond me. – zxq9 Sep 13 '15 at 03:08
  • @mikeserv The OP's specific case is transferring /dev/sda to a backup server. A direct mount + rsync is the ideal for this (either that or physically walking a cloned drive over). That's why it is the focus of my answer. In a few years it is likely that fast, parallel compression will be broadly available, but that is not yet the case as of 2015. There is a difference between "this is a theoretical ideal" and "it is easy to practically implement this solution across various combinations of unknown systems the OP is using transferring unknown data volumes of unknown composition". – zxq9 Sep 13 '15 at 03:16
  • 1
    @zxq9 - the rsync solution should not be preferred for a single application - rsync is applied at a file-system abstraction level and typically only makes sense for an incremental backup system. In most cases there is a significant performance advantage to taking a step back and reading the source device as a bytestream vs. seeking around it to various offsets as instructed by the fs-layer. And a simple </dev/device lz4 is enough to compress input to output at a high-rate of speed. Because the OP specifically notes that speed is priority, I don't think rsync fits very well. – mikeserv Sep 13 '15 at 18:33
6

A nifty tool that I've used in the past is bbcp. As seen here: https://www.slac.stanford.edu/~abh/bbcp/ .

See also http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm

I've had very fast transfer speeds with this tool.

  • 1
    The second link of this answer explains how to tune kernel parameters to reach higher speeds. The author there got 800 megabytes per second in a 10G links and some things seem applicable to 1Gbps links. – Stéphane Gourichon Sep 08 '15 at 09:20
6

If you get a first pass somehow (over the wire/sneakernet/whatever), you can look into rsync with certain options that can greatly speed up subsequent transfers. A very good way to go would be:

rsync -varzP sourceFiles destination

Options are: verbose, archive mode, recursive, compress, Partial progress

A.L
  • 1,586
  • 2
    Rsync is more reliable than netcat, but archive implies recursive, so the r is redundant. – Tanath Sep 09 '15 at 20:04
  • Also, -z can be incrediby slow depending on your CPU and what data you are processing. I've experienced transfers going from 30 MB/s to 125 MB/s when disableing compression. – lindhe Jan 14 '16 at 20:22
4

Added on insistence of the original poster in comments to zackse’s answer, although I’m not sure it is the fastest in typical circumstances.

bash has a special redirection syntax:
For output:     > /dev/tcp/IP/port
For input:      < /dev/tcp/IP/port
IP ban be either dotted-decimal IP or a hostname; port ban be either a decimal number or a port name from /etc/services.

There is no actual /dev/tcp/ directory. It’s a special syntactic kludge that commands bash to create a TCP socket, connect it to the destination specified, and then do the same thing as a usual file redirection does (namely, replace the respective standard stream with the socket using dup2(2)).

Hence, one can stream data from dd or tar at the source machine directly via TCP. Or, conversely, to stream data to tar or something alike directly via TCP. In any case, one superfluous netcat is eliminated.

Notes about netcat

There is an inconsistency in syntax between classical netcat and GNU netcat. I’ll use the classical syntax I’m accustomed to. Replace -lp with -l for GNU netcat.

Also, I’m unsure whether does GNU netcat accept -q switch.

Transferring a disk image

(Along the lines of zackse’s answer.)
On destination:

nc -lp 9999 >disk_image

On source:

dd if=/dev/sda >/dev/tcp/destination/9999
 

Creating a tar.gz archive, with tar

On destination:

nc -lp 9999 >backup.tgz

On source:

tar cz files or directories to be transferred >/dev/tcp/destination/9999

Replace .tgz with .tbz and cz with cj to get a bzip2-compressed archive.

Transferring with immediate expansion to file system

Also with tar.
On destination:

cd backups
tar x </dev/tcp/destination/9999

On source:

tar c files or directories to be transferred |nc -q 1 -lp 9999

It will work without -q 1, but netcat will stuck when data ended. See tar(1) for explanation of the syntax and caveats of tar. If there are many files with high redundancy (low entropy), then compression (e. g. cz and xz instead of c and x) can be tried, but if files are typical and the network is fast enough, it would only slow the process. See mikeserv’s answer for details about compression.

Alternative style (the destination listens port)

On destination:

cd backups
nc -lp 9999 |tar x

On source:

tar c files or directories to be transferred >/dev/tcp/destination/9999
A.L
  • 1,586
Incnis Mrsi
  • 1,988
  • bash can't actually "listen" on a socket apparently, in order to wait and receive a file: https://unix.stackexchange.com/questions/49936/dev-tcp-listen-instead-of-nc-listen/49947#49947 so you'd have to use something else for at least one half of the connection... – rogerdpack Sep 27 '18 at 22:07
3

Try the suggestions regarding direct connections and avoiding encrypted protocols such as ssh. Then if you still want to eke out every bit of performance give this site a read: https://fasterdata.es.net/host-tuning/linux/ for some advice on optimizing your TCP windows.

2

I would use this script I wrote that needs the socat package.

On the source machine:

tarnet -d wherefilesaretosend pass=none 12345 .

On the target machine:

tarnet -d wherefilesaretogo pass=none sourceip/12345

If the vbuf package (Debian, Ubuntu) is there then the file sender will show a data progress. The file receiver will show what files are received. The pass= option can be used where the data might be exposed (slower).

Edit:

Use the -n option to disable compression, if CPU is a bottle neck.

Skaperen
  • 716
2

If budget is not the main concern, you could try connecting the drives with a Intel Xeon E5 12 core "drive connector". This connector is usually so powerful, that you can even run your current server software on it. From both servers!

This might look like a fun answer, but you should really consider why you are moving the data between servers and if a big one with shared memory and storage might make more sense.

Not sure about current specs, but the slow transfer might be limited by the disk speeds, not the network?

1

If you only care about backups, and not about a byte for byte copy of the hard drive, then I would recommend backupPC. http://backuppc.sourceforge.net/faq/BackupPC.html It's a bit of a pain to setup but it transfers very quickly.

My initial transfer time for about 500G of data was around 3 hours. Subsequent backups happen in about 20 seconds.

If your not interested in backups, but are trying to sync things then rsync or unison would better fit your needs.

A byte for byte copy of a hard disk is usually a horrid idea for backup purposes (no incrementals, no space saving, drive can't be in use, you have to backup the "empty space", and you have to back up garbage (like a 16 G swap file or 200G of core dumps or some such). Using rsync (or backuppc or others) you can create "snapshots" in time so you can go to "what your file system looked like 30 mins ago" with very little overhead.

That said, if your really want to transfer a byte for byte copy then your problem is going to lie in the transfer and not in the getting data from the drive. With out 400G of RAM a 320G file transfer is going to take a very ling time. Using protocols that are not encrypted are an option, but no matter what, your just going to have to sit there and wait for several hours (over the network).

coteyr
  • 4,310
  • 1
    how does 400G of RAM speed up data transfer? – Skaperen Sep 07 '15 at 09:39
  • Not sure this was the intent, but I read it as "any medium slower than RAM to RAM transfer is going to take a while", rather than "buy 400 GB of RAM and your HDD to HDD transfer will go faster". – MichaelS Sep 07 '15 at 10:07
  • Yep,, ram will buffer for you, and it will seem faster. You can do a HD to HD transfer with RAM buffering all the way and it will seem very fast. It will also take quite a wile to flush to disk, but HD to RAM to RAM to HD is faster then HD to HD. (Keep in mind you have to do HD to RAM to RAM to HD anyway but if you have less then your entire transfer size of RAM you will have to "flush" in segments.) – coteyr Sep 07 '15 at 12:07
  • Another way to put is that to compress or even just send the entire source drive has to be read in to ram. If it doesn't fit all at once, it has to read a segment, send, discard segment, seek, read segment, etc. If it fits all at once then it just has to read all at one time. Same on the destination. – coteyr Sep 07 '15 at 12:10
  • Huge amount of RAM may only help for second and subsequent transfers/recheck of the same data, but won't help at all for the first transfer. – Stéphane Gourichon Sep 08 '15 at 09:23
  • It's a sily amount of RAM in this case so it's not important, but there would be a small speed boost on the sender (as it doesn't have to page to hold the entire dataset) and a huge perceived speed boost on the receiver (as it caches in ram specially on slower drives) – coteyr Sep 08 '15 at 10:58
  • 1
    HD to RAM to RAM to HD is faster then HD to HD How can it be quicker? – A.L Sep 11 '15 at 17:36
  • Being extremely pedantic wrt the question as of 2016-01-13, OP only specifies reading 320G from a source drive and although they did mention the target machine has hard drives, there was no mention of what the data "being on the target computer" means. So, while buffering in RAM on the source is irrelevant, buffering in RAM on the target wouldn't be. Technically, this is actually the correct answer, though of course it doesn't capture the real intent as everybody understands it. It's not even useless in all scenarios; maybe we want to free the source machine as fast as possible. – tne Jan 13 '16 at 14:09
1

Regardless of program, I have usually found that "pulling" files over a network is faster than "pushing". That is, logging into the destination computer and doing a read is faster than logging into the source computer and doing a write.

Also, if you are going to use an intermediate drive, consider this: Get an external drive (either as a package, or a separate drive plugged into a docking station) which uses eSATA rather than USB. Then on each of the two computers either install a card with an eSATA port, or get a simple adapter cable which brings one of the internal SATA ports to an external eSATA connector. Then plug the drive into the source computer, power up the drive, and wait for it to auto-mount (you could mount manaully, but if you are doing this repeatedly you might as well put it into your fstab file). Then copy; you will be writing at the same speed as to an internal drive. Then unmount the drive, power down, plug into the other computer, power up, wait for an auto-mount, and read.

  • 2
    Can you provide specifics of how you're "pulling" files? What utilities are you using, and can you provide any sample showing this effect? – STW Sep 10 '15 at 13:22
  • I am not sure if this will be a more complete answer, but consider this scenario: Suppose you have two computers, foo and bar, and you want to copy data from foo to bar. (1) You log into foo, then remote mount the drive which is physically attached to bar. Then you copy from foo's disk onto the remotely-mounted directory (which is physically on bar). I called this pushing the data to the other computer. (2) Compare this to the other way of copying the same data. Log into bar, remote-mount the directory attached to foo, and read from foo onto bar's drive. This is pulling. – Mike Ciaraldi Sep 10 '15 at 17:42
  • This copying can be done with the Linux cp command, from a a GUI file manager, or any other way of copying files. I think pulling turns out to be faster because writing is slower than reading, and more of the decisions on how to write to the destination disk are being done on the same computer the drive is attached to, so there is less overhead. But maybe this is no longer the case with more modern systems. – Mike Ciaraldi Sep 10 '15 at 17:45
1

How about an ethernet crossover cable? Instead of relying on wireless speeds you're capped at the wired speed of your NIC.

Here's a similar question with some examples of that kind of solution.

Apparently just a typical ethernet cable will suffice nowadays. Obviously the better your NIC the faster the transfer.

To summarize, if any network setup is necessary, it should be limited to simply setting static IPs for your server and backup computer with a subnet mask 255.255.255.0

Good luck!

Edit:

@Khrystoph touched on this in his answer

  • How will it improve speed rates? Can you please explain it your answer? – A.L Sep 11 '15 at 17:43
  • 1
    It would potentially improve speed because you would not have to worry about the intermediate network slowing you down. Regarding "typical" vs "crossover" ethernet cables - 1Gb ethernet will auto-crossover as necessary. HP ethernet switches will do this at 100Mb. Other brands, generally not, and you'll need a crossover if you're stuck at 100Mb. – Dan Pritts Sep 11 '15 at 20:47
1

I'm going to recommend that you look at NIC-teaming. This involves using multiple network connections running in parallel. Assuming that you really need more than 1Gb transfer, and that 10Gb is cost prohibitive, 2Gbs provided by NIC-teaming would be a minor cost, and your computers may already have the extra ports.

  • If you're referring to LACP (Link Aggregation Control Protocol) then you're not going to see an increase in speed. It Provided redundancy and some ability to serve more concurrent connections, but it won't provide a speed boost for this type of transfer. – STW Sep 09 '15 at 21:04
  • @STW: It requires switch support to aggregate two links to one machine into a 2gbit link, but it is possible. Helpful only if both machines have a 2gbit link to the switch, though. If you have two cables running NIC<->NIC, with no switch, that should work too, but isn't very useful (unless you have a 3rd NIC in one machine to keep them connected to the Internet). – Peter Cordes Sep 10 '15 at 06:03
  • is there a specific name for this feature in switches? – STW Sep 10 '15 at 13:21
  • There are several variations of NIC-teaming, EtherChannel, etc. STW is right for certain configurations, this won't help, but for some configurations, it would. It comes down to whether or not the bonded channel speeds up performance for a single IP socket or not. You'll need to research the specifics to determine if this is a viable solution for you. – Byron Jones Sep 11 '15 at 20:18
  • 802.3ad is the open standard that you'd look for on your switches. As a quick hack, though, you might just connect extra NICs to the network, and give them appropriate IP addresses on separate subnets in private address space. (host 1 port a & host 2 port a get one subnet, host 1 port b and host 2 port b get another subnet). Then just run two parallel jobs to do the transfer. This will be a lot simpler than learning the ins and outs of Etherchannel, 802.3ad, etc. – Dan Pritts Sep 14 '15 at 14:02
1

FWIW, I've always used this:

tar -cpf - <source path> | ssh user@destserver "cd /; tar xf -"

Thing about this method is that it will maintain file/folder permissions between machines (assuming the same users/groups exist on both) (Also I typically do this to copy virtual disk images since I can use a -S parameter to handle sparse files.)

Just tested this between two busy servers and managed ~14GB in 216s (about 64MB/s) - might would do better between dedicated machines and/or compression... YMMV

$ date; tar -cpf - Installers | ssh elvis "cd /home/elvis/tst; tar xf -"; date
Wed Sep  9 15:23:37 EDT 2015
Wed Sep  9 15:27:13 EDT 2015

$ du -s Installers
14211072   Installers
muru
  • 72,889
1

Unless you want to do filesystem forensics, use a dump/restore program for your filesystem to avoid copying the free space that the FS isn't using. Depending on the what filesystem you have, this will typically preserve all metadata, including ctime. inode numbers may change, though, again depending on what filesystem (xfs, ext4, ufs...).

The restore target can be a file on the target system.

If you want a full-disk image with the partition table, you can dd the first 1M of the disk to get the partition table / bootloaders / stuff, but then xfsdump the partitions.

I can't tell from your info-dump what kind of filesystem you actually have. If it's BSD ufs, then I think that has a dump/restore program. If it's ZFS, well IDK, there might be something.

Generally full-copying disks around is too slow for anything except recovery situations. You can't do incremental backups that way, either.

Peter Cordes
  • 6,466
1

You could also setup the systems to have a shared storage!

I am considering that these are next to each other, and you are likely to do this again & again ....

1

Several folks recommend that you skip ssh because encryption will slow you down. Modern CPUs may actually be fast enough at 1Gb, but OpenSSH has problems with its internal windowing implementation that can drastically slow you down.

If you want to do this with ssh, take a look at HPN SSH. It solves the windowing problems and adds multithreaded encryption. Unfortunately you'll need to rebuild ssh on both the client & server.

1

OK I have attempted to answer this question for two computers with "very large pipes" (10Gbe) that are "close" to each other.

The problem you run into here is: most compression will bottleneck at the cpu, since the pipes are so large.

performance to transfer 10GB file (6 Gb network connection [linode], uncompressible data):

$  time bbcp 10G root@$dest_ip:/dev/null
0m16.5s 

iperf:

server: $ iperf3 -s -F /dev/null
client:
$ time iperf3 -c $dest_ip -F 10G -t 20 # -t needs to be greater than time to transfer complete file
0m13.44s
(30% cpu)

netcat (1.187 openbsd):

server: $ nc -l 1234 > /dev/null
client: $ time nc $dest_ip 1234 -q 0 < 10G 
0m13.311s
(58% cpu)

scp:

$ time /usr/local/bin/scp 10G root@$dest_ip:/dev/null
1m31.616s
scp with hpn ssh patch (scp -- hpn patch on client only, so not a good test possibly): 
1m32.707s

socat:

server:
$ socat -u TCP-LISTEN:9876,reuseaddr OPEN:/dev/null,creat,trunc
client:
$ time socat -u FILE:10G TCP:$dest_ip:9876
0m15.989s

And two boxes on 10 Gbe, slightly older versions of netcat (CentOs 6.7), 10GB file:

nc: 0m18.706s (100% cpu, v1.84, no -q option
iperf3: 0m10.013s (100% cpu, but can go up to at least 20Gbe with 100% cpu so not sure it matters)
socat: 0m10.293s (88% cpu, possibly maxed out)

So on one instance netcat used less cpu, on the other socat, so YMMV.

With netcat, if it doesn't have a "-N -q 0" option it can transfer truncated files, be careful...other options like "-w 10" can also result in truncated files.

What is happening in almost all of these cases is the cpu is being maxed out, not the network. scp maxes out at about 230 MB/s, pegging one core at 100% utilization.

Iperf3 unfortunately creates corrupted files. Some versions of netcat seem to not transfer the entire file, very weird. Especially older versions of it.

Various incantations of "gzip as a pipe to netcat" or "mbuffer" also seemed to max out the cpu with the gzip or mbuffer, so didn't result in faster transfer with such large pipes. lz4 might help. In addition, some of the gzip pipe stuff I attempted resulted in corrupted transfers for very large (> 4 GB) files so be careful out there :)

Another thing that might work especially for higher latency (?) is to tune tcp settings. Here is a guide that mention suggested values:

http://pcbunn.cithep.caltech.edu/bbcp/using_bbcp.htm and https://fasterdata.es.net/host-tuning/linux/ (from another answer) possibly IRQ settings: https://fasterdata.es.net/host-tuning/100g-tuning/

suggestions from linode, add to /etc/sysctl.conf:

net.core.rmem_max = 268435456 
net.core.wmem_max = 268435456 
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_no_metrics_save = 1
net.core.default_qdisc = fq 

Additionally, they would like you to run:

 /sbin/ifconfig eth0 txqueuelen 10000 

worth double checking after tweaking to make sure changes don't cause harm too.

Also may be worth tuning the window size: https://iperf.fr/iperf-doc.php#tuningtcp

With slow(er) connections compression can definitely help though. If you have big pipes, very fast compression might help with readily compressible data, haven't tried it.

The standard answer for "syncing hard drives" is to rsync the files, that avoids transfer where possible.

Another option: use "parallel scp" (somehow or other), then it will use more cores...

rogerdpack
  • 1,715