This is a situation I am frequently in:
- I have a source server with a 320GB hard-drive inside of it, and 16GB of ram (exact specs available here, but as this is an issue I run into frequently on other machines as well, I would prefer the answer to work on any "reasonable" Linux machine)
- I have a backup server with several terabytes of hard-drive space (exact specs here, see disclaimer above)
I want to transfer 320GB of data from the source server to the target server (specifically, the data from /dev/sda
).
- The two computers are physically next to each other, so I can run cables between them.
- I'm on a LAN, and I'm using a new-ish router, which means my network speeds should "ideally" be 1000Mbit, right?
- Security is not an issue. I am on a local network, and I trust all machines on the network, including the router.
- (optional) I don't necessarily need a signed checksum of the data, but basic error checking (such as dropped packets, or the drive becoming unreadable) should be detected rather than just disappear into the output.
I searched for this question online, and have tested several commands. The one that appears the most often is this:
ssh user@192.168.1.100 'dd bs=16M if=/dev/sda | gzip' > backup_sda.gz
This command has proven too slow (it ran for an hour, only got about 80GB through the data). It took about 1 minute and 22 seconds for the 1GB test packet, and ended up being twice as fast when not compressed. The results may also have been skewed by the fact that the transferred file is less than the amount of RAM on the source system.
Moreover (and this was tested on 1GB test pieces), I'm getting issues if I use the gzip
command and dd
; the resulting file has a different checksum when extracted on the target, than it does if piped directly. I'm still trying to figure out why this is happening.
ssh
was being too slow, I just didn't have enough free space on the external harddrive. – IQAndreas Sep 07 '15 at 04:45/dev/sda
as an image or just the files. Why is rsync no option? Is/dev/sda
mounted while youdd
ed? – Jodka Lemon Sep 07 '15 at 05:48ssh -o Compression=no
. In between these two, and with a fast cipher (on modern hardware 128-bit AES, on ancient hardware RC4, or since you trust both the systems and the link even none), the overhead of the SSH pipe should be minimal. – user Sep 08 '15 at 08:15320GB
, but I want methods that theoretically should work even if I transfer larger amounts, such as a4TB
image, hence "massive amounts of data". – IQAndreas Sep 08 '15 at 14:30netcat
suggestion by zackse ended up being much faster than SSH. (I'll write a blog post with the exact speed comparisons in a few days) – IQAndreas Sep 08 '15 at 16:49dd
image) is great for cloning machines, swapping drives, and 100% backups. An archive (such as a
tarfile, or even
rsync`) is great if you'll need to retrieve portions of your backup but not the entire thing. – STW Sep 10 '15 at 13:29dd
image, but I occasionally run into the same type of problem when transferring very largetar
images, or even other files. – IQAndreas Sep 10 '15 at 15:09dd bs=16M if=/dev/sda | gzip | ssh user@192.168.1.100 'cat - > backup_sda.gz'
. Caveats about ssh and compression speed still apply. If you have multiple cores to devote to the task, try pigz instead of gzip. – Dan Pritts Sep 14 '15 at 14:08buffer -s 8192 -b 2048 -i /dev/sda|nc target.ip 64738
. On the target server:nc -p 64738 -l|buffer -s 8192 -b 2048 -o /dev/targethd
. It parallelizes everything (including disk & network io), doesn't compress anything (you don't need it here). All the other answers, including the ones in the comments, have at most half the speed than this, mainly for their single-thread executions and internal locks. – peterh Sep 28 '18 at 04:10buffer
. Doing this optimally with any single-threaded tool is impossible. – peterh Sep 28 '18 at 04:15tar
the data on the source and would use again abuffer
-based solution on the target. | The answer saying the physical move of harddisk is also bad, despite its 100+ score,because SATA hard disks have around 60 MB/sec read speed, while gigabit ethernet can pass ideally 120MB/sec (you need also good switch and NICs for that!). – peterh Sep 28 '18 at 04:21dd if=/dev/xvda bs=10M status=progress | nc -q0 dest.localdomain 1234
nc -l -p 1234 </dev/null | dd of=./xvda.img bs=10M status=progress
. I know: iperf over ssh ~1.25GB/s, ddzero-to-ddnull shell pipe ~0.4GB/s, ddzero-to-ddnull over ssh ~0.1GB/s, ddzero-to-ddnull over local nc-to-nc ~1GB/s, too slow: sha512sum ~ 0.25GB/s, md5sum ~ 0.5GB/s, gzip ~0.025GB/s, zstdmt ~0.6GB/s, gpg ~0.1GB/s, gpg nocompress ~0.19GB/s, – ThorSummoner Feb 19 '21 at 20:20