Full DD copy from hdd to hdd

Question

ORIGINAL QUESTION:

If I have 2 identical hard drives with the following characteristics:

• SATA 6.0 Gb/s
• 5400 rpm
• 3TB

How long should a full dd copy take to complete?

So far it's been running for 5 hours and still going...

I am using Linux Ubuntu 12.04 64bit and the command I am using is:

dd if=/dev/sdb of=/dev/sdc

---

UPDATE: 1

I can now see the progress, and it's been 6+ hours for copy 430GB. The HDD is 3TB...

Is there no faster way of doing this?

---

UPDATE: 2

This seems a lot better than before (Thanks to Groxxda for the suggestions):

sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K

ETA is about 9 hours for 3TB, whereas before it reached 430GB after 6 hours, so I am guessing it would have taken about 36 hours using the previous command.

Answer 1

dd was useful in the old days when people used tapes (when block sizes mattered) and when simpler tools such as cat might not be binary-safe.

Nowadays, dd if=/dev/sdb of=/dev/sdc is a just complicated, error-prone, slow way of writing cat /dev/sdb >/dev/sdc. While dd still useful for some relatively rare tasks, it is a lot less useful than the number of tutorials mentioning it would let you believe. There is no magic in dd, the magic is all in /dev/sdb.

Your new command sudo dd if=/dev/sdb bs=128K | pv -s 3000G | sudo dd of=/dev/sdc bs=128K is again needlessly slow and complicated. The data is read 128kB at a time (which is better than the dd default of 512B, but not as good as even larger values). It then goes through two pipes before being written.

Use the simpler and faster cat command. (In some benchmarks I made a couple of years ago under Linux, cat was faster than cp for a copy between different disks, and cp was faster than dd with any block size; dd with a large block size was slightly faster when copying onto the same disk.)

cat /dev/sdb >/dev/sdc

If you want to run this command in sudo, you need to make the redirection happen as root:

sudo sh -c 'cat /dev/sdb >/dev/sdc'

If you want a progress report, since you're using Linux, you can easily get one by noting the PID of the cat process (say 1234) and looking at the position of its input (or output) file descriptor.

# cat /proc/1234/fdinfo/0
pos:    64155648 
flags:  0100000

If you want a progress report and your unix variant doesn't provide an easy way to get at a file descriptor positions, you can install and use pv instead of cat.

Answer 2

dd uses a very small blocksize by default (512 bytes). That results in a lot of overhead (one read() and write() syscall for every 512 bytes).

It goes a lot faster when you use a larger blocksize. Optimal speeds start at bs=64k or so. Most people use a still larger bs=1M so it becomes human readable (when dd says it copied 1234 blocks, you know it's 1234 MiB without doing any math). Using even larger blocksizes are unlikely to result in speed improvements, just higher memory consumption.

So the command should be:

dd bs=1M if=/dev/sdb of=/dev/sdc

If you already have a slow dd running, you can interrupt it and resume with a faster dd instance. For this it is important to know how far the copy progressed already. dd usually prints the progress when you cancel it, or you can send it the USR1 signal while it is running to make it print its progress.

kill -USR1 $(pidof dd)

For example if it copied more than 1234MiB, you can resume at position 1234MiB using:

dd bs=1M seek=1234 skip=1234 if=/dev/sdb of=/dev/sdc

If it copied fewer than 1234MiB, your copy will be incomplete. If it copied more than 1234MiB, it will re-copy some already copied parts, which normally does not do any harm. So if in doubt you should pick a value slightly smaller than what you believe was already copied.

Answer 3

Getting statistics about ongoing dd process

You can use the kill command with the appropriate signal to make dd output statistics to standard error.
From the GNU dd man page:

Sending a USR1 signal to a running 'dd' process makes it print I/O statistics to standard error and then resume copying.
      $ dd if=/dev/zero of=/dev/null& pid=$!
      $ kill -USR1 $pid
      18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB)  copied,  34.6279 seconds, 271 MB/s

Make sure you check your man page for the correct signal first as it may differ on different dd implementations: (BSD dd uses SIGINFO).

Speeding up the process

1. Connect each HDD to it's own SATA port so the data can be read from one device and written to the other at the same time.
2. Use an appropriate blocksize using the bs= argument. Have a look at this thread on superuser and try some values for yourself.
3. Use separate dd invocations for reading and writing and use a pipe to connect them (dd if=/dev/sda bs=1M | dd of=/dev/sdb bs=1M).
   If you do this and specify a blocksize, make sure you use the same blocksize on each invocation.
4. You may try other optimizations like the direct argument.
5. Make sure your hard disks are not mounted or it may result in a corrupt copy.

Answer 4

Have you tried 'gparted' ? You can literally copy-paste a partition from one drive to another and resize it accordingly as needed. You get transfer rate and time remaining. It uses 'e2image' underneath for linux partitions.

Answer 5

There is commercial software called HDClone. It is licensed as both freeware and commercial. Either version can create a bootable pendrive or DVD, as well as copying disks. Connect the source and destination HDs and follow the GUI-based instructions.

The free one will transfer at a rate about 80MB/sec, while the paid versions can go much faster. For NTFS and FAT filesystems, the paid versions can be configured to clone only the occupied bits of the HD, which dramatically increases the speed of each clone.