Does dd do any kind of verification?

Question

I'm using dd to copy data from an old hard drive to a new one. I want to be sure that the data's integrity is secure.

On this answer, Gilles says

If [dd] terminated successfully, then the backup is correct, barring a hardware fault…

What does that mean exactly? Does dd have some kind of built in verification?

If I were to use rsync instead, I run a second pass with --checksum as well, to verify. Is that kind of paranoia justified?

Answer 1

dd or any other application does not have “some kind of built in verification” in the sense you're probably thinking of: it doesn't read back the data from the storage medium to compare with what was written. That's the job of the operating system.

It is not really possible to do a read-verify down to the hardware from an application. It would work in some scenarios, but in most cases it would achieve nothing. The application could read back what it just wrote if it's writing directly to a storage medium, but that would typically read back from an in-memory cache, which wouldn't give any useful assurance. In the example you cite, dd is writing to a pipe, and in that case it has no control over what happens to the data further down the line. In your rsync example, a second pass of rsync --checksum is pointless: in theory it could catch an error, but in practice, if an error does happen, then the second pass would probably not report anything wrong, so you're wasting effort on something that doesn't actually give a useful assurance.

However, applications do verify what happens to the data, in the sense that they verify that the operating system has accepted responsibility for the data. All system calls return an error status. If a system call returns an error status, the application should propagate that error to the user, generally by displaying an error message and returning a nonzero exit status.

Beware that dd is an exception: depending on the command line parameters, dd might ignore some errors. This is extremely unusual: dd is the only common command with this property. Use cat instead of dd, that way you don't risk corruption and it may well be faster.

In a chain of data copying, two kinds of errors can arise.

• Corruption: a bit is flipped during the transfer. There is no way to verify this at the application level, because if that happens, it's due to a programming bug or hardware error that is highly likely to cause the same corruption when reading back. The only useful way to verify that no such corruption happened is to physically disconnect the media and try again, preferably on a different computer in case the problem was with the RAM.
• Truncation: all the data that was copied was copied correctly, but some of the data was not copied at all. This one is worth checking sometimes, depending on the complexity of the command. You don't need to read the data to do that: just check the size.

Answer 2

No, dd does not do an explicit verification. If you want/need a forensically verified copy of your disk or any part of it, use dcfldd which is an enhanced version of dd developed by the U.S. Department of Defense Computer Forensics Lab.

Answer 3

The only way to be "sure" is to do an additional read-and-compare pass (after dropping caches).

Aside from that, dd detects read and write errors the same way all other programs do... it works if the drives (and other components involved) report errors; for drives that accept data silently w/o actually writing them you're out of luck.

Is that kind of paranoia justified?

If you can't trust your hardware to be reliable, things get complicated...

Answer 4

Yes, faulty hardware may insert random error bits in the data at some rate as one bit per number of megabytes, this is possible and takes place in practice sometimes.

Usually, I use md5 or sha1 hash to verify the data is intact, by re-reading both the source and the destination, e.g.:

dd if=/dev/sdb of=~/hd_backup
dd if=/dev/sdb | md5sum
dd if=~/hd_backup | md5sum

This assumes that the data is much larger than the filesystem cache, otherwise, you may need to re-start the system to verify the actual data on the medium and not the cache contents, or use another system for it.

Answer 5

From man dd:

When finished, dd displays the number of complete and partial input and output blocks, truncated input records and odd-length byte-swapping blocks to the standard error output.

A partial input block is one where less than the input block size was read. A partial output block is one where less than the output block size was written. Partial output blocks to tape devices are considered fatal errors. Otherwise, the rest of the block will be written. Partial output blocks to character devices will produce a warning message.

dd verifies the input/ouput block sizes match every time it copies a block. If they don't, it handles the error with a warning or fatal error (overidden with noerror). That's why dd works virtually all the time.

Still, it doesn't replace manually verifiying the integrity of your disk. If the information is valuable to you, then yes, your paranoia is justified. Run a manual verification once dd is finished.