Backing up some USB flash-drives on Lubuntu 16.04.3 LTS.
Plug USB drive into laptop. Automounter does its stuff. I remount all partitions as read-only
sudo mount -o remount,ro /dev/sdxn
Copy USB drive to image file on hard disk using ddrescue (restartable)
sudo ddrescue -b 4096 /dev/sdx sdx.img sdx.log
(Note: blocksize determined by sudo blockdev --getbsz /dev/sdx )
md5sum sdx.img gives result < foo >
purge disk buffers:
sudo blockdev --flushbufs /dev/sdx
- otherwise subsequent command reads disk cache and not the actual disk (as confirmed by looking at the activity light on the USB drive, and the speed of completion of later commands).
sudo md5sum /dev/sdx gives result < bar >
So it seems the copy failed.
BUT
a subsequent sudo dd if=/dev/sdc bs=4096 | md5sum
gives result < foo > again !!! WHAT?
It turns out dd (and ddrescue) generates consistent images, all with the same md5sum, but if I md5sum the device file directly, I get different md5sum results.
Have I missed something obvious? Most howtos advise using md5sum on the device file, so I guess I have either found a bug/feature, or been idiotic and missed something blindingly obvious. Any clues?
(Note, I'm generating images of the USB drives, as some of them are bootable, with a useful set of tools on them. This takes up space, so for the larger drives, I will reduce the storage requirement by using mksquashfs ( mksquashfs sdx.img sdx.squash.img ). This works well, so long as I change the ownership of the image file from root to my username before mksquashfs. This means I get mountable compressed images, so browsing the contents is easy. There's probably a better way.
[Edit to add: having seen hdd-image-file-checksum-does-not-match-with-device-checksum I can say that sudo blockdev --getsize64 /dev/sdx and the size of the image file on disk are precisely the same.]
[Further update:
$ sudo cat /dev/sdx | md5sum
a4db504f408c139cd2517b4038dfdb2f -
$ sudo dd if=/dev/sdx | md5sum
2038272+0 records in
2038272+0 records out
1043595264 bytes (1.0 GB, 995 MiB) copied, 105.895 s, 9.9 MB/s
e6174ed97f010cc7aa2ea9669cd85c50 -
$ sudo md5sum /dev/sdx
823b619d439fbb0abde5841097c76e72 /dev/sdx
Then repeat
$ sudo cat /dev/sdx | md5sum
a4db504f408c139cd2517b4038dfdb2f -
$ sudo dd if=/dev/sdx | md5sum
2038272+0 records in
2038272+0 records out
1043595264 bytes (1.0 GB, 995 MiB) copied, 105.747 s, 9.9 MB/s
e6174ed97f010cc7aa2ea9669cd85c50 -
$ sudo md5sum /dev/sdx
823b619d439fbb0abde5841097c76e72 /dev/sdx
So the behavior is repeatable - using cat, dd, and direct md5sum gives different answers, but apparently cat gives the same answer each time as does dd, and as does md5sum.
Since I can use cat, dd, ddrescue, or pv to copy the contents of the device to an on-disk image, which is 'correct'?]
