5

I have a ~1GB image that I'm writing to a 8GB SD card via the dd tool. I'd like to verify that it was written without corruption by reading it back and comparing its hash with original one.

Obviously, when I read it back via dd the size of resulting image matches size of my SD card, therefore checking hashes are useless.

I believe that I should somehow interpret the output of writing invocation to configure the skip / count parameters to read it back properly.

Command that I used to write my image:

> sudo dd if=my.img of=/dev/sdc bs=1M 
8+50581 records in
8+50581 records out
3947888640 bytes (3.9 GB) copied, 108.701 s, 36.3 MB/s

Command that I used to read my image:

> sudo dd if=/dev/sdc of=same_as_my.img
15523840+0 records in
15523840+0 records out
7948206080 bytes (7.9 GB) copied, 285.175 s, 27.9 MB/s
Kentzo
  • 179

6 Answers6

6

Determine the size of the image, for example with \ls -l my.img (not ls -lh, that would give you an approximate size; \ls protects against an alias like ls='ls -h') or with stat -c %s my.img.

If you want to check the copy against the original just this once, then just compare the files. Using hashes is useless for a one-time comparison, it would only make things slower and require more commands. The command cmp compares binary files. You need to pass it the image file and the corresponding part of the SD card. Use head to extract the beginning of the SD card.

</dev/sdc head -c "$(stat -c %s my.img)" | cmp - my.img

If you want to perform many comparisons, then hashes are useful, because you only need to read each instance once, to calculate its hash. Any hash will do since you're worried about data corruption. If you needed to check that a file hasn't been modified for security reasons, then cksum and md5sum would not be suitable, you should use sha256sum or sha512sum instead.

md5sum <my.img >my.img.md5sum
</dev/sdc head -c "$(stat -c %s my.img)" | md5sum >sd-copy.md5sum
cmp my.img.md5sum sd-copy.md5sum

Note the input redirection in the first command; this ensures that the checksum file doesn't contain file names, so you can compare the checksum files. If you have a checksum file and a copy to verify, you can do the check directly with

</dev/sdc head -c "$(stat -c %s my.img)" | md5sum -c my.img.md5sum

Oh, and don't use dd, it's slow (or at best not faster) and doesn't detect copy errors.

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • Is there a quick one-liner to compare the written SD card with a compressed image, which was written to the card with xzcat and not unpacked on disk? – emk2203 Aug 13 '23 at 08:24
  • 1
    @emk2203 xzcat myimage.xz | md5sum for the image and </dev/foo head -c $(xz --robot --list myimage.xz | awk 'NR==2 {print $5}') | md5sum for the device content. – Gilles 'SO- stop being evil' Aug 15 '23 at 17:25
  • Thanks! </dev/foo head -c $(...) should be equivalent to head -c $(...) /dev/foo, or is there a difference in the case of head? The latter is easier to use sudo with. – emk2203 Aug 16 '23 at 10:33
  • @emk2203 No difference. I just find pipelines easier to read when they're in the order source-filter1-filter2-…-destination rather than filter1-source-filter2-…-destination. – Gilles 'SO- stop being evil' Aug 16 '23 at 12:07
1

What we did was to md5um the actual partition. It doesn't exactly allow you to checksum the image with the disk, but if you have a few disks (like we do), you are able to establish the 'proper' checksum.

For instance, in our case the partitions look like this:

$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 7948 MB, 7948206080 bytes
245 heads, 62 sectors/track, 1021 cylinders, total 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            8192      122879       57344    c  W95 FAT32 (LBA)
/dev/sdc2          122880     7710719     3793920   83  Linux

We can then simply generate a hash on the partition(s) with:

$ sudo md5sum /dev/sdc2

This however assumes the following:

  • You have multiple SD cards that you can test with
  • You do not mount the SD card prior to running the hash

Also, please note that you cannot checksum the entire disk (i.e. '/dev/sdc' in the example above), as that will give a different hash based on the SD card properties.

vpetersson
  • 111
0

md5sum would be a good solution here, to compare the value of the image file with that of the SD card.

steve
  • 21,892
  • Please check the updated Q. The problem is to read the exact same image back, to to calculate their hashes. – Kentzo Jul 08 '15 at 00:21
0

Are you writing it to a filesystem on the SD card ? Or directly ? You need count if you want to limit the run length of the compared data ?

dd should tell you how many bytes were copied.

This needs to be captured and used with appropriate bs=x and count=y. This is to avoid hashing garbage in a cluster tip.

EDIT

capture the output, this is the line you want, use grep for bytes.

254033920 bytes (254 MB) copied, 1.198 s, 212 MB/s

so (excuse verbosity and backticks).

# original dd
dd if=/path/to/file of=/dev/sdc 2> tempfile
# get bytes
CNT=`cat tempfile | grep bytes | cut -d' ' -f1`
# copy it back by byte
dd bs=1 count=$CNT if=/dev/sdc of=/path/to/copy

This will be slow, so (and I am not going to do it here), you need to calculate the optimal blocksize, and the number of whole multiples to read first, and then get the remainder with a skip/seek.

But a better approach would be to read back a rounded up number of blocks and perform the hash on a truncated length, the object is not to read the whole device if not necessary.

mckenzm
  • 327
  • I'm writing to an SD card as a block device, so I supposed that means directly. Could you explain how one can interpret the result of writing operation to properly configure reading? – Kentzo Jul 08 '15 at 00:23
  • 1
    I've tried to. You want the ceiling of bytes/blocksize let's say bytes/4096 rounded up. then get that number of blocks to a new file. md5sum does not accept a run length parameter. man truncate and man md5sum for more. If you do not want to hash try diff or vimdiff but cmp is just as valid. – mckenzm Jul 08 '15 at 01:22
0
sudo sh -c '
     dd bs=64k if="$1" of="$2"
     ! cmp -- "$1" "$2" 2>&1 |
       grep -qvF "EOF on $1"
' -- my.img /dev/sdc

cmp will compare two files byte for byte and return based on whether or not they are identical. If one is shorter than the other, but both files were identical for the entire length of the shorter file, then cmp will return 1 and report EOF on <shorter file>... to stderr. If both files are identical, cmp will return 0 and report nothing at all, but if they differ otherwise, cmp will return 1 and report on the byte where they differ. And so the above statement will return 1 for any cmp run which produces any line of output on stdout or stderr which does not match EOF on $1 or 0 otherwise.

mikeserv
  • 58,310
0

IMO, the best (easiest) way to do this is to when using dd, use a byte size (bs=) that can be evenly divided into your source file. This results in a whole number of block writes. Then you can reverse the process (read it back) ensuring the exact number of bytes are read by us both bs= and count= in the read command - and pipe it through the checksum program.

Example:

ls

-rw-rw-r-- 1 1006632960 myfile.iso

Generate a checksum:

sha256sum myfile.iso

8012fcba8bf71a7dd9e8179af40bce0fec57bb84b7426e4de807130ada63243d myfile.iso

The filesize 1006632960 is divisible by 512. So using dd:

sudo dd if=myfile bs=512b of=/dev/sdi

results in this output:

3840+0 records in

3840+0 records out

The +0 is the important part here. If you see anything other than +0 you need a different bytesize in your dd command.

Then to check the results, use this command:

sudo dd if=/dev/sdi bs=512b count=3840 | sha256sum

and the output matches the previous sha256sum meaning it's a good copy:

3840+0 records in

3840+0 records out

1006632960 bytes (1.0 GB, 960 MiB) copied, 14.788 s, 68.1 MB/s

8012fcba8bf71a7dd9e8179af40bce0fec57bb84b7426e4de807130ada63243d -

The hardest part is getting the bytesize correct. Once you have it, the rest is easy.

Community
  • 1
Stuart K. Smith
  • 1