Hash a file by 64MB blocks?

Question

I have a very large file (200GB). Apparently when I transfer it over it did not copy correctly. The sha1 hash on both are different. Is there a way I can divide the file up to blocks (like 1MB or 64MB) and output a hash for each block? Then compare/fix?

I might just write a quick app to do it.

Answer 1

That "quick app" already exists, and is relatively common: rsync. Of course, rsync will do a whole lot more than that, but what you want is fairly simple:

rsync -cvP --inplace user@source:src-path-to-file dest-path-to-file   # from the destination
rsync -cvP --inplace src-path-to-file user@dest:dest-path-to-file     # from the source

That will by default use ssh (or maybe rsh, on a really old version) to make the connection and transfer the data. Other methods are possible, too.

Options I passed are:

• -c — skip based on checksums, not file size/mtime. By default rsync optimizes and skips transfers where the size & mtime match. -c forces it to compute the checksum (which is an expensive operation, in terms of I/O). Note this is a block-based checksum (unless you tell it to do whole files only), and it'll only transfer the corrupted blocks. The block size is automatically chosen, but can be overridden with -B (I doubt there is any reason to).
• -v — verbose, will give some details (which file it's working on)
• -P — turns on both partial files (so if it gets halfway through, it won't throw out the work) and a progress bar.
• --inplace — Update the existing file, not a temporary file (which would then replace the original file). Saves you from having a 200GB temporary file. Also implies partial files, so that -P is partially redundant.

BTW: I'm not sure how you did the original transfer, but if it was sftp/scp, then something is very wrong—those fully protect from any corruption on the network. You really ought to track down the cause. Defective RAM is a relatively common one.

Answer 2

If you want to re-transfer the file to another machine via a network connection, use rsync.

If you want to get an idea of where the differences are, the easiest way would be to have the two versions on the same machine. If you don't want to do that because bandwidth is too expensive, here are ways you can checkum chunks of files.

This method relies on head -c leaving the file position where it left off, and pre-computes the size to know where to end the loop.

n=$(($(wc -c <very_large_file) / (64*1024*1024) + 1))
i=0
while [ $i -gt $n ]; do
    head -c 64m | sha256sum
    i=$((i+1))
done <very_large_file

This method relies on head -c leaving the file position where it left off, and uses cksum to find the size of each chunk (a short chunk indicates the end of the file).

while true; do
    output=$(head -c 64m | cksum)
    size=${output#* }; size=${output%% *}
    if [ $size -eq 0 ]; then break; fi
    echo "$output"
done <very_large_file

This method calls dd to skip to the desired start position for each chunk.

n=$(($(wc -c <very_large_file) / (64*1024*1024) + 1))
i=0
while [ $i -gt $n ]; do
    dd if=very_large_file ibs=64m skip=$i count=1 | sha256sum
    i=$((i+1))
done <very_large_file

Answer 3

You should probably look at split:

Here is the man page with examples:

https://ss64.com/bash/split.html