How to merge two folders so as to remove identical files from one, while keeping checksummed differentials?

Question

I have a scenario where I have a recovery from a drive from possibly a different point in time that is not only potentially different in terms of files being present, but also there was some corruption where a number of files are obviously corrupt.

We'll call the left folder "A" and the right folder "B".

It falls to me to merge these two images such that:

1. any files that are present in B that are not in A should be moved to A, and
2. any files that are present in both locations and are the same should be deleted from B, and finally
3. any files that have differences in checksum should remain in B such that these different files can be manually compared between A and B, but nothing else should be left in B except for these files that have different checksums (ie, actual contents).

NOTE: Dates are pretty much immaterial at this point, although it would be nice to preserve the older date in the metadata.

How can this be done cleanly? Unfortunately I have to do this for several tens of TB of data and thus it will be a very lengthy process if I do not figure out how to automate this. It appears that 90-95% of the contents ARE the same, so a "set it and forget it" method should work out for preparation to do the manual comparison.

Answer 1

Step 2 and 3 seem the most difficult, so let's start with those.

There is a tool called rdfind which finds duplicate files. You can decide what to do when a duplicate is detected: in your case you would want to delete it from B: rdfind -deleteduplicates true A B. If identical files exist in A and in B, the one from A is preserved. Other options are to replace the copy with a hard link or a softlink, or just to report the findings.

Then the files that remain in B are either unique to B, or are different in B than in A. To move the unique ones from B to A: mv -i B/* A/ and answer no each time when asked if you want to overwrite. You can automate this with yes no | mv -i B/* A/. If you're using GNU mv, you can use mv --no-clobber B/* A/.

Of course, you want to practice this first before doing it on your real data. You can easily make a tree of hard links to your files in A and B: mkdir training; cp -lr A training; cp -lr B training, and do your practice there.

Answer 2

Here's a method that's simple, but inefficient if a lot of files are missing in A. Just do each step in turn. I assume that there are only directories and regular files (comparing metadata for special files is doable with a bit more work). Warning: untested code.

First, copy files that are present in B but not in A to A. Preserve metadata (timestamps, permissions) as much as possible.

rsync -a --ignore-existing B A

Second, remove duplicates from B. Note that at this point, files that were originally not present in A are now identical.

cd B
find . -type f -exec sh '
  for x; do
    if cmp -s "$x" "$0/$x"; then rm "$x"; fi
  done
' /path/to/A {} +

Optionally, remove empty directories from B.

find B -depth -type d -exec rmdir {} + 2>/dev/null

This is inefficient because at step 2, every file that was already missing from A is now duplicated and will be compared then removed from B. If a lot of files are missing in A, it would be more efficient to make a single pass on B to both move files to A and remove duplicates. This is especially true if A and B are on the same filesystem, which allows moving files cheaply rather than by copying and then deleting the source.

Answer 3

I would start by challenging your requirements. You are trying to do everything in one step. You'd be much better off knowing what you want the restored system to look like before you start restoring files.

It's actually easier to get the diff first than you might think.

Step 1

Get the hash of every file on disk. You're going to have to do this anyway. So might as well get it over and done with. The command below works well if there aren't too many hard links. Assume directories are named /media/A and /media/B.

cd /media/A
find . -type f -exec sha256sum {} + > ~/hashes.txt

This will create a hash of every regular file on the disk. If a file is hard linked it will appear under every name (and have been scanned once for each name).

Step 2

Identify changes with

cd /media/B
sha256sum -c ~/hashes.txt > ~/check.txt

check.txt will now contain three types of line:

• good/file: OK
• missing/file: FAILED open or read
• changed/file: FAILED

Step 3

As a shortcut, you can copy all files that are missing using:

rsync -a --ignore-existing /media/A/ /media/B/

Step 4

Then you only need to worry about changed files:

grep 'FAILED$' ~/check.txt | while read file ; do
    echo "${file%: FAILED}"
done > ~/changed.txt

This gives you changed.txt which will have one file name per line. Every one is a file on both systems that has changed.

It's now up to you to sort through changed.txt and identify which files to keep and which to overwrite to A from B.

Answer 4

assuming you don't have any "newlines" in filenames, this should work:

cd B
find . -type f -print | while read f
do
    [[ -f "A/$f" ]] || { echo mv "$f" "A/$f" ; continue; }
    cmp "$f" "A/$f" && echo rm "$f"
done

Run it, and if it looks good, remove the "echo" words to get the actual commands to run.