How to compare two directories recursively, and for each pair of files with the same name, location, and contents, say which file is older?

Question

In a Linux shell, how to compare two directories recursively, and for each pair of files (including symlinks and directories) with the same location (including the name) in the two directories and the same contents and different modification times, say which file is older? If the two files of a pair have the same age, there should be NO output for this pair. If the two files of a pair have different contents (and the same location), there should be no output for this pair.

Examples of usage:

# First, ensure that /tmp/d1, /tmp/d2, ~/d3, and ~/f don't exist. Then:
$ cd /tmp
$ mkdir d1 d2 ~/d3
$ touch d1/f && sleep 1 && touch d2/f ~/d3/f ~/f
$ echo "g1" > d1/g
$ echo "g2" > d2/g
$ echo "g3" > ~/d3/g
$ compare_times.sh d1 d2
d1/f is older than d2/f
$ cd d1
$ compare_times.sh . ../d2
f is older than ../d2/f
$ cd ../d2
$ compare_times.sh . ../d1
../d1/f is older than f
$ cd ..
$ compare_times.sh /tmp/d1 d2
/tmp/d1/f is older than d2/f
$ compare_times.sh d1 /tmp/d2
d1/f is older than /tmp/d2/f
$ compare_times.sh d1 ~/d3
d1/f is older than ~/d3/f
$ cd d1
$ compare_times.sh ~ .
f is older than ~/f

Our comparison script compare_times.sh (naturally, you may opt for compare_times.zsh instead if you happen to program in zsh instead of dash or bash) should accept two arguments, which in general can be absolute or relative paths of directories (including simply .), potentially terminated by /.

The unquoted string ~ should be interpreted as the home directory as usual in any of the two arguments. It'd probably be an overkill to actually print the home directory in the output; the concise ~ in the output should do.

The output should be possibly concise as usual: e.g., a path // should be shortened to / (for all nonempty and all ), a path /./ should be shortened to / (for all nonempty and all ), a path / should be shortened to (for all nonempty ), a path ./ should be shortened to (for all nonempty not starting with /), and a prefix of 3 or more / not followed by a / should be shortened to the prefix /.

(Rationale. We prefer concise output because it will be manually inspected later to try to find the reasons for the different timestamps; after inspection, the user plans to manually equalize the timestamps of the otherwise equal files (depending on the result of the inspection). We found out that superfluous characters produce longer output lines and waste the user's time while he/she registers information unrelated to his/her primary question. Longer output lines tend to wrap at the window border more often, and selecting a two-line path with a mouse might take slightly longer than selecting a single-line path.)

There was a question with answers (including one particular solution with a bash script with four arguments and another solution with a zsh script) earlier somewhere on one of the SE sites, but I can't find this question any longer. Whoever finds it, if it still exists, please feel free to mark this question as a duplicate of the found one.

Many shells expand an unquoted ~ to the value of $HOME, but ~ is also otherwise a valid name for a file or directory. To me, it would be as wrong for your script to treat ~ as meaning $HOME as it would be for say rm. For instance, what should your script output after mkdir '~~' '~'; echo a > '~~/file'; echo b > '~/file'; your-script '~' '~~'? — Stéphane Chazelas, Jan 27 '24 at 14:56
Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on [meta], or in [chat]. Comments continuing discussion may be removed. — Jeff Schaller, Jan 27 '24 at 17:25

Chris Davies · Answer 1 · 2024-01-28T15:03:50.407

4

This script will compare two directory trees per your requirement. File names are not restricted. (If you don't have GNU comm, find and sort, or other equivalent tools that can handle NULL-terminated records, you'll have to forego the ability to handle newlines in file names.)

#!/bin/bash
#
d1=$1 d2=$2
Identify the set of matching file names
LC_ALL=C comm -z -12 

        <( cd -P -- "$d1" && find . -type f -print0 | LC_ALL=C sort -z ) 

        <( cd -P -- "$d2" && find . -type f -print0 | LC_ALL=C sort -z ) |
    while IFS= read -rd '' fn
    do
        # Tidy the filenames
        fn=${fn#./}
        f1="$d1/$fn"
        f2="$d2/$fn"
    # Compare content
    if cmp -s -- &quot;$f1&quot; &quot;$f2&quot;
    then
        # Report older/newer file pairs (not those the same)
        [[ &quot;$f1&quot; -ot &quot;$f2&quot; ]] &amp;&amp; printf '%s is older than %s\n' &quot;${f1#./}&quot; &quot;${f2#./}&quot;
        [[ &quot;$f2&quot; -ot &quot;$f1&quot; ]] &amp;&amp; printf '%s is older than %s\n' &quot;${f2#./}&quot; &quot;${f1#./}&quot;
    fi
done

edited Jan 28 '24 at 15:03

answered Jan 26 '24 at 22:55

Chris Davies

116,213
16
160
287

The -print0 aspect is brilliant, thank you for that. It never ceases to amaze me why people put SPACE characters in directory names. – J_H Jan 27 '24 at 23:01
First, thank you a lot, and thank you once again. Second, tidying up doesn't work: mkdir d1 d2 && touch d1/f && sleep 1 && touch d2/f && cd d1 && compare_times.sh . ../d2 yields ./f is older than ../d2/f, whereas we expect f is older than ../d2/f. Third, the second comparison can be skipped if the first one succeeded: if [[ "$d1/$fn" -ot "$d2/$fn" ]]; then printf '%s is older than %s\n' "$d1/$fn" "$d2/$fn"; else if [[ "$d2/$fn" -ot "$d1/$fn" ]]; then printf '%s is older than %s\n' "$d2/$fn" "$d1/$fn"; fi; fi. – AlMa1r Jan 28 '24 at 01:00
Finally, why do we say printf '%s is older than %s' "$d1/$fn" "$d2/$fn" and not simpler echo "$d1/$fn" is older than "$d2/$fn"? – AlMa1r Jan 28 '24 at 01:03
You specified the directory . to search in, so the file that matched is ./f. There's nothing stopping you taking this code and adapting it further in your own environment

Chris Davies

Jan 28 '24 at 07:58

I felt mine was easier to read

– Chris Davies Jan 28 '24 at 07:58

Stéphane Chazelas · Answer 2 · 2024-01-28T13:01:00.923

With zsh:

#! /bin/zsh -
d1=${1?} d2=${2?}
All regular files in $d1, with $d1/ removed from the start of their
path in the second step
l1=( ${d1?}/*/(ND-.) ); l1=( ${l1#$d1/} )
same for $2
l2=( ${d2?}/*/(ND-.) ); l2=( ${l2#$d2/} )
loop over the intersection of the $l1 and $l2 arrays
for f ( ${l1:*l2} ) {
  f1=$1/$f f2=$2/$f
  cmp -s -- $f1 $f2 || continue
  if [[ $f1 -nt $f2 ]] {
    print -r -- $f1 is newer than $f2
  } elif [[ $f1 -ot $f2 ]] {
    print -r -- $f1 is older than $f2
  } else {
    print -r -- $f1 and $f2 are the same age
  }
}

I'm not attempting to replace /home/your-home-dir with ~ or ./file with file or /path/to/current/dir/foo/bar with foo/bar as IMO it would only introduce ambiguity and confusion in addition to making the code more complex.

If you still wanted to, there are several approaches you could take. Using d1=${1:P} would store the absolute canonical path of $1 in $d1. Being a full path would make it easier to identify ~, ~users in them, but would fall apart if $HOME or the expansion of ~user actually contains symlinks components. And the fact that symlinks are resolved could easily confuse the user.

A compromise would be to do only the safest of path simplification such as simplify the //././.// to /, and remove $PWD/ and ./ from the start of paths, and then use the D parameter expansion flag to identify ~user in paths. For instance, you could compute a $display_f1 from $f1 (and same for $f2) with something like:

set -o extendedglob
display_f1=${(D)${${f1//\/((.|)\/)##/\/}#$PWD/}#./}

For example:

$ f1='///home/././chazelas//foo/bar baz'
$ pwd
/home/chazelas
$ display_f1=${(D)${${f1//\/((.|)\/)##/\/}#$PWD/}#./}
$ typeset display_f1
display_f1='foo/bar\ baz'

$ cd /
$ display_f1=${(D)${${f1//\/((.|)\/)##/\/}#$PWD/}#./}
$ typeset display_f1
display_f1='~/foo/bar\ baz'

$ f1='./~/foo'
$ display_f1=${(D)${${f1//\/((.|)\/)##/\/}#$PWD/}#./}
$ typeset display_f1
display_f1='\~/foo'

But:

$ cd /home
$ f1=./chazelas/foo
$ display_f1=${(D)${${f1//\/((.|)\/)##/\/}#$PWD/}#./}
$ typeset display_f1
display_f1=chazelas/foo

(and not ~/foo).

The full solution with that included would look like:

#! /bin/zsh -
set -o extendedglob
d1=${1?} d2=${2?}
All regular files in $d1, with $d1/ removed from the start of their
path in the second step
l1=( ${d1?}/*/(ND-.) ); l1=( ${l1#$d1/} )
same for $2
l2=( ${d2?}/*/(ND-.) ); l2=( ${l2#$d2/} )
loop over the intersection of the $l1 and $l2 arrays
for f ( ${l1:*l2} ) {
  f1=$1/$f f2=$2/$f
  display_f1=${(D)${${f1///((.|)/)##//}#$PWD/}#./}
  display_f2=${(D)${${f2///((.|)/)##//}#$PWD/}#./}
  cmp -s -- $f1 $f2 || continue
  if [[ $f1 -nt $f2 ]] {
    print -r -- $display_f1 is newer than $display_f2
  } elif [[ $f1 -ot $f2 ]] {
    print -r -- $display_f1 is older than $display_f2
  } else {
    print -r -- $display_f1 and $display_f2 are the same age
  }
}

Thank you! First, I added the rationale for concise paths to the question. Second, I'm new to zsh, so, unfortunately, I cannot responsibly comment on your code yet – it'll take me some time to finish the official zsh tutorial. Having said this, I think that instead of lexicographically comparing the times via > and <, it might be possible compare the timestamps of regular files via [[ file1 -ot file2]], of links via find file2 -newer file1, and of directories via find dir2 -prune -newer dir1. — AlMa1r, Jan 28 '24 at 02:33
Yes, you're right, there's no benefit to computing all mtime upfront like I did. I had reservations about using -nt, -ot which can give misleading results if files are not accessible after symlink resolution, but they are rather irrelevant here as the -. qualifier would guarantee they're accessible. See edit. — Stéphane Chazelas, Jan 28 '24 at 09:59
We shouldn't print anything if the two files of a pair have the same age. I emphasized this now in the original question. — AlMa1r, Jan 28 '24 at 12:57
@AlMa1r, sorry typo, should have been //./././/. Fixed now. It's true that exactly two /s at the start of a path may have a special meaning on some systems, but I'm ignoring it here. I meant those as found in a path to be replaced with / like in the example. — Stéphane Chazelas, Jan 28 '24 at 12:58
I've just realized that the simplification of a path at its beginning should be even more conservative than I thought previously. I repaired the original post in this matter. — AlMa1r, Jan 28 '24 at 14:54

AlMa1r · Answer 3 · 2024-01-29T01:38:10.227

For regular files, symbolic links, and directories, based on http://stackoverflow.com/a/66468913 and http://unix.stackexchange.com/a/767825 (thanks to both!):

#!/bin/bash
# Syntax: compare_times.sh directory_1 directory_2
# Semantics: for pairs of files, directories, and symlinks with the same path at any depth in both directory_1 and directory_2 and the same type, compare their modification times and, if they differ, say which file/dir/symlink is older.
thanks to http://stackoverflow.com/a/66468913. The original is in http://github.com/python/cpython/blob/main/Lib/posixpath.py#L377 .
normpath() {
  local IFS=/ initial_slashes='' comp comps=()
  if [[ $1 == /* ]]; then
    initial_slashes='/'
    [[ $1 == //* && $1 != ///* ]] && initial_slashes='//'
  fi
  for comp in $1; do
    if [[ -n ${comp} && ${comp} != '.' ]]; then
      if [[ ${comp} != '..' || (-z ${initial_slashes} && ${#comps[@]} -eq 0) || (${#comps[@]} -gt 0 && ${comps[-1]} == '..') ]]; then
        comps+=("${comp}")
      elif ((${#comps[@]})); then
        unset 'comps[-1]'
      fi
    fi
  done
  comp="${initial_slashes}${comps[*]}"
  printf '%s\n' "${comp:-.}"
}
thanks to http://unix.stackexchange.com/a/767825
if [[ $# -eq 2 ]]; then
  dir1=$1
  if [[ -d $dir1 ]]; then
    dir2=$2
    if [[ -d $dir2 ]]; then
      # Identify the set of matching names of regular files:
      LC_ALL=C comm -z -12 

               <( cd -P -- "$dir1" && find . -type f -print0 | LC_ALL=C sort -z ) 

               <( cd -P -- "$dir2" && find . -type f -print0 | LC_ALL=C sort -z ) |
        while IFS= read -rd '' fn; do
          # Tidy the file names:
          file1="$(normpath "$dir1/$fn")"
          file2="$(normpath "$dir2/$fn")"
          # Compare file contents:
          if cmp -s -- "$file1" "$file2"; then
            # Report older/newer regular-file pairs:
            if [[ "$file1" -ot "$file2" ]]; then
              printf '%s is older than %s\n' "$file1" "$file2"
            elif [[ "$file2" -ot "$file1" ]]; then
              printf '%s is older than %s\n' "$file2" "$file1"
            fi
          fi
        done
      # Identify the set of matching names of symlinks:
      LC_ALL=C comm -z -12 

               <( cd -P -- "$dir1" && find . -type l -print0 | LC_ALL=C sort -z ) 

               <( cd -P -- "$dir2" && find . -type l -print0 | LC_ALL=C sort -z ) |
        while IFS= read -rd '' fn; do
          # Tidy the link names:
          link1="$(normpath "$dir1/$fn")"
          link2="$(normpath "$dir2/$fn")"
          # Compare the symlinks:
          if cmp -s <(readlink -- "$link1") <(readlink -- "$link2"); then
            ## Report older/newer symlink pairs:
            # if (find "$link2" -prune -newer "$link1" -printf 'a\n' | grep -q a) then
            #   printf '%s is older than %s\n' "$link1" "$link2"
            # elif (find "$link1" -prune -newer "$link2" -printf 'a\n' | grep -q a) then
            #   printf '%s is older than %s\n' "$link2" "$link1"
            # fi
            find "$link2" -prune -newer "$link1" -printf "$link1 is older than $link2\n"
            find "$link1" -prune -newer "$link2" -printf "$link2 is older than $link1\n"
          fi
        done
      # Identify the set of matching names of directories:
      LC_ALL=C comm -z -12 

               <( cd -P -- "$dir1" && find . -type d -print0 | LC_ALL=C sort -z ) 

               <( cd -P -- "$dir2" && find . -type d -print0 | LC_ALL=C sort -z ) |
        while IFS= read -rd '' fn; do
          # Tidy the directory names:
          subdir1="$(normpath "$dir1/$fn")"
          subdir2="$(normpath "$dir2/$fn")"
          # Compare the directory contents:
          if cmp -s <(ls -Al --full-time -- "$subdir1") <(ls -Al --full-time -- "$subdir2"); then
            # Report older/newer directory pairs:
            if [[ "$subdir1" -ot "$subdir2" ]]; then
              printf '%s is older than %s\n' "$subdir1" "$subdir2"
            elif [[ "$subdir2" -ot "$subdir1" ]]; then
              printf '%s is older than %s\n' "$subdir2" "$subdir1"
            fi
          fi
        done
    else
      printf '%s is not an existing directory.\n' "$dir2"
      exit 66 # EX_NOINPUT

    fi
  else
    printf '%s is not an existing directory.\n' "$dir1"
    exit 66 # EX_NOINPUT

  fi
else
  echo "Wrong number of arguments." >&2
  exit 64 # EX_USAGE
fi

Improvements are welcome.

Since this is essentially a rework of my own answer, please you woukd ✔ accept mine as credit — Chris Davies, Jan 28 '24 at 17:56
@ChrisDavies With the greatest respect and admiration to your efforts: no or at least not now because your version doesn't work for symlinks or directories, and I still have to test everything more thoroughly, as well as to see first how Stéphane's version works (which doesn't happen before I finish going through my zsh tutorial) before even thinking of ticking anything at all. I also found minor issues that have to be tackled first. So this answer might be changed, replaced, or deleted, and your answer, which I upvoted already, might be unvoted or changed as well. — AlMa1r, Jan 28 '24 at 18:27
@ChrisDavies So I'd be happy if I could kindly ask you to bear with me for a few days (perhaps even a week) at least. — AlMa1r, Jan 28 '24 at 18:29
You didn't show those in your example scenario, and didn't mention them in your question at all. From here it does feel a little like shifting the goalposts. If Stéphane's works better for you, great; please accept his. — Chris Davies, Jan 28 '24 at 19:03
@ChrisDavies What you wrote is partially wrong. Though it is true that the scenario doesn't compare symlinks and directories, the part “files (including symlinks and directories)“ has always been there. As for Stéphane's version with zsh, I still have to understand it – so nothing will happen soon in this regard. The goals are of course slightly changing, but only on the level of the details (e.g., how to simplify the output properly). This is normal in software development and not specific to the original question — in fact, many questions are reformulated and improved here at SE. — AlMa1r, Jan 28 '24 at 19:08

How to compare two directories recursively, and for each pair of files with the same name, location, and contents, say which file is older?

3 Answers3

Identify the set of matching file names

All regular files in $d1, with $d1/ removed from the start of their

path in the second step

same for $2

loop over the intersection of the $l1 and $l2 arrays

All regular files in $d1, with $d1/ removed from the start of their

path in the second step

same for $2

loop over the intersection of the $l1 and $l2 arrays

thanks to http://stackoverflow.com/a/66468913. The original is in http://github.com/python/cpython/blob/main/Lib/posixpath.py#L377 .

thanks to http://unix.stackexchange.com/a/767825