With
diff -r
I can do this task, however it takes so long because diff checks file's content.
I want something that determine that two files are the same regarding of their size, last modified, etc. But no checking bit by bit the file (for example a video takes sooo long).
Is there any other way?
rsync compares only metadata by default.
rsync -n -a -i --delete source/ target/
explanation:
-n compare but do not actually copy or delete <-- THIS IS IMPORTANT!!1-a compare all metadata-i print one line of information per file--delete also report files which are in target but not in sourcenote: it is important to append the directory names with a slash. this is an rsync thing.
also note: rsync is a powerful tool. some explanation above is crudely simplified for the context of this question. especially -a is much more complex than just "all metadata".
you can shorten the one letter options like this
rsync -nai --delete source/ target/
you can provide -i twice to also have information printed for files that are identical
rsync -naii --delete source/ target/
example output:
.d..t...... ./ (directory with different timestamp)
>f.st...... modifiedfile (file with different size and timestamp)
>f+++++++++ newfile (file in source but not in target)
*deleting removedfile (file in target but not in source)
.f samefile (file that has same metadata. only with -ii)
remember that rsync only compares metadata. that means if the file content changed but metadata is still the same then rsync will report that file is same. this is an unlikely scenario. typically if data changes then metadata will also change. so either trust that when metadata is same then data is same, or you have to compare file data bit by bit.
bonus: for progress information see here: Estimate time or work left to finish for rsync?
I've just discovered tree.
tree old_dir/ > tree_old
tree new_dir/ > tree_new
vimdiff tree_old tree_new
Use the -q (--brief) option with diff -r (diff -qr). From the info page for GNU diff:
1.6 Summarizing Which Files Differ
When you only want to find out whether files are different, and you don't care what the differences are, you can use the summary output format. In this format, instead of showing the differences between the files,
diff' simply reports whether files differ. The--brief' (`-q') option selects this output format.This format is especially useful when comparing the contents of two directories. It is also much faster than doing the normal line by line comparisons, because `diff' can stop analyzing the files as soon as it knows that there are any differences.
This will not compare line by line, but rather the file as a whole, which greatly speeds up the processor (what' you're looking for).
Here's a quick Python script that will check that the filenames, mtimes, and file sizes are all the same:
import os
import sys
def get_stats(path):
for pathname, dirnames, filenames in os.walk(path):
for filename in (os.path.join(pathname, x) for x in filenames):
stat = os.stat(filename)
yield filename[len(path):], stat.st_mtime, stat.st_size
left = tuple(get_stats(sys.argv[1]))
right = tuple(get_stats(sys.argv[2]))
print(set(left) ^ set(right))
sys.exit(left != right)
If you only need to know if files from two file system branch are different (without look inside files) you can do something like this:
find /opt/branch1 -type f | sort | xargs -i md5sum {} >/tmp/branch1;
find /opt/branch2 -type f | sort | xargs -i md5sum {} >/tmp/branch2;
diff /tmp/branch1 /tmp/branch2;
HTH
Based on Chris Down's script, this script is a little more "visual". Calling it with two arguments folder1 and folder2, it walks the first folder and for each file searches a corresponding file in the second folder. If it is found, the relative path is printed in green, if they have different modified time or size, it is printed in yellow, and if it is not found then it is printed in red.
#!/usr/bin/env python
import os
import sys
from termcolor import colored
def compare_filestats(file1,file2):
"""
Compares modified time and size between two files.
Return:
-1 if file1 or file2 does not exist
0 if they exist and compare equal
1 if they have different modified time, but same size
2 if they have different size, but same modified time
3 if they have different size, and different modified time
"""
if not os.path.exists(file1) or not os.path.exists(file2):
return -1
stat1 = os.stat(file1)
stat2 = os.stat(file2)
return (stat1.st_mtime != stat2.st_mtime) \
+ 2*(stat1.st_size != stat2.st_size)
def compare_folders(folder1,folder2):
"""
folder1: serves as reference and will be walked through
folder2: serves as target and will be querried for each file in folder1
Prints colored status for each file in folder1:
missing: file was not found in folder2
mtime : modified time is different
size : filesize is different
ok : found with same filestats
"""
for dirpath, dirnames, filenames in os.walk(folder1):
for file1 in ( os.path.join(dirpath, x) for x in filenames ):
relpath = file1[len(folder1):]
file2 = os.path.join( folder2, relpath )
comp = compare_filestats(file1,file2)
if comp < 0:
status = colored('[missing]','red')
elif comp == 1:
status = colored('[mtime ]','yellow')
elif comp >= 2:
status = colored('[size ]','yellow')
else:
status = colored('[ok ]','green')
print status, relpath
if __name__ == '__main__':
compare_folders(sys.argv[1],sys.argv[2])
Note that this is not sufficient to decide whether the two folders are the same, you would need to run it both ways to make sure. In practice if you just want to know whether the folders are the same, then Chris' script is better. If you want to know what's missing or different from one folder to another, then my script will tell you.
NOTE: you will need termcolor installed, pip install termcolor.
If you'd like to compare only a structure and some basic info about files, you can try something like this:
diff <(cd $DIR1 && ls -laR) <(cd $DIR2 && ls -laR)
I didn't test it, so any edits are welcome :)
<() has its own environment. Edited.
– user
Dec 11 '16 at 12:43
Save the next code as: diffm.sh:
# DIFF with Modification date - a .sh (dash; bash; zsh - compatible)
# "diff utility"-like script that can compare files in two directory
# trees by path, size and modification date
Disclaimer:
By using this program you are assuming full responsibility (and the
author of this program shall not be held liable) for any data loss
or damage that may result from the use or misuse of this program.
Proc1 () {
new="$line"
first_remaining_part="${new%"///////"}"
time="${first_remaining_part#"///////"}"
first_part="${new%%" ///////"}"
#last_part="${new##"///////"}"
seconds_since_epoch="${new##*" "}"
#convert seconds_since_epoch to base 10 (remove 0's padding):
temp="${seconds_since_epoch#"0"*[1-9]}"
zeros="${seconds_since_epoch%"$temp"}"
zeros="${zeros%?}" #Remove last non-0 digit
seconds_since_epoch="${seconds_since_epoch#"$zeros"}"
time_zone="${time##*" "}"
time="${time%"."*}"
new="$first_part ///// $time $time_zone ///// $seconds_since_epoch"
printf '%s\n' "$new"
}
Proc2 () {
cd "$dir1" >/dev/null 2>/dev/null
count_files=0
{
IFS="
"
set -f
#FilePath / FileType / FileSize / FileTime / SecondsSinceEpoch
if [ "$OS" = "Linux" -o "$OS" = "Windows" ]; then
find . -not -type d -printf '%p ///// <_ ///// %s ///////%TY-%Tm-%Td %TT %Tz///////' -exec stat --printf ' %Y\n' {} ; 2>/dev/null|
{
while read line; do {
count_files=$((count_files + 1));
PrintJustInTitle "Analyzing file $count_files...";
Proc1;
}; done;
printf "$count_files">$RAMLocation/diffm_count.txt;
}
elif [ "$OS" = "Mac" ]; then
gfind . -not -type d -printf '%p ///// <_ ///// %s ///////%TY-%Tm-%Td %TT %Tz///////' -exec gstat --printf ' %Y\n' {} ; 2>/dev/null|
{
while read line; do {
count_files=$((count_files + 1));
PrintJustInTitle "Analyzing file $count_files...";
Proc1;
}; done;
printf "$count_files">$RAMLocation/diffm_count.txt;
}
fi
unset IFS
set +f
}
cd "$initial_dir"
}
Proc3 () {
cd "$dir2" >/dev/null 2>/dev/null
#Get previous line count value from diffm_count.txt:
read count_files<"$RAMLocation/diffm_count.txt"
#Validate count_files as a number:
count_files=$((count_files))
{
IFS="
"
set -f
#FilePath / FileType / FileSize / FileTime / SecondsSinceEpoch
if [ "$OS" = "Linux" -o "$OS" = "Windows" ]; then
find . -not -type d -printf '%p ///// >_ ///// %s ///////%TY-%Tm-%Td %TT %Tz///////' -exec stat --printf ' %Y\n' {} ; 2>/dev/null|
{
while read line; do {
count_files=$((count_files + 1));
PrintJustInTitle "Analyzing file $count_files...";
Proc1;
}; done;
printf "$count_files">$RAMLocation/diffm_count.txt;
}
elif [ "$OS" = "Mac" ]; then
gfind . -not -type d -printf '%p ///// >_ ///// %s ///////%TY-%Tm-%Td %TT %Tz///////' -exec gstat --printf ' %Y\n' {} ; 2>/dev/null|
{
while read line; do {
count_files=$((count_files + 1));
PrintJustInTitle "Analyzing file $count_files...";
Proc1;
}; done;
printf "$count_files">$RAMLocation/diffm_count.txt;
}
fi
unset IFS
set +f
}
#Get total line count value from diffm_count.txt:
read count_files<"$RAMLocation/diffm_count.txt"
#Validate count_files as a number:
count_files=$((count_files))
#"." in front of the number is for displaying it as first line after the sort operation:
printf ".$count_files"
cd "$initial_dir"
}
GetOS () {
OS_kernel_name=$(uname -s)
case "$OS_kernel_name" in
"Linux")
eval $1="Linux"
;;
"Darwin")
eval $1="Mac"
;;
"CYGWIN"*|"MSYS"*|"MINGW"*)
eval $1="Windows"
;;
"")
eval $1="unknown"
;;
*)
eval $1="other"
;;
esac
}
PrintInTitle () {
printf "\033]0;%s\007" "$1"
}
PrintJustInTitle () {
PrintInTitle "$1">/dev/tty
}
trap1 () {
CleanUp
printf "\nAborted.\n">/dev/tty
}
CleanUp () {
#Clear the title:
PrintJustInTitle ""
#Restore "INTERRUPT" (CTRL-C) and "TERMINAL STOP" (CTRL-Z) signals:
trap - INT
trap - TSTP
}
GetSetRAMLocation () {
GetOS OS
if [ "$OS" = "Linux" ]; then
RAMLocation='/dev/shm'
elif [ "$OS" = "Mac" ]; then
RAMLocation='/Volumes/RamDiskDIFFM'
sizeInMB = "1"
if [ ! -e "/Volumes/RamDiskDIFFM" ]; then
diskutil partitionDisk $(hdiutil attach -nomount ram://$((2048sizeInMB))) 1 GPTFormat APFS 'RamDiskDIFFM' '100%'&&{
printf "">"$RAMLocation/diffm_count.txt"&&{
error="false"
}||{
printf '\n%s\n\n' "Error: Could not write to RAM ("$RAMLocation/diffm_count.txt"). Write to current directory instead? [ Yes (=continue) / No (=Enter=abort) ]">/dev/tty
read answer
case "$answer" in
"Y"|"y"|"Yes"|"yes")
answer="yes"
;;
"N"|"n"|"No"|"no")
answer="no"
;;
"")
answer="no"
;;
)
answer="no"
;;
esac
if [ "$answer" = "yes" ]; then
RAMLocation="$initial_dir"
error="false"
elif [ "$answer" = "no" ]; then
printf "\nAborted.\n\n">/dev/tty
error="true"
fi
}
}||{
printf '\n%s\n\n' "ERROR: Could not create RAM Volume!">/dev/tty
error="true"
}
fi
if [ "$error" = "true" ]; then
CleanUp
exit 1
fi
elif [ "$OS" = "Windows" ]; then
#On Windows, the RAMLocation variable is not currently designed to point to a RAM Drive path:
RAMLocation="$initial_dir"
elif [ $OS" = "other ]; then
#On other operating systems, the RAMLocation variable is not currently designed to point to a RAM Drive path:
RAMLocation="$initial_dir"
else
printf '\n%s\n\n' "ERROR: Could not get OS!">/dev/stderr
CleanUp
exit 1
fi
}
DestroyArray () {
eval array_len=$(($1_0))
j=0;
while [ "$j" -lt "$array_len" ]; do
j=$((j+1))
unset $1\_$j
done
unset $1\_0
}
DisplayHelp () {
printf "\n"
printf "diffm - DIFF with Modification date\n"
printf "\n"
printf " What it does:\n"
printf " - compares the files (recursively) in the two provided directory tree paths (<dir_tree1> and <dir_tree2>) by:\n"
printf " 1. Path\n"
printf " 2. Size\n"
printf " 3. Modification date\n"
printf " Syntax:\n"
printf " <caller_shell> '/path/to/diffm.sh' <dir_tree1> <dir_tree2> [flags]\n"
printf " - where:\n"
printf " - <caller_shell> can be any of the shells: dash, bash, zsh, or any other shell compatible with the "dash" shell syntax\n"
printf " - '/path/to/diffm.sh' represents the path of this script\n"
printf " - <dir_tree1> and <dir_tree2> represent the directory trees to be compared\n"
printf " - [flags] can be:\n"
printf " --help or -h\n"
printf " Displays this help information\n"
printf " Output:\n"
printf " - lines starting with '<' signify files from <dir_tree1>\n"
printf " - lines starting with '>' signify files from <dir_tree2>\n"
printf " Notes:\n"
printf " - only files in the two provided directory tree paths are compared, not also directories\n"
printf "\n"
}
##################
MAIN START
##################
GetOS OS
if [ "$OS" = "Linux" -o "$OS" = "Windows" ]; then
error1="false"
error2="false"
error3="false"
{ sort --version-sort --help >/dev/null 2>/dev/null; } || { error1="true"; }
{ stat --help >/dev/null 2>/dev/null; } || { error2="true"; }
{ find --help >/dev/null 2>/dev/null; } || { error3="true"; }
if [ "$error1" = "true" -o "$error2" = "true" -o "$error3" = "true" ]; then
{
printf "\n"
if [ "$error1" = "true" ]; then printf '%s' "ERROR: Could not run \"sort --version-sort\" (necessary in order for this script to function correctly)!"; fi
if [ "$error2" = "true" ]; then printf '%s' "ERROR: Could not run \"stat\" (necessary in order for this script to function correctly)"; fi
if [ "$error3" = "true" ]; then printf '%s' "ERROR: Could not run \"find\" (necessary in order for this script to function correctly)"; fi
printf "\n"
}>/dev/stderr
exit
else
sort_command="sort --version-sort"
fi
elif [ "$OS" = "Mac" ]; then
error1="false"
error2="false"
error3="false"
{ gsort --version-sort --help >/dev/null 2>/dev/null; } || { error1="true"; }
{ gstat --help >/dev/null 2>/dev/null; } || { error2="true"; }
{ gfind --help >/dev/null 2>/dev/null; } || { error3="true"; }
if [ "$error1" = "true" -o "$error2" = "true" -o "$error3" = "true" ]; then
{
printf "\n"
if [ "$error1" = "true" ]; then printf '%s' "ERROR: Could not run \"gsort --version-sort\" (necessary in order for this script to function correctly)!"; fi
if [ "$error2" = "true" ]; then printf '%s' "ERROR: Could not run \"gstat\" (necessary in order for this script to function correctly)"; fi
if [ "$error3" = "true" ]; then printf '%s' "ERROR: Could not run \"gfind\" (necessary in order for this script to function correctly)"; fi
printf "\n"
printf "\n"
printf '%s\n' "You can install them by installing \"homebrew\" and then GNU \"coreutils\":"
printf '%s\n' "sudo ruby -e \"\$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\""
printf '%s\n' "sudo brew install coreutils"
}>/dev/stderr
exit
else
sort_command="gsort --version-sort"
fi
elif [ "$OS" = "unknown" ]; then
printf '\n%s\n\n' "ERROR: Could not get OS!">/dev/stderr
CleanUp
exit 1
elif [ "$OS" = "other" ]; then
printf '\n%s\n\n' "ERROR: OS currently not supported!">/dev/stderr
CleanUp
exit 1
fi
#Get the program parameters into the array "params":
params_count=0
for i; do
params_count=$((params_count+1))
eval params_$params_count="$i"
done
params_0=$((params_count))
if [ "$params_0" = "0" ]; then #if no parameters are provided: display help
DisplayHelp
CleanUp && exit 0
fi
#Create a flags array. A flag denotes special parameters:
help_flag="0"
i=1;
j=0;
while [ "$i" -le "$((params_0))" ]; do
eval params_i="${params_$i}"
case "${params_i}" in
"--help" | "-h" )
help_flag="1"
;;
* )
j=$((j+1))
eval selected_params_$j="$params_i"
;;
esac
i=$((i+1))
done
selected_params_0=$j
#Rebuild params array:
DestroyArray params
for i in $(seq 1 $selected_params_0); do
eval params_$i="${selected_params_$i}"
done
params_0=$selected_params_0
if [ "$help_flag" = "1" ]; then
DisplayHelp
else
#Check program arguments:
if [ "$params_0" -lt "2" ]; then
printf '\n%s\n' "ERROR: To few program parameters provided (expected two: <dir_tree1> and <dir_tree2>)!">/dev/stderr
exit 1
elif [ "$params_0" -gt "2" ]; then
printf '\n%s\n' "ERROR: To many program parameters provided (expected two: <dir_tree1> and <dir_tree2>)!">/dev/stderr
exit 1
else #two program arguments are provided:
mask1="$params_1"
mask2="$params_2"
fi
#If two program arguments are provided (<dir_tree1> and <dir_tree2>) proceed to checking them:
initial_dir="$PWD" #Store initial dir
error_encountered="false"
[ -e "$params_1" -a -d "$params_1" ] && cd "$params_1" >/dev/null 2>/dev/null && {
dir1="$PWD"
cd "$initial_dir"
}||{
printf '\n%s\n' "ERROR: \"$params_1\" does not exist as a directory or is not accessible!">/dev/stderr
error_encountered="true"
}
printf "\n">/dev/tty
[ -e "$params_2" -a -d "$params_2" ] && cd "$params_2" >/dev/null 2>/dev/null && {
dir2="$PWD"
cd "$initial_dir"
}||{
printf '%s\n' "ERROR: \"$params_2\" does not exist as a directory or is not accessible!">/dev/stderr
error_encountered="true"
}
if [ "$error_encountered" = "true" ]; then
printf "\n">/dev/stderr
exit
fi
GetSetRAMLocation
#Trap "INTERRUPT" (CTRL-C) and "TERMINAL STOP" (CTRL-Z) signals:
trap 'trap1' INT
trap 'trap1' TSTP
#Proceed to <dir_tree1> and <dir_tree2> compare:
IFS="
"
set -f
unset count_total
unset previous_line
unset current_line
unset previous_case
skip="0"
count="0"
found_previous="false"
found_previous_last_line_case="false"
for l in $(
{
PrintJustInTitle "Loading file data, please wait...";
Proc2;
Proc3;
PrintJustInTitle "Sorting results, please wait...";
}|eval $sort_command;
); do
if [ -z "$count_total" ]; then
# #? is for removing the "." in front of the number:
count_total=$((${l#?}))
else
count=$((count + 1))
PrintJustInTitle "Analyzing file $count of $count_total"
if [ -z "$current_line" ]; then
#FilePath / FileType / FileSize / FileTime / SecondsSinceEpoch
#Get previous FileType = if from <dir_tree1> or <dir_tree2>:
previous_line_type="${l#*"///// "}"
#extract first character from $previous_line_type ("<" or ">"):
previous_line_type_temp="${previous_line_type#?}"
previous_line_type="${previous_line_type%"$previous_line_type_temp"}"
#Remove FileType from line:
previous_line="$l"
previous_line_file_path="${l%%" /////"*}"
previous_line_last_part="${previous_line#*"///// ""$previous_line_type""_ ///// "}"
previous_line="$previous_line_file_path /////$previous_line_last_part"
previous_line_file_size="${previous_line_last_part%%" /////"*}"
previous_line_seconds_since_epoch="${l##*" "}"
previous_line_file_time="${previous_line#*"///// "}"
previous_line_file_time="${previous_line_file_time%%" /////"*}"
current_line="$previous_line"
current_line_file_path="$previous_line_file_path"
current_line_file_size="$previous_line_file_size"
current_line_seconds_since_epoch="$previous_line_seconds_since_epoch"
current_line_type="$previous_line_type"
current_line_file_time="$previous_line_file_time"
else
#FilePath / FileType / FileSize / FileTime / SecondsSinceEpoch
previous_line="$current_line"
previous_line_file_path="$current_line_file_path"
previous_line_file_size="$current_line_file_size"
previous_line_seconds_since_epoch="$current_line_seconds_since_epoch"
previous_line_type="$current_line_type"
previous_line_file_time="$current_line_file_time"
#Get current FileType = if from <dir_tree1> or <dir_tree2>:
current_line_type="${l#*"///// "}"
#extract first character from $current_line_type ("<" or ">"):
current_line_type_temp="${current_line_type#?}"
current_line_type="${current_line_type%"$current_line_type_temp"}"
#Remove FileType from line:
current_line="$l"
current_line_file_path="${l%%" /////"*}"
current_line_last_part="${current_line#*"///// ""$current_line_type""_ ///// "}"
current_line="$current_line_file_path /////$current_line_last_part"
current_line_file_size="${current_line_last_part%%" /////"*}"
current_line_seconds_since_epoch="${l##*" "}"
current_line_file_time="${current_line#*"///// "}"
current_line_file_time="${current_line_file_time%%" /////"*}"
if [ ! "$skip" = "$count" ]; then
if [ "$found_previous" = "false" ]; then
seconds_difference=$(($current_line_seconds_since_epoch - $previous_line_seconds_since_epoch))
if [ \
\( "$current_line" = "$previous_line" \) -o \
\( \
\( "$current_line_file_path" = "$previous_line_file_path" \) -a \
\( "$current_line_file_size" = "$previous_line_file_size" \) -a \
\( "$seconds_difference" = "1" -o "$seconds_difference" = "-1" \) \
\) \
]; then
found_previous="true"
found_previous_last_line_case="false"
skip=$((count+1))
else
printf '%s\n' "$previous_line_type $previous_line_file_path - ""Size: ""$previous_line_file_size"" Bytes"" - ""Modified Date: ""$previous_line_file_time"
found_previous="false"
found_previous_last_line_case="true"
fi
else
printf '%s\n' "$previous_line_type $previous_line_file_path - ""Size: ""$previous_line_file_size"" Bytes"" - ""Modified Date: ""$previous_line_file_time"
found_previous="false"
found_previous_last_line_case="true"
fi
else
found_previous="false"
found_previous_last_line_case="true"
fi
fi
fi
done
#Treat last case sepparatelly:
if [ "$found_previous_last_line_case" = "true" ]; then
printf '%s\n' "$current_line_type $current_line_file_path - ""Size: ""$current_line_file_size"" Bytes"" - ""Modified Date: ""$current_line_file_time"
fi
unset IFS
set +f
fi
CleanUp
How to use it:
Syntax: <caller_shell> '/path/to/diffm.sh' <dir_tree1> <dir_tree2>
where: <caller_shell> can be any of the shells: dash, bash, zsh, or any other shell compatible with the dash shell syntax.
What it does: Compares the files (recursively) in the two provided directory tree paths (<dir_tree1> and <dir_tree2>) by:
Notes:
This script compares only the files, not also the directories in <dir_tree1> and <dir_tree2>.
To find out how to use the script, you can call it with the --help flag.
Midnight commander (mc in Debian linux) is a two-pane command-line file manager with a Compare Directories function (C-x d; that is, holding the control key, press x, release, then press d). It has three choices: Quick, Size only, and Thorough. Quick or Size only may be helpful to you. It then highlights the files that are different in both panes, so you can do something to one or both sets.
I use the rsync method given above when I have a lot of files and a clear source and destination directory. But I often find I have two directories that each have new files to synchronize and I end up using mc frequently.
source/andtarget/are also both very important! (Without them, you will compare source and target directory names along with the child file names, so all file names will differ.) – peschü Feb 16 '19 at 14:10*deleting, but files, which are in source only did show up. The slashes are easy to accidentally forget and then you get a plausible but wrong output. – user643011 Aug 27 '19 at 19:10rsync -n -a -i --delete /Volumes/share/AdvantageProxy/Development/bin/. /Volumes/share/AdvantageProxy/Deploy-QA-2020-02-21/bin/.– Cognitiaclaeves Feb 24 '20 at 20:25