Preserve directory structure when moving files using find

Question

I have created the following script that move old days files as defined from source directory to destination directory. It is working perfectly.

#!/bin/bash

echo "Enter Your Source Directory"
read soure

echo "Enter Your Destination Directory"
read destination 

echo "Enter Days"
read days



 find "$soure" -type f -mtime "-$days" -exec mv {} "$destination" \;

  echo "Files which were $days Days old moved from $soure to $destination"

This script moves files great, It also move files of source subdirectory, but it doesn't create subdirectory into destination directory. I want to implement this additional feature in it.

with example

/home/ketan     : source directory

/home/ketan/hex : source subdirectory

/home/maxi      : destination directory

When I run this script , it also move hex's files in maxi directory, but I need that same hex should be created into maxi directory and move its files in same hex there.

Answer 1

I know find was specified, but this sounds like a job for rsync.

Examples

Mirror files with same directory structure (source remains in tact):

rsync -axuv --progress Source/ Target/

Move files with same directory structure (removing from source and removing empty directories):

rsync -axuv --prune-empty-dirs --remove-source-files --progress Source/ Target/

Move files of a particular file-type (example):

rsync -rv --include '*/' --include '*.js' --exclude '*' --prune-empty-dirs Source/ Target/

Move files resulting from an advanced find search:

cd "$source" &&
  rsync -av --remove-source-files --prune-empty-dirs --progress --files-from <(find . -type f -mtime -$days) . "$destination"

A note about file removal

There are options for rsync that can ensure your destination directory mirrors your source directory, which then remove files in the destination that are no longer in your source (i.e. --delete-before, --delete-after, --delete-during, and --delete-delay.

You can also allow removing files from the source directory when they have been moved from the destination directory, i.e. --remove-source-files.

This is use case dependent, so how you handle that is up to you.

Note: As Sridar Sarnobat points out, if you rsync a a directory with symlinks to the directory you are rsync'ing to, and pass --remove-source-files`, you could end up with data loss.

Answer 2

Instead of running mv /home/ketan/hex/foo /home/maxi, you'll need to vary the target directory based on the path produced by find. This is easier if you change to the source directory first and run find .. Now you can merely prepend the destination directory to each item produced by find. You'll need to run a shell in the find … -exec command to perform the concatenation, and to create the target directory if necessary.

destination=$(cd -- "$destination" && pwd) # make it an absolute path
cd -- "$source" &&
find . -type f -mtime "-$days" -exec sh -c '
  mkdir -p "$0/${1%/*}"
  mv "$1" "$0/$1"
' "$destination" {} \;

Note that to avoid quoting issues if $destination contains special characters, you can't just substitute it inside the shell script. You can export it to the environment so that it reaches the inner shell, or you can pass it as an argument (that's what I did). You might save a bit of execution time by grouping sh calls:

destination=$(cd -- "$destination" && pwd) # make it an absolute path
cd -- "$source" &&
find . -type f -mtime "-$days" -exec sh -c '
  for x do
    mkdir -p "$0/${x%/*}"
    mv "$x" "$0/$x"
  done
' "$destination" {} +

Alternatively, in zsh, you can use the zmv function, and the . and m glob qualifiers to only match regular files in the right date range. You'll need to pass an alternate mv function that first creates the target directory if necessary.

autoload -U zmv
mkdir_mv () {
  mkdir -p -- $3:h
  mv -- $2 $3
}
zmv -Qw -p mkdir_mv $source/'**/*(.m-'$days')' '$destination/$1$2'

Answer 3

It's not as efficient, but the code is easier to read and understand, in my opinion, if you just copy the files and then delete afterwards.

find /original/file/path/* -mtime +7 -exec cp {} /new/file/path/ \;
find /original/file/path/* -mtime +7 -exec rm -rf {} \;

Notice: Flaw discovered by @MV for automated operations:

Using two separate operations is risky. If some files become older than 7 days while the copy operation is done, they won't be copied but they will be deleted by the delete operation. For something being done manually once this may not be an issue, but for automated scripts this may lead to data loss

Answer 4

you could do it using two instances of find(1)

There's always cpio(1)

(cd "$soure" && find … | cpio -pdVmu "$destination")

Check the arguments for cpio. The ones I gave

Answer 5

You can do this by appending the absolute path of the file returned by find to your destination path:

find "$soure" -type f -mtime "-$days" -print0 | xargs -0 -I {} sh -c '
    file="{}"
    destination="'"$destination"'"
    mkdir -p "$destination/${file%/*}"
    mv "$file" "$destination/$file"'

Answer 6

Better (fastest & without consuming storage space by doing copy instead of move), also is not affected by the file-names if they contain special characters in their names:

export destination
find "$soure" -type f "-$days" -print0 | xargs -0 -n 10 bash -c '
for file in "$@"; do
  echo -n "Moving $file to $destination/"`dirname "$file"`" ... "
  mkdir -p "$destination"/`dirname "$file"`
  \mv -f "$file" "$destination"/`dirname "$file"`/ || echo " fail !" && echo "done."
done'

Or faster, moving a bunch of files in the same time for multi-CPU, using the "parallel" command:

echo "Move oldest $days files from $soure to $destination in parallel (each 10 files by "`parallel --number-of-cores`" jobs):"
function move_files {
  for file in "$@"; do
    echo -n "Moving $file to $destination/"`dirname "$file"`" ... "
    mkdir -p "$destination"/`dirname "$file"`
    \mv -f "$file" "$destination"/`dirname "$file"`/ || echo " fail !" && echo "done."
  done
}
export -f move_files
export destination
find "$soure" -type f "-$days" -print0 | parallel -0 -n 10 move_files

P.S.: You have a typo, "soure" should be "source". I kept the variable name.

Answer 7

This is less elegant but easy if the number / size of files isn't too great

Zip your files together into a zip archive, and then unzip at the destination without the -j option. By default, zip will create the relative directory structure.

Answer 8

I am using it this way

cp -r source/ destination/
find destination/ -not -path "*/mypattern/*.py" -delete

Basically copy everything from source to destination and delete everythings other than the required stuff.

Answer 9

Here's what I've been using, with find, tar, and rm. Replace find arguments with arguments you would need, but retain the -type f, files only option.

cd srcdir        # OR pushd srcdir
find . -mtime +7 -type f |  while read fn; do echo Moving $fn; tar cf - "$fn" | ( cd destdir; tar xf - );  rm -f "$fn"; done
popd

(Skip the "echo Moving $fn" command to execute the job silently.) The above method may leave empty directories in the source tree. One could use

find srcdir -empty -type d -delete 

to remove empty directories.

Answer 10

Because there seem to be no really easy solution to this and I need it very often, I created this open source utility for linux (requires python): https://github.com/benapetr/smv

There are multiple ways how you could use it to achieve what you need but probably most simple would be something like this:

 # -vf = verbose + force (doesn't stop on errors)
smv -vf `find some_folder -type f -mtime "-$days"` target_folder

You can additionally run it in dry mode so that it doesn't do anything but print what it would do

smv -rvf `find some_folder -type f -mtime "-$days"` target_folder

Or in case that list of files is too long to fit into argument line and you don't mind executing python for every single file, then

find "$soure" -type f -mtime "-$days" -exec smv {} "$destination" \;

Answer 11

Try this way:

IFS=$'\n'
for f in `find "$soure" -type f -mtime "-$days"`;
do
  mkdir -p "$destination"/`dirname $f`;
  mv $f "$destination"/`dirname $f`;
done

Answer 12

#!/bin/bash

# '+' here shows that 'find' looking for files 45 and more days older from NOW
# can be replaced with '-' (-45), to search for files with modification date from NOW to 45 days(max)
days='+45'

source=/var/log

destination=/root/logsbackups

# searching can take some time, so inform user/script what is happening
echo "\nSearching for files..."

# collecting files list as Array to variable
LIST_OF_FILES=(`find $source -type f -mtime $days`)

echo "Moving files..."

# process collected file paths Array for proper moving
for file in ${LIST_OF_FILES[@]}; do

  # real full path to file (without filename)
  filepath=$(dirname $file)

  # full file name (without path)
  filename=$(basename $file)

  # log info (unnecessary)
  echo "Moving to $destination$filepath$filename"

  # make sure that target directory exists
  mkdir -p $destination$filepath

  # moving file
  mv -f $file $destination$filepath$filename

done