How to move a file recursively based on size and keep structure

Question

I want to move files bigger than "300Mb" from one directory tree where each file is located in subfolders

Example: I have a directory structure:

dirA/
dirA/file1
dirA/x/
dirA/x/file2
dirA/y/
dirA/y/file3

Here is the expected result, a "move" of the directory tree where each file is a moved to the sub folders:

dirB/            #  normal directory
dirB/file1       #  moved from dirA/file1
dirB/x/          #  normal directory
dirB/x/file2     #  moved from dirA/x/file2
dirB/y/          #  normal directory
dirB/y/file3     #  moved from dirA/y/file3

The find /path/ -type f -size +300m but then what? and unfortunately some of the files have all sorts of characters you can find on your keyboard.

I have been looking at this thread where someone is talking about cpio but I don't know that program...

PS: have GNU Parallel installed if this could speed up things?

Answer 1

The easy way is with zsh. You can use glob qualifiers to match files according to criteria such as their type and size. The wildcard pattern **/ matches any level of subdirectories. The history modifiers h and t are easy ways of extracting the directory and the base part of a filename. Call mkdir -p to create the directories when needed.

cd dirA
for x in **/*(.Lm+300); do
  mkdir -p ../dirB/$x:h &&
    mv -- $x ../dirB/$x
done

The portable way is with find. Use -exec to invoke a shell snippet for every file.

cd dirA
find . -type f -size +300000k -exec sh -c 'for x do
  mkdir -p "../dirB/${x%/*}"
  mv "$x" "../dirB/$x"
done' sh {} +

Parallelization is rarely useful for input/output: it lets you take advantage of multiple CPUs but the CPU is rarely a bottleneck in I/O.

Answer 2

Perl rename is the obvious choice. It may be installed as ren, rename, or pren:

find dirA -type f -size +300M | ren 's:^dirA/:dirB/:'

It does, however, not work if the files are moved to a different mount point, and will fail if the dirs are not there.

GNU Parallel will be slower:

cd dirA
find . -type f -size +300M | parallel mkdir -p ../dirB/{//}
find . -type f -size +300M | parallel mv {} ../dirB/{}

but will work even if it needs to do the copy-then-remove routine to get the files onto a different file system.

Answer 3

In short:

find dirA -type f -size +300m -printf "mv %p dirB/%P\n" | sh

But, all the subdirectories in dirB must be exists before you start. For this reason I suggest you to do the following two steps:

cd dirA
find . -type f -size +300m -printf "mkdir -p ../dirB/%h\nmv %p ../dirB/%P\n" | sh

Regarding cpio (actually it resolves the subdirectories problem):

(cd dirA; find . -type f -size +300m) | cpio -p -md  dirB

(Regarding the cp(1) in same thread you mention, it is not good for you, because it will copy all the files and build subdirectory named dirA under dirB. The flag -T can resolve this problem)

Answer 4

This ought to cover it.

find /path -type f -size +300m | while read A ; do DEST=${A/dirA/dirB} ; echo mkdir -p $(dirname $DEST) 2>/dev/null; echo mv $A $DEST ; done

Run it as-is first, sanity check, and if happy with the proposed commands, rerun it without the echo elements.

In your file structure example, would generate the following commands

mkdir -p ./dirB
mv ./dirA/file1 ./dirB/file1
mkdir -p ./dirB/x
mv ./dirA/x/file2 ./dirB/x/file2
mkdir -p ./dirB/y
mv ./dirA/y/file3 ./dirB/y/file3