10

I have millions of files with the following nomenclature on a Linux machine:

1559704165_a1ac6f55fef555ee.jpg

The first 10 digits are timestamp and the ones followed by a _ are specific ids. I want to move all the files matching specific filename ids to a different folder.

I tried this on the directory with files

find . -maxdepth 1 -type f | ??????????_a1ac*.jpg |xargs mv -t "/home/ubuntu/ntest"

However I am getting an error indicating:

bash 1559704165_a1ac6f55fef555ee.jpg: command not found

When I tried, mv ??????????_a1ac*.jpg I am getting argument list too long error. I have atleast 15 different filename patterns. How do I move them.

terdon
  • 242,166
Apricot
  • 487
  • 1
    The bash says it all: it tries to execute that filename as it is the first on the line in the 2nd stage of the pipe (your 2nd stage pipe is: | ??????????_a1ac*.jpg: bash expands it to several filename, the first being 1559704165_a1ac6f55fef555ee.jpg, si you end up, in that 2nd pipe stage, trying to execute: 1559704165_a1ac6f55fef555ee.jpg next_matching_filename 3rd_matching_filename ... nth_matching_filename. I guess you tried instead to filter to that filename (see answers below for that) – Olivier Dulac Jul 08 '19 at 13:08

5 Answers5

16

You should use:

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' \
-exec mv -t destination "{}" +

So maxdepth 1 means that you want to search in current directory no subdirectories.

type f means find only files.

name '??????????_a1ac*.jpg' is a pattern that matches with file you are searching.

mv -t destination "{}" + means move matched files to destination. Here + adds new matched files to previous one like:

mv -t dest a b c d

Here a b c d are different files.

Prvt_Yadav
  • 5,882
  • Thanks for concisely answering this persons question. Rather than simply dumping a solution, maybe you could explain how/what/why. Instead of being useful to one person, one time, it can be useful to everyone, all the time. Same question has been asked & answered countless times over the last 40-50 years. Problem is, it's never explained well. Teach a man to fish.. In the meantime: http://www.gnu.org/software/findutils/manual/html_node/find_html/Multiple-Files.html#Multiple-Files and as is often the case Wikipedia's more useful than the official docs: https://en.wikipedia.org/wiki/Find_(Unix) – voices Jul 08 '19 at 17:22
  • See updated answer. – Prvt_Yadav Jul 08 '19 at 17:48
  • Note that -t is a GNU extension and so may not be available on other types of UNIX derivatives. – Kevin Jul 08 '19 at 18:22
  • When you say "Double quotes prevents word splitting." I presume you are referring to "{}", in which case I want to point out that {} is not expanded by the shell and does not need to be quoted. The shell passes {} to find, and find sees {} and replaces it with pathnames. Find exec does not use the shell parser and does not do any word-splitting of its own. Quoting it does not do any harm, it is just that the justification given is a bit inaccurate. – jw013 Jul 09 '19 at 18:59
  • @jw013 thanks . – Prvt_Yadav Jul 10 '19 at 06:49
11

Your command,

find . -maxdepth 1 -type f | ??????????_a1ac*.jpg |xargs mv -t "/home/ubuntu/ntest"

Pipes the list of all the files TO all the files!

find . -maxdepth 1 -type f -name `*_a1ac*.jpg` -print0 |\
xargs  -0 -r mv -t "/home/ubuntu/ntest"

will do the trick.

waltinator
  • 4,865
8

You're very close. You should use the -name option to find. And remember to quote the pattern.

So

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' |xargs mv -t "/home/ubuntu/ntest"
Stephen Harris
  • 44,540
  • Many thanks...your solution worked too....additional thanks for letting me know i was close to the solution....its a motivator for a novice like me – Apricot Jul 06 '19 at 14:12
  • 1
    you should add a -print0 as the last argument to the find (instead of the default: -print), and add a -0 as the first option to xargs (ie: xargs -0 mv -t "/home/ubuntu/ntest" ). that way, all kind of weird filenames (with spaces in it, with "newline" in it, etc) can be handled. find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' -print0 |xargs -0 mv -t "/home/ubuntu/ntest" (works only with GNU-like find, though) – Olivier Dulac Jul 08 '19 at 13:10
2

Not as "good" as the find solutions, but another valid solution is to make the mv commands more granular.

This does 4096 moves, with a fewer number of files moved per mv operation.

FILEPAT=a1ac
for i in $(seq $((0x000)) $((0xfff))); 
do 
   H=$(printf '%x\n' $i)
   mv 1559704165_${FILEPAT}${H}*.jpg /home/ubuntu/ntest
done
RonJohn
  • 1,148
-2

If you want to move files on the same host system, which I guess you are doing with your mv, rsync could be a faster option:

rsync -av --inplace -W /source/??????????_a1ac*.jpg /home/ubuntu/ntest/

--inplaceand -W are set to speed up the process.

Should this yield another argument list too long error then you could feed lists to rsync

Make the list with find, for example

find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' > /tmp/my_image_list.txt

and give it to rsync

rsync -av --inplace -W --files-from=/tmp/my_image_list.txt /path/to/files /home/ubuntu/ntest/

The source here is /path/to/files, because rsync will treat the list you give it as relative to your source.

The point being: rsync is faster than mv, if the files are not on the same filesystem.

Robert Riedl
  • 173
  • This is likely to hit the same "argument list too long" error the OP mentioned – Grump Jul 08 '19 at 13:24
  • @Grump, to avoid this, OP could write the list of files to copy to a file, i.e. find . -maxdepth 1 -type f -name '??????????_a1ac*.jpg' > /tmp/my_image_list.txt and then pass it to rsync with --files-from=/tmp/my_image_list.txt. The point being that rsync is faster. Unless the files reside on the same filesystem, which OP has not indicated. – Robert Riedl Jul 08 '19 at 13:46
  • @RobertRiedl: you should edit your answer and add this information. Comments can be impermanent. – NickD Jul 10 '19 at 04:13
  • @NickD, I've updated my answer. – Robert Riedl Jul 10 '19 at 06:31
  • It's not true that rsync is faster unless you need to do incremental copies. – forest Nov 20 '21 at 01:16
  • @forest if they are on the same filesystem, then mv is faster. If they are not, then rsync is faster (most of the time). Depends heavily on the environment. rsync can also do compression, etc - OP never stated that the files will be moved inside the same filesystem. If bandwidth is not the issue, then you can even pipe tar trough nc which will blow mv right out of the water. – Robert Riedl Nov 22 '21 at 16:08