Batch copy to multiple directories

Question

I have about 9000 files in a directory and I want to mv them into 90 files in 100 directories by filename order, ignoring any remainder. In Copy multiple files from filename X to filename Y?, Caleb wrote in a comment:

If the object is just to batch them, there is a MUCH easier way! ls | xargs -iX -n20 cp X target_folder/ would run cp on files in batches of 20 until they were all done.

So based off of using xargs, how could I switch the target_folder to create a new folder and loop the command 100 times?

U&L is a wiki, not a message board: please rewrite your question as a question, removing the reference to a community member and including a link to the post your reference. See the FAQ for help on asking questions — jasonwryan, Jun 07 '12 at 22:46

score 2 · Answer 1 · answered Oct 07 '12 at 02:12

In bash, try the following code :

#!/bin/bash

c=0

for f; do
    if ! ((c % 100)); then
        folder=folder_$(printf "%03d\n" $c)
        mkdir -p $folder
    fi

    [[ -d "$f" ]] || mv "$f" "$folder"
    ((c++))
done

Run the script like that :

./script.bash *

Gilles 'SO- stop being evil' · Answer 2 · 2012-08-07T21:22:53.457

1

That command doesn't look close to what you're trying to do. xargs can help, but it's cumbersome to use unless you know your file names do not contain any whitespace or quoting character. Here's a shell loop that dispatches files into newly-created subdirectories, 90 per directory.

set -- *
# Set args to "$1" "$2" ... "$90"
i=1 args=
while [ $i -le 90 ]; do
  slice="$slice \"\${$i}\""
  i=$((i+1))
done
# Move files 90 at a time
i=0
while [ $# -ge 90 ]; do
  mkdir part$i
  eval "mv $slice part$i"
  shift 90
done
# 0 to 89 files remain in the current directory

edited Aug 07 '12 at 21:22

answered Jun 07 '12 at 23:53

Gilles 'SO- stop being evil'

829,060

xargs also has an option -0 which turns off the whitespace and quote processing. But I agree this kind of script is the best approach. – James Youngman Jul 07 '12 at 17:12
@Gilles There's an error in the script: $slice should be $args – daniel kullmann Aug 07 '12 at 07:26

score 0 · Answer 3 · answered Jun 16 '14 at 04:55

parallel supports running multiple commands with \; and {#} is the sequence number:

ls|parallel -n90 mkdir {#}\;mv {} {#}

Or if the filenames don't contain spaces, quotes, or backslashes:

i=1;while read l;do mkdir $i;mv $l $((i++));done< <(ls|xargs -n90)

Batch copy to multiple directories

3 Answers3