5

I have about 1 million files in this directory: /home/username/images/

Each of the files are called something like: 012345678910(Place)_0_20120414185957_28841.jpg with the timestamp part of the filename changing on each picture.

The code below contains code to sort/move the files into this date structure: /home/username/sorted/2012/04/14/18/name_of_file.jpg

For a small sample of files it works fine, but for the huge directory my putty terminal gets disconnected after outputting

Directory $newdir does not exist.  Creating same.

I had other code which always died with the error code argument list too long.

Here is the code:

#!/bin/bash
ALLFILES=(images/*)
for ((i=0; i<${#ALLFILES[*]}; i+=30000));
do
    set $(echo "${ALLFILES[@]:i:30000}" | awk -F_ '{print $1, $2, $3, $4, $5}')
    fullyear=$3
    year=$(echo $fullyear |cut -c1-4)
    month=$(echo $fullyear |cut -c5-6)
    day=$(echo $fullyear |cut -c7-8)
    hour=$(echo $fullyear |cut -c9-10)
    newdir=$(echo /home/username/sorted/$year/$month/$day/$hour/)
    if ! [ -d $newdir ]; then
        echo Directory $newdir does not exist.  Creating same.
        mkdir -p $newdir;
    fi
    mv "${ALLFILES[@]:i:30000}" $newdir;
done

Any ideas why the connection will not hold while performing the large loop?

Gilles 'SO- stop being evil'
  • 829,060
PadraigD
  • 153
  • Have you tried turning on xtrace (set -x at the beginning or invoke bash with -x) for debugging? – jw013 Apr 17 '12 at 10:52

4 Answers4

5

Try to run it in screen session. Or even try another construction. I believe find + sed will work better then pure bash:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%'

This is just to show, how sed make commands to execute. Adding e after last % will force command executing:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%e'

ps. You don't need to use in bash 

day=$(echo $fullyear |cut -c7-8)

Bash can do it itself without echo | cut :

day=${fullyear:6:2}
rush
  • 27,403
  • I never knew about the sed command. The snippet you gave works perfectly ... well it's been working away for the last few mins generating the folders and moving the files in correct order. This was my first time ever creating a *nix script, I'm coming from Windows. The server I'm connecting to is a web-hosting provider, not mine so I can only use Putty to connect. – PadraigD Apr 17 '12 at 13:13
  • I also didn't know that Bash can do the segmenting of the variables. That's good to know. Thanks very much for your help Rush. – PadraigD Apr 17 '12 at 13:19
  • This will break horribly on special characters in filenames and is also a security hole as it allows malicious filenames to cause arbitrary code execution. See Why is looping over find's output bad practice? – Wildcard Nov 14 '16 at 22:22
1

I'm using this shell script in the root of a directory packed with files to move them all to a year/month-like structure:

#!/usr/bin/env bash

if [ ! $1 ]; then
    echo "Usage: ./pictures.sh jpg"
    exit 1
fi

for f in *."$1"; do
    FILENAME="$f"
    YEAR=`date -j -f "%s" $(stat -f "%m" "$FILENAME") +"%Y"`
    MONTH=`date -j -f "%s" $(stat -f "%m" "$FILENAME") +"%m_%B"`
    DEST="$YEAR/$MONTH"

    if [ ! -d "$DEST" ]; then
        mkdir -p "$DEST"
    fi

    echo "Moving $FILENAME to $DEST/$FILENAME ..."
    mv "$FILENAME" "$DEST/$FILENAME"
done

Usage: $ ./pictures.sh JPG to move *.JPG to the correct structure.

Giel Berkers
  • 111
0

I'm also sorting images into date-structured directories, but I have a slightly different approach. I want my images to go to their respective YYYY-MM directories, based on their timestamps. So what I do is that I start by ls -l *.jpg > tmp.txt the image folder, then I feed this tmp.txt into a loop to get the timestamp for each file. I haven't found a way to get the timestamp otherwise.

Here is my code:

#!/bin/bash
hostdir="/home/Photos/"
destdir="/tmp/sorted"

cd $hostdir

touch /tmp/tmpsort.txt
ls -l *.jpg > /tmp/tmpsort.txt

while read line
do
    filename=$(echo $line | awk '{print $8}')
    filedate=$(echo $line | awk '{print $6}')
    filedir=${filedate:0:7}

    if [ ! -d $destdir/$filedir ]; then
        mkdir -p $destdir/$filedir
    fi

    # Let's skip files that were already sorted from a previous run
    if [ ! -f $destdir/$filedir/$fiename ]; then
        cp $filename $destdir/$filedir/
    fi

done < /tmp/tmpsort.txt
rm /tmp/tmpsort.txt

I don't have millions of images to sort and if I did, this code would take quite some time to execute. But it works as intended.

Raphael Ahrens
  • 9,771
user73209
  • 1
0

The following one-liner will create a shell script to move the files to the correct folder based on modified time.

find . -type f -not -name ".DS*" -exec stat -f "mkdir -p %Sm; mv \"%N\" %Sm" -t "%Y/%m/%d" {} \; > move.sh
sh move.sh

I have excluded .DS* files (-not -name ".DS*") Before execute the move.sh, you can edit it to remove the unwanted files.

Divya
  • 1