1

I have a list of files in /tmp/drop directory. I have taken the essential element of my question into test script zz.sh. I have taken a part of the script and put it here. These are the files and the lines listed - and when it given to while loop - how does the variable file gets assigned only the file name and not the various other strings in those lines?

Any help would be appreciated and would be helpful to others too.

machine001:/home/aaa999999> ls -l /tmp/drop | grep "2017-01-29" | egrep  '\.gz$' | tail -10
+ ls -F -l /tmp/drop
+ tail -10
+ grep 2017-01-29
+ egrep \.gz$
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-21:33:13.888.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-21:48:14.632.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:03:19.098.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:18:19.416.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:33:19.878.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-22:48:25.636.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:03:26.515.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:18:28.279.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:33:33.059.complete.gz
-rw-r--r--   1 user1001      user1001           20 Jan 29  2017 pattern101_worker-2_2017-01-29-23:48:33.841.complete.gz
machine001:/home/aaa999999> cat zz.sh
+ cat zz.sh
#!/bin/bash
set -x

DROP_DIR=/tmp/drop
ARCHIVE_DIR=/tmp/arch
YESTERDAY="2017-01-29"
# move from drop dir
ls -1 ${DROP_DIR} | grep ${YESTERDAY} | egrep '\.gz$' | while read file; do
    mv  ${DROP_DIR}/$file ${ARCHIVE_DIR}
        echo 'File name is ' ${file}
done
Yaron
  • 4,289
Venkat M
  • 11
  • What do you want to do? Looks to me like mv /tmp/drop/*2017-01-29*.gz /tmp/arch? – pfnuesel Jan 31 '18 at 00:53
  • 4
    In the first case, you have ls -l (letter ell) while in the script it appears to be ls -1 (digit one) - different options, different behaviors. Regardless, please see Why not parse ls? – steeldriver Jan 31 '18 at 01:01
  • @steeldriver Good catch! – G-Man Says 'Reinstate Monica' Jan 31 '18 at 04:04
  • 2
    for file in "$DROP_DIR/*$YESTERDAY*.gz" ; do ... ; done - parsing ls is not only potentially dangerous, it isn't even necessary. Also, that while read loop is in a pipe subshell, so can't affect the environment or variables of the parent script (see https://unix.stackexchange.com/questions/9954/why-is-my-variable-local-in-one-while-read-loop-but-not-in-another-seemingly for why). – cas Jan 31 '18 at 04:58
  • @cas drop those double quotes surrounding the wildcard globs – Chris Davies Jan 31 '18 at 19:18
  • @steeldriver The 1 versus l was the difference that I did not notice earlier. Thanks. – Venkat M Jan 31 '18 at 20:49

3 Answers3

2

ls -1 prints just the filenames. ls -l prints a bunch of other stuff, too. The other is the number one, the other a lowercase letter el.

Though the -1 is unnecessary here, since if the output is redirected to a pipe, the "one-column" mode is implied.

But really, you don't need the ls and the pipe here, at all. The loop can be replaced with just:

for file in "$DROP_DIR/"*"$YESTERDAY"*.gz ; do 
    mv "$file" "$ARCHIVE_DIR"
    echo "File name is ${file#$DROP_DIR/}"
done

See also: ParsingLs on Greg's Bash wiki.

ilkkachu
  • 138,973
0

Suggestion using GNU date:

yesterday=$( date -d yesterday +%F )
drop_dir=/tmp/drop
archive_dir=/tmp/arch

mkdir -p "$archive_dir"

mv -f "$drop_dir"/*"$yesterday"*.gz "$archive_dir"

Here, "$drop_dir"/*"$yesterday"*.gz will expand to all files matching the given pattern. This will work as long as the pattern expand to several thousands of characters, in which case the shell will complain with "Argument list too long".

Or, if you really want to do looping to output each filename as they are moved (only the mv command from above is changed to the following):

for name in "$drop_dir"/*"$yesterday"*.gz; do
    [ -f "$name" ] || continue
    mv "$name" "$archive_dir"
    printf 'Moved "%s"\n' "$name"
done

Here, the for loop will iterate over the same names as was used in the previous variation, but one by one. Each name will be held in $name (which includes the directory name). If there were no files matching the pattern, the pattern will remain unexpanded. This is why we do an explicit test to make sure that $name really corresponds to an existing file before moving it.

Or, with find:

find "$drop_dir" -maxdepth 1 -type f -name "*$yesterday*.gz" \
    -print -exec mv {} "$archive_dir" ';'

Here, find will pick up the names from $drop_dir that corresponds to regular files (-type f) and that matches the pattern given to -name. For each such name, mv is called to move the file into tho destination folder. Before calling mv though, -print will list the filename (including path) on the terminal.

The -maxdepth 1 option stops find from recursing into subfolders of $drop_dir.

Kusalananda
  • 333,661
-1

Your question isn't too clear to me, but I think what you seem to be looking for is the -o option to your egrep statement:

 egrep -o "[^[:space:]]+\.gz$"

The -o option (--only-matching) causes egrep to only output the matching portion of matching lines. The [^[:space:]] matches anything not a space character.

user1404316
  • 3,078