0

I have this part of shell script :

#!/bin/bash
shopt -s extglob
currentDate=$(date +%F)

echo $currentDate
command="grep $currentDate"
gcs3='s3://gc-reporting-pud-production/splunk_printer_log_files'
gcs3ls='aws s3 ls 's3://gc-reporting-pud-production/splunk_printer_log_files/SOUTH_ASIA/' --recursive '
ssyss3=s3://ssyssplunk

gcs3Current=$($gcs3ls|$command|sed 's/^.*\(splunk_printer.*\)/\1/g')

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
s3ls='aws s3 ls --human-readable --summarize 's3://ssyssplunk/' --recursive'
echo "ls: " $s3ls
egrepCommand="'$currentDate|Total'"
echo "grep: " $egrepCommand
totalSize=$($s3ls|egrep $currentDate\|Total|awk -F 'Total Size:' '{print $2}'|sed '/^$/d')
echo "total size: "  $totalSize
IFS=$SAVEIFS

and im getting this error :

2019-05-27 ls: aws s3 ls --human-readable --summarize s3://ssyssplunk/ --recursive grep: '2019-05-27|Total' ./copyFilesFromS13.sh: line 54: aws s3 ls --human-readable --summarize s3://ssyssplunk/ --recursive: No such file or directory total size:

what am i doing wrong ?

ctrl-alt-delor
  • 27,993
sarit
  • 11
  • 1
    it is not answering my question – sarit May 27 '19 at 08:09
  • I see several errors, including anti-pattern “use of ls in a script”, this has lead to treating total size as a file-name. – ctrl-alt-delor May 27 '19 at 09:08
  • so what is the best way to do it ? – sarit May 27 '19 at 11:19
  • It can't find the command "aws" - can you run the command outside the script from the same location – jai_s May 27 '19 at 13:37
  • i did and it is working. @muru suggested that it is because of the IFS that im using to ignore spaces so i changed the order and pasted the commands after IFS=$SAVEIFS so it now looks like : IFS=$SAVEIFS IFS=$(echo -en "\n\b") s3ls='aws s3 ls --human-readable --summarize 's3://ssyssplunk/' --recursive' echo "ls: " $s3ls egrepCommand="'$currentDate|Total'" echo "grep: " $egrepCommand totalSize=$($s3ls|egrep $currentDate|Total|awk -F 'Total Size:' '{print $2}'|sed '/^$/d') echo "total size: " $totalSize

    so now there is no error but it is stuck on "totalSize"

     – sarit May 27 '19 at 14:00

1 Answers1

0

You have set IFS to just the newline and backspace. So, $s3ls, after expansion and word splitting, will be aws s3 ls --human-readable --summarize s3://ssyssplunk/ --recursive taken as a single word. Bash tries to execute this single word as a command, instead of executing aws with a bunch of arguments.

You really shouldn't be storing commands in variables. Use arrays, instead:

s3ls=(aws s3 ls --human-readable --summarize 's3://ssyssplunk/' --recursive)
#...
totalSize=$("${s3ls[@]}" | ...)
muru
  • 72,889
  • well, i moved it after this line : IFS=$SAVEIFS there is no error now but nothing happens... it looks like it stuck – sarit May 27 '19 at 08:08