1

I have a shell script to download files from S3. I'm able to run the script manually and it's working fine, But when I schedule it from Cron it's triggered but except the echo command, nothing is executed.

Script

#!/usr/bin/env bash

set -e
echo "starting"
y_day=$(date --date="-1 day" +%d)
y_month=$(date --date="-1 day" +%m)
y_year=$(date --date="-1 day" +%Y)

files=$( aws s3 ls  s3://bucket/log/$y_year/$y_month/$y_day/ | grep _mylog_ | awk -F ' ' '{print $4}')

for file in $files
do
aws s3 cp s3://bucket/log/$y_year/$y_month/$y_day/$file /opt/local_log/
done

## Next files
count=$( aws s3 ls  s3://bucket/testlog/$y_year/$y_month/$y_day/ | grep _testlog_ | wc -l)

if [ $count -gt 0 ]
then
files=$( aws s3 ls  s3://bucket/testlog/$y_year/$y_month/$y_day/ | grep _testlog_ | awk -F ' ' '{print $4}')

for file in $files
do
aws s3 cp s3://bucket/testlog/$y_year/$y_month/$y_day/$file /opt/localtestlog/
done
else
echo "No files for userlog"
fi

Cronjob

10 13 * * * /opt/script/copy.sh > /tmp/copy.log

log file (/tmp/copy.log)

starting
No files for userlog

s3://bucket/testlog/$y_year/$y_month/$y_day/ No files in this location, so the the log correct. But I first set of command, didn't download anything.

Ubuntu syslog

Jun  5 19:04:01 ip-10-23-53-161 CRON[6984]: (root) CMD (/opt/script/copy.sh > /tmp/copy.log
)
Jun  5 19:04:01 ip-10-23-53-161 cron[6952]: sendmail: fatal: open /etc/postfix/main.cf: No such file or directory
Jun  5 19:04:01 ip-10-23-53-161 postfix/sendmail[7012]: fatal: open /etc/postfix/main.cf: No such file or directory
Jun  5 19:04:01 ip-10-23-53-161 CRON[6979]: (root) MAIL (mailed 1490 bytes of output but got status 0x004b from MTA#012)

I don't have mail server, so the last 3 lines are fine. But my question is, the complete command ran, except the echo, nothing processed their work. But this count command ran and the value is 0, so it did echo No files for userlog.

terdon
  • 242,166
TheDataGuy
  • 365
  • Also redirect the error stream to your log with >logfile 2>&1, or redirect the error log separately with >logfile 2>errorlog. It's the error output that cron is likely trying to mail to you. Then investigate the error log and update your question. – Kusalananda Jun 05 '20 at 13:58
  • where is the command aws? My guess is that cron is running as root and can't find one or more commands in the script – user1794469 Jun 05 '20 at 13:59
  • aws s3 cp is the command – TheDataGuy Jun 05 '20 at 14:01
  • thanks @Kusalananda and @user1794469 I redirect the error output to a file as Kusalananda suggested and I found aws command not found error as user1794469 mentioned. – TheDataGuy Jun 05 '20 at 14:03

1 Answers1

1

The primary issue seems to be that the aws command is not found.

You could solve this in one of two ways:

  1. Use the full path of the command each time you invoke it in the script.
  2. Modify PATH in the script to include the path of the directory where the command is found.

See also:

Kusalananda
  • 333,661
  • I didn't understand this double quote? Are you referring my echo command? – TheDataGuy Jun 05 '20 at 14:08
  • @Bhuvanesh it's general good practice. It is probably not causing issues for you here, but it can cause problems generally and they can be quite hard to debug. – terdon Jun 05 '20 at 14:10
  • Im sorry, still im sure, are you saying instead of echo "starting" we can use echo 'starting' Is my understanding correct? – TheDataGuy Jun 05 '20 at 14:11
  • @Bhuvanesh I just wanted to make you aware when it's proper to double quote expansions, as in if [ "$count" -gt 0 ] or using "s3://bucket/testlog/$y_year/$y_month/$y_day/". It may not give you issues in this script, but it would give you issues if a variable contains spaces or filename globbing characters. – Kusalananda Jun 05 '20 at 14:13
  • @Bhuvanesh Compare, for example, echo $string with echo "$string" if you have string='*' – Kusalananda Jun 05 '20 at 14:14