10

I'm setting up a docker container which requires a cronjob to do a backup using awscli.

I'm having a problem with the cron job being able to access the environment variables of the docker container. As I work around on startup I print all environment variables to a file printenv > /env.

When I try to use source from the cron job (I have tried both directly in crontab and in a script called by crontab) it doesn't seem to work.

I made a simplified version of my project to demonstrate the issue (including rsyslog for logging):

Dockerfile:

FROM debian:jessie

# Install aws and cron
RUN apt-get -yqq update
RUN apt-get install -yqq awscli cron rsyslog

# Create cron job
ADD crontab /etc/cron.d/hello-cron
RUN chmod 0644 /etc/cron.d/hello-cron

# Output environment variables to file
# Then start cron and watch log
CMD printenv > /env && cron && service rsyslog start && tail -F /var/log/*

crontab:

# Every 3 minutes try to source /env and run `aws s3 ls`.
*/3 * * * * root /usr/bin/env bash & source /env & aws s3 ls >> /test 2>&1

When I start the container I can see /env was created with my variables but it never gets sourced.

Philip Kirkbride
  • 10,236
  • What are you doing? Do you have a file in your root directory named source? I'm pretty sure it should just be source and not /source – jesse_b Oct 14 '17 at 13:44
  • Also do you know you are creating a file in your root directory named env and not using the actual env command? – jesse_b Oct 14 '17 at 13:47
  • @Jesse_b sorry typo fixed. – Philip Kirkbride Oct 14 '17 at 13:48
  • @Jesse_b Cron doesn't have access to my environment variables so I'm outputting them to a file on startup and sourcing them in the cronjob. – Philip Kirkbride Oct 14 '17 at 13:49
  • 1
    @Jesse_b creating? I assumed there's a file called /env that has the environment variables that need to be sourced. Is that wrong Philip? – terdon Oct 14 '17 at 13:49
  • @terdon exactly – Philip Kirkbride Oct 14 '17 at 13:49
  • he's creating it with this line: printenv > /env and if it already exists he's overwriting it. I don't see much of an issue with creating such file but the fact that it is being created in the root directory makes it seem like an oversight. – jesse_b Oct 14 '17 at 13:50
  • @Jesse_b yes, but that has nothing to do with the command env. The command is being used to find bash in the current $PATH and the file /env has the variables that need to be sourced. – terdon Oct 14 '17 at 13:52
  • @Jesse_b this is a simplification of my actual project to demonstrate the problem in the shortest possible script. My actual script is longer. Using /env saves space. – Philip Kirkbride Oct 14 '17 at 13:52
  • Understood. In that case it's still seems unnecessary, but also how am I the only person that takes issue with these files being saved in /? – jesse_b Oct 14 '17 at 13:53
  • @Jesse_b "My actual script is longer. Using /env saves space". The files could be at /some/random/path/here/env for all we know, this is just an example. – terdon Oct 14 '17 at 13:55
  • @terdon, I read that...don't see how it changes anything I said though...The way I originally read this script was as if someone was doing this: cat $file | /grep 'string' it's not unreasonable to ask if a slash in front of a well known command name is a mistake. Especially since the slash in front of source was a mistake too. – jesse_b Oct 14 '17 at 13:56
  • @Jesse_b the file isn't actually stored in / so there is no issue to be taken with files being saved in /, because no files are being saved in /. Using /env in the question saves space since it avoids needing to write out a long path. We have no idea where the files actually are and that's not really relevant to the main issue of the question anyway. – terdon Oct 14 '17 at 13:58

2 Answers2

13

First of all, the command's (well, shell builtin's) name is source. Unless you have written a script called source and put it in /, you want source and not /source.

The next issue is that cron usually uses whatever you have as /bin/sh and source is a bashism (or other such more complex shells). The portable, POSIX-compliant command for sourcing a file is .. So, try that instead of source:

*/3 * * * * root /usr/bin/env bash & . /env & aws s3 ls >> /test 2>&1

Also, I don't quite understand what that is supposed to be doing. What's the point of starting a bash session and sending it to the background? If you want to use bash to run the subsequent commands, you'd need:

*/3 * * * * root /usr/bin/env bash -c '. /env && aws s3 ls' >> /test 2>&1

I also changed the & to && since sourcing in the background is pointless as far as I can see.

terdon
  • 242,166
  • I tried your original and suggestion and it didn't work. I'm going to re-build with your update and see if that works. – Philip Kirkbride Oct 14 '17 at 13:53
  • @PhilipKirkbride if you mean the first one with root /usr/bin/env bash &, then that won't work, no. You're just sending a bash instance to the background. If the second one also fails, please explain exactly how it fails since "doesn't work" isn't very informative. – terdon Oct 14 '17 at 13:54
  • @PhilipKirkbride if && breaks, that means the source failed yes. Most likely because you're not using bash so you need to do .. But running . file & (sourcing in the background) will never work because backgrounded commands run in their own subshell, so the variables you sourced will only be available there. If you want to run the second command even if the first failed, use ;, but don't use & here. – terdon Oct 14 '17 at 14:01
  • sorry I spoke too soon and deleted my comment right after – Philip Kirkbride Oct 14 '17 at 14:02
  • 1
    @PhilipKirkbride I know, but I wanted to explain why & will always be wrong here. – terdon Oct 14 '17 at 14:03
  • The issue is actually that despite my environment variables being set, (I switched the aws command to echo $AWS_DEFAULT_REGION >> /test to confirm), awscli doesn't seem to see them. Though normally it works in the main terminal of Docker and recommends authenticating via environment variables http://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html

    The problem is out of the scope of this question, so I will mark as solved and maybe post something on their github.

     – Philip Kirkbride Oct 14 '17 at 14:04
  • 2
    @PhilipKirkbride they are set, but are they exported? That's a big difference. That you can see the variable when you echo it doesn't mean that child processes inherit it. Compare foo='bar'; bash -c 'echo $foo' and export foo='bar'; bash -c 'echo $foo' to see what I mean. – terdon Oct 14 '17 at 14:21
1

You can use . since as terdon's answer already mentioned that cron uses /bin/sh by default and doesn't recognize source command sometimes.

I wanted to add one more thing: you can't source file and use that in second command, so something like this might not work:

* * * * * source ~/.bash_profile
* * * * * do some stuff with bash_variables

Because each line is session itself so sourced variables in first command will not be available in later commands. You would have to do something like this:

* * * * * source ~/.bash_variable && do some stuff with bash_variables
chaos
  • 48,171
Vikram Nimbalkar
  • 11