8

This is an extension of the question I asked . Passing directory from command line to shell script

I have a script which writes another script using a heredoc. I need to be able to write unexpanded variables in the heredoc, so I use single quotes ('EOF'). However, I need one variable to be expanded. Given the script below, how can I write the value of $sourcedir inside the heredoc?

#!/bin/bash 

sourcedir="$1"
cd $sourcedir

find "$PWD" -maxdepth 2 -name \*_R1*.fastq.gz > list1

fastq_list=$sourcedir/list1 echo `cat $fastq_list` num_files=$(wc -l < 
 $sourcedir/list1) echo $num_files



cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
#$ -j y
#$ -cwd -S /bin/sh
#$ -l h_vmem=10G
#$ -pe smp 12

if [ -z "${SGE_TASK_ID}" ]; then   echo "Need to set SGE_TASK_ID"   exit 1 fi


 BASEDIR=$sourcedir

 echo "BASEDIR" echo $BASEDIR

 BASEFILES=$( ls *_R1.fastq.gz)
 BASEFILES_ARRAY=(${BASEFILES})
 BASEFILE=${BASEFILES_ARRAY[(${SGE_TASK_ID} - 1)]}
 echo $BASEFILE


 ...................
 ...................

 EOF

 qsub -t 1-${num_files}  run_array_job.sh

I am running this script using 

bash script.sh /home/dir/data

I am able to pass /home/dir/data as $1 to sourcedir but it also needs to be passed to BASEDIR , in array script which is submitted to cluster using qsub.

Ron
  • 1,057
  • As simple as BASEDIR=$1. You need to escape all $'s in your embedded script. If you don't get what I mean I can write an answer. – Weijun Zhou Mar 12 '19 at 19:05
  • @WeijunZhou the actual script is much, much longer than this (I know this from chatting with the OP), so escaping all $ just to keep the value of one variable will be complicated. – terdon Mar 12 '19 at 19:08
  • @ron it would help if you could show how you want $sourcedir in the array script. – terdon Mar 12 '19 at 19:08
  • @terdon I made the change. BASEDIR = $sourcedir is what I want, inside my array script – Ron Mar 12 '19 at 19:10
  • 1
    In this case I usually do cat > wrapperscript.sh << EOF, then put run_array_job.sh $1 in the heredoc for wrapperscript.sh. – Weijun Zhou Mar 12 '19 at 19:10
  • 1
    You can leave the EOF quoted so that you don't need to escape the $s for run_array_job.sh, but leave out the quote when you write heredoc for wrapperscript.sh – Weijun Zhou Mar 12 '19 at 19:12
  • @WeijunZhou I didnt get the above two comments.I tried BASEDIR=$1 suggestion,it did not work – Ron Mar 12 '19 at 19:17
  • That's because you used <<'EOF', if you use <<EOF it will work but you need to escape all $'s. This is also why terdon edited your question to clarify. – Weijun Zhou Mar 12 '19 at 19:18
  • How can I escape all $'s – Ron Mar 12 '19 at 19:20
  • Prefix them with backslash. – Weijun Zhou Mar 12 '19 at 19:21
  • Is there any reason this needs to be a single heredoc? It doesn't look like it from the example since it's going into a file anyway, but maybe there's something about the real version that requires it. – Michael Homer Mar 12 '19 at 19:25
  • @WeijunZhou There are too many variables, as this script is very long.Not sure if that would be the best way to do this.Also you mean escaping it only for the variables or all the $'s no matter what – Ron Mar 12 '19 at 19:25
  • I have shown an alternative way above. – Weijun Zhou Mar 12 '19 at 19:28
  • sure.can you send link to it – Ron Mar 12 '19 at 19:28
  • https://chat.stackexchange.com/rooms/90962/chat-room-for-https-unix-stackexchange-com-questions-505949-expanding-only-cert – Weijun Zhou Mar 12 '19 at 19:31
  • @WeijunZhou I am in the chat room – Ron Mar 12 '19 at 19:34

2 Answers2

4

That's somewhat hard to do, since unlike with quotes, you can't just stop and restart expansion in an here-doc. But we could post-process the here-doc with sed:

#!/bin/bash
sourcedir=/some/path
sed -e "s,%%sourcedir%%,$sourcedir,g" << 'EOF'
some commands here with $variables not expanded 
except for the special %%sourcedir%%, which is
EOF

Running that produces the output:

some commands here with $variables not expanded 
except for the special /some/path, which is

The sed command simply changes all instances of %%sourcedir%% with whatever the value of $sourcedir (as long as it doesn't contain commas; you'd need to change the separator for the s command to something else then.)

I changed the placeholder to another format for clarity's sake, but you could also leave it as $sourcedir and then use sed -e "s,\$sourcedir,$sourcedir,". (Though note that it would also match $sourcedirectory and other similar variables, but not ${sourcedir} even though that would be equivalent in the shell.)

Alternatively, use GNU envsubst on the document, if you have it (it's part of gettext):

#!/bin/bash
export sourcedir=/some/path
envsubst '$sourcedir' << 'EOF'
some commands here with $variables not expanded 
except for the special $sourcedir, which is
EOF
ilkkachu
  • 138,973
4

You can most straightforwardly do this for your usage just by breaking your heredoc in two parts:

cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
...
EOF

printf 'BASEDIR="%s"\n" "$sourcedir" >> run_array_job.sh

cat >> run_array_job.sh<<'EOF'
echo "BASEDIR" echo $BASEDIR
...
EOF

qsub -t 1-${num_files}  run_array_job.sh

This just builds the first part of the file, appends the single line that you wanted the variable available for at the end of that first part using >>, and then joins the rest of the document on the end in the same way.

You end up with the same coherent file at the end, and only write a couple of lines extra. If you have multiple variables to pass through, you can put them all in at once as well.

Michael Homer
  • 76,565