here-document basename conflicts?

Question

I am using EOF to generate bash scripts that run the Rscripts. In the Rscript I used basename to specify the output file name.

When I use EOF to generate a list of bash scripts, I could not get basename to work. The error message is shown below. I was still able to get the bash scripts generated but the ${AF} turned into a blank in both places where it presented. Very strange!

I had the bash script tested and it is working so I know the problem is somewhere between EOF and basename.

How can I use basename with EOF? Or is there any alternative methods? Thank you.

for letter in {A..Z}
      do cat <<- EOF > batch_${letter}.sh
    #!/bin/bash
    module load R/3.5.1
    R_func="/home/dir/R_func"
    TREAT="/home/dir/POP"
    BASE="/home/dir/base"
    OUTPUT="/home/dir/tmp"

    for AF in ${BASE}/${letter}*.txt_step3; do
    Rscript ${R_func}P_tools.R \
    --ptool ${R_func}/P_tools_linux \
    --group ${AF} \
    --treat ${TREAT}/pop_exclude24dup \
    --out ${OUTPUT}/OUT_$(basename ${AF%%_txt_step3})_noregress \
    --binary-target F; done

    EOF
       done

This is the error message

basename: missing operand Try 'basename --help' for more information.

Answer 1

Command substitutions like your $(basename ...) and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $ from $(basename ...) and also any $ inside it.

Corrected version of your script:

for letter in {A..Z}
        do cat <<- EOF > batch_${letter}.sh
                #!/bin/bash
                module load R/3.5.1
                R_func="/home/dir/R_func"
                TREAT="/home/dir/POP"
                BASE="/home/dir/base"
                OUTPUT="/home/dir/tmp"

                for letter in {A..Z} do {
                for AF in \${BASE}/${letter}*.txt_step3; do
                Rscript \${R_func}P_tools.R \
                --ptool \${R_func}/P_tools_linux \
                --group \${AF} \
                --treat \${TREAT}/pop_exclude24dup \
                --out \${OUTPUT}/OUT_\$(basename \${AF%%_txt_step3})_noregress \
                --binary-target F; done
                }
        EOF
done

That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<- which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.

Answer 2

The << EOF...EOF construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF is common.

The issue you're facing is that the here-document acts like a double-quoted string, so the variables and the command substitution in it are expanded when cat runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func in the batch_x.sh you're writing, but ${R_func} would expand to whatever value R_func has in the generating script.

You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF' instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.

---

Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh) then looks something like this:

module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in ${BASE}/A*.txt_step3; do
   Rscript ${R_func}P_tools.R \
   ...
done

Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1":

#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1

for AF in ${BASE}/${letter}*.txt_step3; do
    Rscript "${R_func}P_tools.R" \
    --ptool "${R_func}/P_tools_linux" \
    --group "${AF}" \
    --treat "${TREAT}/pop_exclude24dup" \
    --out "${OUTPUT}/OUT_$(basename "${AF%%_txt_step3}")_noregress" \
    --binary-target F;
done

The variable letter would now be taken from the command line, so you could run batch.sh A to process the A files, etc.