How to use file in the loop as a input/output file in awk?

Question

This is wrong:

for f in a*.dat; do 
    awk '.....' file1 "$f" > temp
    awk '.....' temp > "$f_out"
done

I would like to use a*.dat as input and then as output with suffix _out. Many thanks

Answer 1

The _ is seen as a part of the variable name, so you are using the variable f_out in stead of f and append out. Using {} will solve your problem.

for f in a*.dat; do 
    awk '.....' "$f" > "${f}_out"
done

Answer 2

1. You need to use curly-braces to disambiguate the variable $f from the remainder of the string (_out).

   i.e. $f_out refers to the variable named f_out (which probably doesn't exist, so evaluates to an empty string), while ${f}_out means the value of variable f with _out appended.

2. Depending on what your awk scripts are doing, you probably, almost certainly, don't need to run awk twice - it's almost never useful to pipe the output of an awk script into another awk script, and this:

   awk '.....' file1 "$f" > temp
   awk '.....' temp > "${f}_out"
   

   is effectively the same as this:

   awk '.....' file1 "$f" | awk '.....' > "${f}_out"
   

   and is better written as just:

   awk '{....}; {.....}' "$f" > "${f}_out"
   

   In a loop:

   for f in *.dat; do
     awk '{....}; {.....}' "$f" > "${f}_out"
   done
   

---

BTW, awk has a built-in variable called "FILENAME" which is equal to the filename of the file currently being processed. Which means that you probably don't need the for loop, and can just do something like:

   awk '{....}; { .... }; { print > FILENAME "_out" }' a*.dat

Or just make sure that every print statement ends with > FILENAME "_out".

Without knowing exactly what the two awk scripts do, it's impossible to be more specific than that.

Answer 3

You command should be something like:

for f in a*.dat; do 
    awk '.....' "$f" > "${f}_out"
done

The output need to be on this way otherwise shell will try to interpret variable

f_out