Shell programming, avoiding tempfiles

Question

I often write KSH shell scripts that follow the same pattern:

• (1) retrieve output from one or more command
• (2) format it using grep|cut|awk|sed and print it to the screen or to a file

In order to do that, I often store the output of (1) in a tempfile, and then do the formatting in (2) on that file.

Take that code for instance:

TMPFILE=file.tmp

# If tmpfile exists rm it.
[ -f $TMPFILE ] && rm -f $TMPFILE

for SERVICE in $(myfunc); do
    getInfo $SERVICE > $TMPFILE # Store raw output in the TMPFILE

    # I retrieve the relevant data from the TMPFILE
    SERV_NAME=$(head -1 $TMPFILE | sed -e 's/ $//')
    SERV_HOSTNAME=$(grep HOSTNAME $TMPFILE | cut -d "=" -f2)
    SERV_ARGS=$(grep Arguments $TMPFILE | cut -d ":" -f2)

    print $SERV_NAME $SEP $SERV_HOSTNAME $SEP $SERV_ARGS
    rm -f $TMPFILE #rm the TMPFILE in vue of next iteration
done

Is there a way, using pipes, redirections and whatnots, to avoid writing a file to disk each time?

If it helps, I'm using ksh Version M-11/16/88i

Answer 1

Your code looks like an entirely justified example of using tempfiles to me. I'd stay: stick with this approach. The only thing that really needs to be changed is the way you create the tempfile. Use something like

 TMP=$(tempfile)

or

 TMP=$(mktemp)

or at least

 TMP=/tmp/myscript_$$

This way you won't let the name be easily predicted (security) and out rule interference between several instances of the script running at the same time.

Answer 2

You could use a variable:

info="$(getInfo $SERVICE)"
SERV_NAME="$(head -1 $TMPFILE <<<"$info" | sed -e 's/ $//')"
...

From man ksh:

<<<word       A  short  form of here document in which word becomes the
              contents of the here-document after any parameter  expan-
              sion,  command  substitution, and arithmetic substitution
              occur.

Advantages include:

• Enables parallel execution.
• In my experience this is tons faster than temporary files. Unless you have so much data that you end up swapping it should be orders of magnitude faster (only barring HD caching buffers, which might be about as quick for small data amounts).
• Other processes or users can't mess up your data.

Answer 3

You have two options:

1. You retrieve the data once (in your example with getInfo) and store it in a file as you do.

2. You fetch the data on each time and do not store it locally, i.e., you call getInfo every time

I do not see the problem in creating a temporary file to avoid reprocessing/re-fetching.

If you are worried about leaving the temporary file somewhere you can always use trap to be sure to delete it in case the script is killed/interrupted

trap "rm -f $TMPFILE" EXIT HUP INT QUIT TERM

and use mktemp to create a unique filename for your temporary file.

Answer 4

Instead of generating a file, construct shell assignment statements and evaluate that output.

for SERVICE in $(myfunc); do
    eval $(getInfo $SERVICE |
               sed -n -e '1/\(.*\) *$/SERV_NAME="\1"/p' \
                   -e '/HOSTNAME/s/^[^=]*=\([^=]*\).*/SERV_HOSTNAME="\1"/p' \
                   -e '/Arguments/^[^:]*:\([^:]*\).*/SERV_ARGS="\1"/p')
    print $SERV_NAME $SEP $SERV_HOSTNAME $SED $SERV_ARGS
done

Or if you just want to print the information:

for SERVICE in $(myfunc); do
    getInfo $SERVICE | awk -vsep="$SEP" '
        BEGIN{OFS=sep}
        NR == 1 { sub(/ *$/,""); SERV_NAME=$0 }
        /HOSTNAME/ { split($0, HOST, /=/; SERV_HOSTNAME=HOST[2]; }
        /Arguments/ { split($0, ARGS, /:/; SERV_ARGS }
        END { print SERV_NAME, SERV_HOSTNAME, SERV_ARGS }'
done