`echo "${var}"` outputs leading spaces

Question

could someone help me? I am trying to create a lot of files, but when I try to write to it with echo it adds a space and single quotes, for example:

' (here is the space)  file.tmp'
'  file1.tmp'
'  file3.tmp' 

the code is this.

cat limpio.tmp | while read line; do
  srcip=$(echo $line | awk -F"," '{print $1}')
  destip=$(echo $line | awk -F"," '{print $2}')
  port=$(echo $line | awk -F"," '{print $NF}')
  echo "${srcip}" >> "$destip.tmp"
done

The problem comes when I try to iterate and view its content with cat and I can't because of the spaces.

thank you very much

Answer 1

The echo utility inserts spaces, but only as delimiters between arguments when outputting them:

$ echo 1    2 3       4 'A B     C'
1 2 3 4 A B     C

The spaces between B and C are preserved as these are part of the same argument to echo.

So it isn't that echo is inserting spaces, but rather that your data already contains spaces.

I'm assuming that the single quotes that you see are part of the output from ls on your system (see Why is 'ls' suddenly wrapping items with spaces in single quotes?), or possibly part of the shell's tracing output while it's running with set -x active (see Why does bash add single quotes to unquoted failed pathname expansions in a command before executing it?). That is, it's not actually part of the filenames but only some tool's way of rendering the names.

Assume, for example, that your input looks like

174.18.418.83, 10.404.401.94, 53144
174.18.418.83, 10.404.401.94, 53144
174.18.418.83, 10.404.401.94, 53144

You will notice that there are spaces after the comma delimiters. If you read the 2nd field from the above file using the comma as the delimiter, you will therefore retain the space(s) at the start of the field, and if you use that data as a filename, the filename will start with space.

(The observant reader will notice that the code in the question uses the variable line unquoted, this means that the shell would split the variable's value on the spaces and present echo with separate arguments like 174.18.418.83,, 10.404.401.94,, and 53144. So it is true, in a sense, that it is echo that inserts the spaces that needs to be removed, but it would not do that had there been no spaces/tabs in the data to start with. See also When is double-quoting necessary?)

The solution is to use both the comma and any subsequent blank character as the delimiter with awk:

awk -F ',[[:blank:]]*' '{ print $2 }'

The expression ,[[:blank:]]* matches comma followed by any number of blank characters. A "blank" character is a space or a tab. You may change this to , * ("comma, space, asterisk") if you know you have literal spaces.

Now to avoid calling awk three times for each line in the input file.

The print statement in awk can be "redirected" in pretty much the same way as you may redirect output in the shell, using > followed by a filename.

The filename you seem to want to use is whatever is in the 2nd field, with .tmp appended.

This mean you can replace the whole loop in your question with the following single awk command:

awk -F ',[[:blank:]]*' '{ print $1 > ($2 ".tmp") }' limpio.tmp

The expression ($2 ".tmp") evaluates to the string resulting from adding .tmp to whatever is in the 2nd field.

Using > with print means that the output file will be created or truncated (emptied) on the first print and that only subsequent print to the same name would append new lines. If you were to use >> in place of >, the first print would append to the file rather than truncate the file.

There is no built-in check for sanity here, so the above may clobber important files anywhere on the system where you have write access, depending on what's in the 2nd column of the input.

If you use GNU awk, then this is all you need to do. But if you use some other awk and need to write to a vast number of files, you may need to manually call close() on the files as you have processed them, or you may run out of available file descriptors.

awk -F ',[[:blank:]]*' '{ fn = $2 ".tmp"; print $1 >>fn; close(fn) }' limpio.tmp

Here, I'm saving the output filename in a variable, fn. I then append the output to that file, and then I close the file. This ensures that I won't exhaust the available file descriptors, but it also requires me to always use >> with print and manually clean up (remove) the output files if I need to run the awk command again.

GNU awk keeps track of how many file descriptors have been opened and starts closing them if needed (and re-opens the files for appending if further data needs to be written to a file later).

See also:

• Why is using a shell loop to process text considered bad practice?

Answer 2

This is the code you originally posted:

cat limpio.tmp | while read line; do
  srcip=$(echo $line | awk -F"," '{print $1}')
  destip=$(echo $line | awk -F"," '{print $2}')
  port=$(echo $line | awk -F"," '{print $NF}')
  echo "${srcip}" >> "$destip.tmp"
done

That is unnecessarily complicated. Seeing that you're reading comma delimited fields, you could read them directly into variables:

while IFS=$' \t\n,' read -r srcip destip port junk ;do
  echo "$srcip" >> "${destip}.tmp"
done <limpio.tmp

Answer 3

The easiest fix probably is to remove the double-quotes in echo "${srcip}".

See yourself:

var=' spacy '
echo X$(echo "$var")X
echo X$(echo $var)X