Why does awk stop and wait if the filename contains = and how to work around that?

Question

awk 'processing_script_here' my=file.txt

seems to stop and wait indefinitely...
What's going on here and how do I make it work ?

Answer 1

In most versions of awk, arguments after the program to execute are either:

1. A file
2. An assignment of the form x=y

Since your filename is being interpreted as case #2, awk is still waiting for something to read on stdin (since it doesn't perceive that there has been any filename passed).

Portably, this behaviour is documented in POSIX:

Either of the following two types of argument can be intermixed:

• file: A pathname of a file that contains the input to be read, which is matched against the set of patterns in the program. If no file operands are specified, or if a file operand is '-', the standard input shall be used.
• assignment: An operand that begins with an underscore or alphabetic character from the portable character set (see the table in the Base Definitions volume of IEEE Std 1003.1-2001, Section 6.1, Portable Character Set), followed by a sequence of underscores, digits, and alphabetics from the portable character set, followed by the '=' character, shall specify a variable assignment rather than a pathname.

As such, portably, you have a few options (#1 is likely the least intrusive):

1. Use awk ... ./my=file, which sidesteps this since . is not "an underscore or alphabetic character from the portable character set".
2. Put the file on stdin using awk ... < my=file. However, this doesn't work well with multiple files.
3. Make a hardlink to the file temporarily, and use that. You can do something like ln my=file my_file, and then use my_file as normal. No copying will be performed, and both files will be backed by the same data and inode metadata. After using it, it's safe to remove the link created as the number of references to the inode will still be greater than 0.

Answer 2

As Chris says, arguments of the form variablename=anything are treated as variable assignment (that are performed at the time the arguments are processed as opposed to the (newer) -v var=value ones which are performed before the BEGIN statements) instead of input file names.

That can be useful in things like:

awk '{print $1}' FS=/ RS='\n' file1 FS='\n' RS= file2

Where you can specify a different FS/RS per file. It's also commonly used in:

awk '!file1_processed{a[$0]; next}; {...}' file1 file1_processed=1 file2

Which is a safer version of:

awk 'NR==FNR{a[$0]; next}; {...}' file1 file2

(which doesn't work if file1 is empty)

But that gets in the way when you have files whose name contains = characters.

Now, that's only a problem when what's left of the first = is a valid awk variable name.

What constitutes a valid variable name in awk is stricter than in sh.

POSIX requires it to be something like:

[_a-zA-Z][_a-zA-Z0-9]*

With only characters of the portable character set. However, the /usr/xpg4/bin/awk of Solaris 11 at least is not compliant in that regard and allows any alphabetical characters in the locale in variable names, not just a-zA-Z.

So an argument like x+y=foo or =bar or ./foo=bar is still treated as an input file name and not an assignment as what's left of the first = is not a valid variable name. An argument like Stéphane=Chazelas.txt may or may not, depending on the awk implementation and locale.

That's why with awk, it's recommended to use:

awk '...' ./*.txt

instead of

awk '...' *.txt

for instance to avoid the problem if you can't guarantee the name of the txt files won't contain = characters.

Also, beware that an argument like -vfoo=bar.txt may be treated as an option if you use:

awk -f file.awk -vfoo=bar.txt

(also applies to awk '{code}' -vfoo=bar.txt with the awk from busybox versions prior to 1.28.0, see corresponding bug report).

Again, using ./*.txt works around that (using a ./ prefix also helps with a file called - which otherwise awk understands as meaning standard input instead).

That's also why

#! /usr/bin/awk -f

shebangs don't really work. While the var=value ones can be worked around by fixing the ARGV values (add a ./ prefix) in a BEGIN statement:

#! /usr/bin/awk -f
BEGIN {
  for (i = 1; i < ARGC; i++)
    if (ARGV[i] ~ /^[_[:alpha:]][_[:alnum:]]*=/)
      ARGV[i] = "./" ARGV[i]
}
# rest of awk script

That won't help with the option ones as those ones are seen by awk and not the awk script.

One potential cosmetic issue with using that ./ prefix is it ends up in FILENAME, but you can always use substr(FILENAME, 3) to strip it if you don't want it.

The GNU implementation of awk fixes all those issues with its -E option.

After -E, gawk expects only the path of the awk script (where - still means stdin) and then a list of input file paths only (and there, not even - is treated specially).

It's specially designed for:

#! /usr/bin/gawk -E

shebangs where the list of arguments are always input files (note that you're still free to edit that ARGV list in a BEGIN statement).

You can also use it as:

gawk -e '...awk code here...' -E /dev/null *.txt

We use -E with an empty script (/dev/null) just to make sure those *.txt afterwards are always treated as input files, even if they contain = characters.

Answer 3

To quote gawk documentation ( note emphasis added ):

Any additional arguments on the command line are normally treated as input files to be processed in the order specified. However, an argument that has the form var=value, assigns the value value to the variable var—it does not specify a file at all.

Why does the command stop and wait ? Because in the form awk 'processing_script_here' my=file.txt there is no file specified by the above definition - my=file.txt is interpreted as variable assignment, and if there's no file defined awk will read stdin ( also evident from strace which shows that awk in such command is waiting on read(0,'...) syscall.

This is also documented in POSIX awk specifications, see OPERANDS section and assignments part of that )

Variable assignment is evident in awk '{print foo}' foo=bar /etc/passwd that value of foo is printed for every line in /etc/passwd. Specifying ./foo=bar or full path however does work.

Note that running strace on awk '1' foo=bar as well as checking with cat foo=bar shows that this is awk-specific issue, and execve does show filename as argument passed, so shells have nothing to do with env variable assignments in this case.

Additionally, please note that awk '...script...' foo=bar will not cause environment variable creation by shell, since environment variable assignments should be preceding a command to take effect. See POSIX Shell Grammar Rules, point number 7. Additionally this can be verified via awk '{print ENVIRON["foo"]}' foo=bar /etc/passwd