Why doesn't shell automatically fix "useless use of cat"?

Question

Many people use oneliners and scripts containing code along the lines

cat "$MYFILE" | command1 | command2 > "$OUTPUT"

The first cat is often called "useless use of cat" because technically it requires starting a new process (often /usr/bin/cat) where this could be avoided if the command had been

< "$MYFILE" command1 | command2 > "$OUTPUT"

because then shell only needs to start command1 and simply point its stdin to the given file.

Why doesn't the shell do this conversion automatically? I feel that the "useless use of cat" syntax is easier to read and shell should have enough information to get rid of useless cat automatically. The cat is defined in POSIX standard so shell should be allowed to implement it internally instead of using a binary in path. The shell could even contain implementation only for exactly one argument version and fallback to binary in path.

Answer 1

"Useless use of cat" is more about how you write your code than about what actually runs when you execute the script. It's a sort of design anti-pattern, a way of going about something that could probably be done in a more efficient manner. It's a failure in understanding of how to best combine the given tools to create a new tool. I'd argue that stringing several sed and/or awk commands together in a pipeline also sometimes could be said to be a symptom of this same anti-pattern.

Fixing instances of "useless use of cat" in a script is a primarily matter of fixing the source code of the script manually. A tool such as ShellCheck can help with this by pointing out the obvious cases:

$ cat script.sh
#!/bin/sh
cat file | cat

$ shellcheck script.sh

In script.sh line 2:
cat file | cat
    ^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.

---

Getting the shell to do this automatically would be difficult due to the nature of shell scripts. The way a script executes depends on the environment inherited from its parent process, and on the specific implementation of the available external commands.

The shell does not necessarily know what cat is. It could potentially be any command from anywhere in your $PATH, or a function.

If it was a built-in command (which it may be in some shells), it would have the ability to reorganise the pipeline as it would know of the semantics of its built-in cat command. Before doing that, it would additionally have to make assumptions about the next command in the pipeline, after the original cat.

Note that reading from standard input behaves slightly differently when it's connected to a pipe and when it's connected to a file. A pipe is not seekable, so depending on what the next command in the pipeline does, it may or may not behave differently if the pipeline was rearranged (it may detect whether the input is seekable and decide to do things differently if it is or if it isn't, in any case it would then behave differently).

This question is similar (in a very general sense) to "Are there any compilers that attempt to fix syntax errors on their own?" (at the Software Engineering StackExchange site), although that question is obviously about syntax errors, not useless design patterns. The idea about automatically changing the code based on intent is largely the same though.

Answer 2

Because it's not useless.

In the case of cat file | cmd, the fd 0 (stdin) of cmd will be a pipe, and in the case of cmd <file it may be a regular file, device, etc.

A pipe has different semantics from a regular file, and its semantics are not a subset of those of a regular file:

• a regular file cannot be select(2)ed or poll(2)ed on in a meaningful way; a select(2) on it will always return "ready". Advanced interfaces like epoll(2) on Linux will simply not work with regular files.

• on Linux there are system calls (splice(2), vmsplice(2), tee(2)) which only work on pipes [1]

Since cat is so much used, it could be implemented as a shell built-in which will avoid an extra process, but once you started on that path, the same thing could be done with most commands -- transforming the shell into a slower & clunkier perl or python. it's probably better to write another scripting language with an easy to use pipe-like syntax for continuations instead ;-)

[1] If you want a simple example not made up for the occasion, you can look at my "exec binary from stdin" git gist with some explanations in the comment here. Implementing cat inside it in order to make it work without UUoC would have made it 2 or 3 times bigger.

Answer 3

The 2 commands are not equivalent: consider error handling:

cat <file that doesn't exist> | less will produce an empty stream that will be passed to the piped program... as such you end up with a display showing nothing.

< <file that doesn't exist> less will fail to open bar, and then not open less at all.

Attempting to change the former to the latter could break any number of scripts that expect to run the program with a potentially blank input.

Answer 4

Because detecting useless cat is really really hard.

I had a shell script where I wrote

cat | (somecommand <<!
...
/proc/self/fd/3
...
!) 0<&3

The shell script failed in production if the cat was removed because it was invoked via su -c 'script.sh' someuser. The apparently superfluous cat caused the owner of standard input to change to the user the script was running as so that reopening it via /proc worked.

Answer 5

tl;dr: Shells don't do it automatically because the costs exceed the likely benefits.

Other answers have pointed out the technical difference between stdin being a pipe and it being a file. Keeping that in mind, the shell could do one of:

1. Implement cat as a builtin, still preserving the file v. pipe distinction. This would save the cost of an exec and maybe, possibly, a fork.
2. Perform a full analysis of the pipeline with knowledge of the various commands used to see if file/pipe matters, then act based on that.

Next you have to consider the costs and benefits of each approach. The benefits are simple enough:

1. In either case, avoid an exec (of cat)
2. In the second case, when redirect substitution is possible, avoidance of a fork.
3. In cases where you have to use a pipe, it might be possible sometimes to avoid a fork/vfork, but often not. That's because the cat-equivalent needs to run at the same time as the rest of the pipeline.

So you save a little CPU time & memory, especially if you can avoid the fork. Of course, you only save this time & memory when the feature is actually used. And you're only really saving the fork/exec time; with larger files, the time is mostly the I/O time (i.e., cat reading a file from disk). So you have to ask: how often is cat used (uselessly) in shell scripts where the performance actually matters? Compare it to other common shell builtins like test — it's hard to imagine cat is used (uselessly) even a tenth as often as test is used in places that matter. That's a guess, I haven't measured, which is something you'd want to do before any attempt at implementation. (Or similarly, asking someone else to implement in e.g., a feature request.)

Next you ask: what are the costs. The two costs that come to mind are (a) additional code in the shell, which increases its size (and thus possibly memory use), requires more maintenance work, is another spot for bugs, etc.; and (b) backwards compatibility surprises, POSIX cat omits a lot of features of e.g., GNU coreutils cat, so you'd have to be careful exactly what the cat builtin would implement.

1. The additional builtin option probably isn't that bad — adding one more builtin where a bunch already exist. If you had profiling data showing it'd help, you could probably convince your favorite shell's authors to add it.

2. As for analyzing the pipeline, I don't think shells do anything like this currently (a few recognize the end of a pipeline and can avoid a fork). Essentially you'd be adding a (primitive) optimizer to the shell; optimizers often turn out to be complicated code and the source of a lot of bugs. And those bugs can be surprising — slight changes in the shell script could wind up avoiding or triggering the bug.

Postscript: You can apply a similar analysis to your useless uses of cat. Benefits: easier to read (though if command1 will take a file as an argument, probably not). Costs: extra fork and exec (and if command1 can take a file as an argument, probably more confusing error messages). If your analysis tells you to uselessly use cat, then go ahead.

Answer 6

The cat command can accept - as a marker for stdin. (POSIX, "If a file is '-', the cat utility shall read from the standard input at that point in the sequence.") This allows simple handling of a file or stdin where otherwise this would be disallowed.

Consider these two trivial alternatives, where the shell argument $1 is -:

cat "$1" | nl    # Works completely transparently
nl < "$1"        # Fails with 'bash: -: No such file or directory'

Another time cat is useful is where it's intentionally used as a no-op simply to maintain shell syntax:

file="$1"
reader=cat
[[ $file =~ \.gz$ ]] && reader=zcat
[[ $file =~ \.bz2$ ]] && reader=bzcat
"$reader" "$file"

Finally, I believe the only time that UUOC can really be correctly called out is when cat is used with a filename that is known to be a regular file (i.e. not a device or named pipe), and that no flags are given to the command:

cat file.txt

In any other situation the oroperties of cat itself may be required.

Answer 7

The cat command can do things that the shell can't necessarily do (or at least, can't do easily). For example, suppose you want to print characters that might otherwise be invisible, such as tabs, carriage returns, or newlines. There *might* be a way to do so with only shell builtin commands, but I can't think of any off the top of my head. The GNU version of cat can do so with the -A argument or the -v -E -T arguments (I don't know about other versions of cat, though). You could also prefix each line with a line number using -n (again, IDK if non-GNU versions can do this).

Another advantage of cat is that it can easily read multiple files. To do so, one can simply type cat file1 file2 file3. To do the same with a shell, things would get tricky, although a carefully-crafted loop could most likely achieve the same result. That said, do you really want to take the time to write such a loop, when such a simple alternative exists? I don't!

Reading files with cat would probably use less CPU than the shell would, since cat is a pre-compiled program (the obvious exception is any shell that has a builtin cat). When reading a large group of files, this might become apparent, but I have never done so on my machines, so I can't be sure.

The cat command can also be useful for forcing a command to accept standard input in instances it might not. Consider the following:

echo 8 | sleep

The number "8" will be not accepted by the "sleep" command, since it was never really meant to accept standard input. Thus, sleep will disregard that input, complain about a lack of arguments, and exit. However, if one types:

echo 8 | sleep $(cat)

Many shells will expand this to sleep 8, and sleep will wait for 8 seconds before exiting. You can also do something similar with ssh:

command | ssh 1.2.3.4 'cat >> example-file'

This command with append example-file on the machine with the address of 1.2.3.4 with whatever is outputted from "command".

And that's (probably) just scratching the surface. I'm sure I could find more example of cat being useful if I wanted to, but this post is long enough as it is. So, I'll conclude by saying this: asking the shell to anticipate all of these scenarios (and several others) is not really feasible.

Answer 8

Remember that a user could have a cat in his $PATH which is not exactly the POSIX cat (but perhaps some variant which could log something somewhere). In that case, you don't want the shell to remove it.

The PATH could change dynamically, and then cat is not what you believe it is. It would be quite difficult to write a shell doing the optimization you dream of.

Also, in practice, cat is a quite quick program. There are few practical reasons (except aesthetics) to avoid it.

See also the excellent Parsing POSIX [s]hell talk by Yann Regis-Gianas at FOSDEM2018. It gives other good reasons to avoid attempting doing what you dream of in a shell.

If performance was really an issue for shells, someone would have proposed a shell which uses sophisticated whole program compiler optimization, static source code analysis, and just-in-time compilation techniques (all these three domains have decades of progress and scientific publications and dedicated conferences, e.g. under SIGPLAN). Sadly, even as an interesting research topic, that is not currently funded by research agencies or venture capitalists, and I am deducing that it is simply not worth the effort. In other words, there is probably no significant market for optimizing shells. If you have half a million euro to spend on such research, you'll easily find someone to do it, and I believe it would give worthwhile results.

On a practical side, rewriting, to improve its performance, a small (un hundred lines) shell script in any better scripting language (Python, AWK, Guile, ...) is commonly done. And it is not reasonable (for many software engineering reasons) to write large shell scripts: when you are writing a shell script exceeding a hundred lines, you do need to consider rewriting it (even for readability and maintenance reasons) in some more suitable language: as a programming language the shell is a very poor one. However, there are many large generated shell scripts, and for good reasons (e.g. GNU autoconf generated configure scripts).

Regarding huge textual files, passing them to cat as a single argument is not good practice, and most sysadmins know that (when any shell script takes more than a minute to run, you begin considering optimizing it). For large gigabytes files, cat is never the good tool to process them.

Answer 9

Adding to @Kusalananda answer (and @alephzero comment), cat could be anything:

alias cat='gcc -c'
cat "$MYFILE" | command1 | command2 > "$OUTPUT"

or

echo 'echo 1' > /usr/bin/cat
cat "$MYFILE" | command1 | command2 > "$OUTPUT"

There is no reason that cat (on its own) or /usr/bin/cat on the system is actually cat the concatenate tool.

Answer 10

Two "useless" uses for cat:

sort file.txt | cat header.txt - footer.txt | less

...here cat is used to mix file and piped input.

find . -name '*.info' -type f | sh -c 'xargs cat' | sort

...here xargs can accept a virtually infinite number of filenames and run cat as many times as needed while making it all behave like one stream. So this works for large file lists where direct use of xargs sort does not.

Answer 11

Aside from other things, cat-check would add additional performance overhead and confusion as to which use of cat is actually useless, IMHO, because such checks can be inefficient and create problems with legitimate cat usage.

When commands deal with the standard streams, they only have to care about reading/writing to the standard file descriptors. Commands can know if stdin is seekable/lseekable or not, which indicates a pipe or file.

If we add to the mix checking what process actually provides that stdin content, we will need to find the process on the other side of the pipe and apply appropriate optimization. This can be done in terms of shell itself, as shown in the SuperUser post by Kyle Jones, and in terms of shell that's

(find /proc -type l | xargs ls -l | fgrep 'pipe:[20043922]') 2>/dev/null

as shown in the linked post. This is 3 more commands (so extra fork()s and exec()s) and recursive traversals ( so whole lot of readdir() calls ).

In terms of C and shell source code, the shell already knows the child process,so there's no need for recursion, but how do we know when to optimize and when cat is actually useless ? There are in fact useful uses of cat, such as

# adding header and footer to file
( cmd; cat file; cmd ) | cmd
# tr command does not accept files as arguments
cat log1 log2 log3 | tr '[:upper:]' '[:lower:]'

It would probably be waste and unnecessary overhead to add such optimization to the shell. As Kusalanda's answer mentioned already, UUOC is more about user's own lack of understanding of how to best combine commands for best results.