Why do these two commands to write text-processing results back to the input file behave so differently?

Question

I have a file authorized_keys and want to deduplicate the content, i.e. remove duplicate entries.

I found two possible solutions to achieve this:

Use cat and uniq, then redirect the output to the original file:
```
cat authorized_keys | uniq > authorized_keys
```

Use sort with the -o option:

sort -u ~/.ssh/authorized_keys -o ~/.ssh/authorized_keys

However, only the second option works. The first approach will result in an empty file.

If I redirect the output to a different file, as in

cat authorized_keys | uniq > authorized_keys_2

the new file with has correct content, while

cat authorized_keys | uniq > authorized_keys

leaves me with an empty authorized_keys.

Please help me understand the difference between these two methods.

See https://unix.stackexchange.com/questions/159513/what-are-the-shells-control-and-redirection-operators#186126 — steeldriver, Mar 24 '22 at 00:52
Does this answer your question? What are the shell's control and redirection operators? — Greenonline, Mar 30 '22 at 23:10
Does this answer your question? Bending a pipeline back into its origin — muru, Mar 31 '22 at 02:25
Btw, you should sort the file when using uniq, so the first way should actually be sort authorized_keys | uniq — muru, Mar 31 '22 at 02:26

score 1 · Answer 1 · answered Mar 24 '22 at 00:54

Your first solution won't work because you're reading and writing to a file at the same time. Doing so causes an empty file because a redirection, i.e. the '>' symbol is handled by your shell and instructs the shell to first open the file for writing, thus emptying it. Then your command, i.e.

cat authorized_keys | uniq

is executed, but since the file authorized_keys is now empty it results in an empty output. To avoid problems like this, you can use a temporary file, like you used authorized_keys_2 to store the input then overwrite the original file with the temporary one

Greenonline · Answer 2 · 2022-03-30T23:12:26.423

To expand upon SeetheMoar's answer...

From the bash man page

Redirecting Output
Redirection of output causes the file whose name results from the expansion
of word to be opened for writing on file descriptor n, or the standard 
output (file descriptor 1) if n is not specified. If the file does not exist 
it is created; if it does exist it is truncated to zero size.

Your file authorized_keys does exist, and so is truncated to zero, before the command is executed.

You then cat a zero length file, hence the resulting file is empty.

This process is explained in depth by this answer to What are the shell's control and redirection operators?

Why do these two commands to write text-processing results back to the input file behave so differently?

2 Answers2