Why redirecting output sometimes produces an empty file?

Question

The power of shell pipeline is so great that sometimes fails me.

Example

Just as an example, the pipeline

echo abc > file.txt
cat file.txt | sed 's/a/1/' > file.txt

gives me an empty file.txt. Realizing that the shell probably calls > first, I made a change:

echo abc > file.txt
{cat file.txt | sed 's/a/1/'} > file.txt

Again it surprises me by another empty file file.txt. An ugly way that finally works is

echo abc > file.txt
echo $(cat file.txt | sed 's/a/1') > file.txt

which forces the shell to run a subshell first, and then redirect.

Question

While I'm aware of better practice of sed, which allows you to get rid of echo, cat, grep.. etc, what I am curious about here is to learn shell's grammar completely. This questions is not about how to fix the particular problem above.

Q1(EDIT: off-topic) Is there a good resource for me to learn the grammar?

I'm afraid that different shells could have different grammars, so

Q2 Can I make shell verbose, and see clearly what it is doing every completely every single time I run a command?

Q3(EDIT: off-topic) Any other advice on good practices? Thank you!

Q1 & Q3 are off-topic for U&L; please edit your question down to Q2 only. Thank you! — Jeff Schaller, May 13 '20 at 19:23
The easiest way to enable "verbose" is to turn on the trace flag: bash -x script.sh or in an interactive shell set -x — glenn jackman, May 13 '20 at 20:20
Good resources to learn the grammar: not really. In my experience, the best way is to practice practice practice, ask questions, and follow the [tag:shell] tag here and on stackoverflow. — glenn jackman, May 13 '20 at 20:22
Thank you.. @SergiyKolodyazhnyy but that was just an example to illustrate my situation: I not only want to solve this problem, but also want to avoid similar problems in the future. — Student, May 14 '20 at 03:38
You could also use exec 3<file.txt; rm file.txt; sed 's/a/1/' <&3 >file.txt (At least with bash. Not 100% sure about sh.) But of course, using sed's -i option is better. — ikegami, May 14 '20 at 04:44
@ikegami That's a pretty good solution. Can you post that to the linked question please ? Let me know once you do — Sergiy Kolodyazhnyy, May 14 '20 at 04:49
@Sergiy Kolodyazhnyy, Done, with an explanation. And it does appear to work in sh. (Well, I tested with dash, which I believe is suppose to be a compliant implementation of sh?) — ikegami, May 14 '20 at 05:22
@ikegami Yes, on Ubuntu/Debian-based OSs /bin/sh is dash, but it can differ by distro. But theoretically any POSIX-compliant version of /bin/sh have exec ( listed in built-in spec ) — Sergiy Kolodyazhnyy, May 14 '20 at 07:02
Same question on Stack Overflow: ubuntu - Why doesnt "tail" work to truncate log files? - Stack Overflow — user202729, Oct 19 '21 at 15:37

score 19 · Answer 1 · answered May 13 '20 at 20:01

Consider 3.1.1 Shell Operation, particularly the order in which things are done:

redirections are processed before the command is actually executed, but
expansions are processed before redirections.

This means that for cat file > file, the output redirection (which truncates the file) occurs before cat is spawned, and cat now has an empty file to work with.

But echo "$(cat file)" > file does what you expect because the Command Substitution is a Shell Expansion, and that happens before redirections.

The typical advice is to do

cat file > tmpfile && mv tmpfile file

You can use mktemp here.

Or install the moreutils package and use sponge

cat file | sponge file

Although to address the specific command you're using, replace

cat file.txt | sed 's/a/1/' > file.txt

with (assuming GNU sed)

sed -i 's/a/1/' file.txt

echo does what you expect, if you expect the wildly different behaviors echo can have (https://unix.stackexchange.com/questions/65803). The shortest example would be for file to contain only the string -n. — Kusalananda, May 13 '20 at 20:19

score 6 · Accepted Answer · answered May 13 '20 at 19:18

You have successfully found one of the things you should not do :-) Never redirect to the file you are working on!

A1: a good resource to learn shell grammar would be the absolute bash scripting guide, IMO

A2: For bash-scripts, you can use set +x for more verbose output, but I don't know how to achieve the same at a 'run things at the prompt'-level.

A3: Add [solved] to your search-terms. Finds you the solution to your problem instead of more of the problem you already know.

.. Having set +x, it still doesn't hint me why there's a problem. I guess I should just make friend with shell(s), and know all of its (their) perks. — Student, May 14 '20 at 00:53

score 0 · Answer 3 · answered May 14 '20 at 07:59

One way of solving the problem is learning exactly what order redirections, expansions, subshells etc happen so that you can catch this type of error before it happens.

I do not recommend this. You will still mess up occasionally and the cost can be lost data.

It is better to not trust your ability to keep all this straight and write more idiot-proof code. We are all idiots sometimes.

So

cp file.txt file.txt.old
cat file.txt.old | sed 's/a/1/' > file.txt

This is simpler in that there is no dependencies on order of events, other than line 1 happening before line 2.

As an added benefit it guards against other errors, when you shouldn't have run this script at all.

The cost of this practice is that you will have .old files laying around everywhere. This is an insurance premium you will be glad you paid the day you need it. Disk is cheap.

@Kusalananda Well, mostly I recommend not depending on that knowledge. But feel free to learn it if you want to. — Stig Hemmer, May 14 '20 at 08:20
Well, I guess @StigHemmer just recommends to backup always. Thanks for reminding :) — Student, May 14 '20 at 11:07

Why redirecting output sometimes produces an empty file?

Example

Question

3 Answers3

Linked