0

I'm writing a command to prepend some text to a bunch of files which match a certain pattern:

for file in $(find . -name "*.txt"); do sed "1s/^/hello/" $file; done

If I write to stdout or a different file (e.g. "$file.bak"), everything works according to plan, but when I try to have the command write back to the same file, the content of the files gets wiped out:

for file in $(find . -name "*.txt");do sed "1s/^/hello/" $file > $file; done

I find the behavior surprising, could someone explain why this happens, conceptually? I'm on macOS and I've tried this in bash and zsh to the same effect.

(Note that I'm aware of sed -i "" for in-place writing, but I'd like to pipe the result to a different command before writing in-place.)

Dan
  • 133
  • 1
    Some more detailed discussion you may find useful is here – Tagwint Jun 09 '20 at 15:34
  • @Inian I wouldn't mark my question as a duplicate, but it did, indirectly, show me a possible solution to my problem: the tee utility, which I'm learning about now, seems to allow me to write back to the original file. – Dan Jun 09 '20 at 15:42
  • A correction to my previous comment: as per this Q & A it does not seem that tee is a viable alternative to sponge. – Dan Jun 09 '20 at 15:53

1 Answers1

4

The conceptual surprise probably is in this statement here

but when I try to have the command write back to the same file

When using cmd file > file it's not cmd opening the file to write back into but the shell itself. And the first thing a shell does when seeing > file is to open the file in write mode at position 0 (thereby truncating it) and then redirect stdout to this file. So at the moment cmd starts to read from file it is already empty.

Or, a bit more formalized:

  • Shell forks a new process to run cmd in
  • New process (still running shell code) opens file for writing at position 0 (thereby truncating it)
  • New process executes cmd, practically starting a new binary within the current process
  • cmd runs with stdout pointing to file
  • cmd opens file for reading but it is already empty at this moment

On the other hand, if you use sed -i '1s/^/hello/' file the file is solely managed by sed (which technically creates a temporary file in the background then).

nohillside
  • 3,251