1

When I do my_script < filename.txt > filename.txt, the file is overwritten and truncated.

Is there some way on the Unix command line to specify that redirection is not done concurrently, i.e. the output does not begin until the input has completed?

I am trying to write a utility that reads a file, and based on the command line options, regenerates and overwrites it. I realize I could add support in the program for not using stdin/stdout, but I like the flexibility and convenience of redirection.

Keith Bennett
  • 315
  • 3
    No, the shell handles the redirections before starting the program, it can't mess with the child process's file descriptors afterwards. Even if that would be worked around, the shell still couldn't know when the program is "done" with the input, i.e. when the shell would be allowed to truncate the output file. Just do it in the program, let it have an option like -o outputfile, as in sort. Or use sponge – ilkkachu Sep 16 '21 at 21:48
  • I don't think sponge helps; I just tried it (cat file.txt | sponge > file.txt) and the file is still empty after the command is executed. – Keith Bennett Sep 16 '21 at 22:11
  • 3
    Use an argument NOT a redirect: sponge file.txt. This is shown on the man page. Please learn to look at the man page for a program you want to use or at least one you are having trouble with. Or just follow the link @ikkachu gave to a previous Q -- you did notice your browser shows that word is a hyperlink, right? – dave_thompson_085 Sep 17 '21 at 00:57
  • 1
    Thanks for letting me know about sponge. Regarding my failure to RTFM, I am guilty as charged. Sorry. – Keith Bennett Sep 17 '21 at 17:30

2 Answers2

6

You can do this:

(rm -f foo && yourprogram > foo) < foo

E.g.:

(rm -f foo && wc > foo) < foo

It opens foo for reading. Then it starts a subshell, and removes the i-node of foo while keeping the file open. Finally it opens foo for output, thus creating foo.

It requires write permission to the dir, so if you only have write permission to the file, you are out of luck.

It will change the i-node (so permission, owner, ctime is lost), but if only the name is important, it should be OK.

Contrary to sponge this works even if the output of yourprogram is bigger than memory.

Ole Tange
  • 35,514
  • Could you explain exactly what is happening here? When I test this out, I get things like 2 2 13 for cat foo. I would be very interested in understanding what is going on with this one liner in a little more depth. – Kahn Sep 17 '21 at 20:22
  • @Kahn, cat or wc? – ilkkachu Sep 17 '21 at 20:31
  • @zevzek You can do that, too. But you need to make sure you make the filename in a secure manner (mktemp) as you may otherwise be open to security issues. – Ole Tange Sep 17 '21 at 21:50
  • For these reasons I think the 'sponge' solution is the best. It does, of course, assume that one is willing and able to install 'sponge', and that may not always be true. That said, I think this answer is interesting and useful. – Keith Bennett Sep 18 '21 at 00:37
  • @Kahn What's happening there is that the content of the 'foo' file is being replaced with the text output when wc foo is run. Those numbers are the output of wc. – Keith Bennett Sep 20 '22 at 09:46
1

It is possible to read a file into a here string buffer and then overwrite the same file (shell bash):

my_script <<<$(<filename.txt) >filename.txt
nezabudka
  • 2,428
  • 6
  • 15
  • 1
    With the caveat that $() removes all trailing newlines and <<< puts back exactly one, so a file with 0, 2, or more than 2 newlines at the end will not be read exactly right. – ilkkachu Sep 18 '21 at 08:49