9

Possible Duplicate:
How can I make iconv replace the input file with the converted output?

I frequently connect to amazon ec2 using their public DNS names
(ec2-12-34-56-78.compute-1.amazonaws.com) and because of this my known_hosts file gets overwhelmed with a lot of ec2 entries I will never use again.

I know I could probably use sed -i to edit in place but I wanted to use grep and so I did this:

grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/known_hosts

That leaves known_hosts as an empty file. If I do:

grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/tmp
mv ~/.ssh/tmp ~/.ssh/known_hosts

then things are fine, but I am confused why reading and writing to the same file leaves it blank, and if there is any way around this when using grep, cat, etc.

Community
  • 1
cwd
  • 45,389
  • 1
    Even if the program were doing the redirection itself, opening the file for read and then for write before actually reading from it would result in reading nothing (unless the language's standard library speculatively prebuffered, but that means you can't configure the buffer first). What were you expecting to happen? – geekosaur Apr 11 '12 at 19:33
  • Sounds good. Thanks. Why is this not part of your answer? – cwd Apr 11 '12 at 19:58
  • 1
    Because it's actually a related issue, not the original problem (the shell is doing the file create, so grep would not have a chance to do anything about it anyway). I'm just confused as to how it would be expected to work in the first place. – geekosaur Apr 11 '12 at 20:00
  • 1
    This is not the first time I've experienced this, so I don't "expect" to to work anymore, but initially I assumed that input flows from left to right and you operate on it on the left side and then would output it to the file on the right side. Guess part of the problem is that years ago I was a windows user and something similar would work in DOS. – cwd Apr 11 '12 at 20:34
  • 1
    Even DOS (2 and up) would do "the wrong thing" in most circumstances, although it at least would do the speculative prebuffer in many cases so you would see it appeart to work if the file was small enough; moreover, DOS 2 and later had shell redirection in COMMAND.COM and so had the overwrite happen before the command even ran. – geekosaur Apr 11 '12 at 20:38
  • 1
    Well thank goodness for sponge then :) – cwd Apr 11 '12 at 21:37
  • 1
    What's wrong with sed -i /ec2/d? – ephemient Apr 11 '12 at 22:49
  • 1
    @ephemient - nothing is wrong with that for this specific example. – cwd Apr 11 '12 at 22:57
  • @Gilles - I believe that question is written with too much specificity to be found or read when searching for the core of the problem being discussed here. – cwd Apr 11 '12 at 23:00
  • 3
    @cwd Hence the value of having this one closed as a duplicate, with a highly-visible link to the other question. – Gilles 'SO- stop being evil' Apr 11 '12 at 23:06
  • @Gilles - wouldn't you want to have the more versatile question stay open? – cwd Apr 12 '12 at 15:28
  • @cwd It's better to use the one with the best answers. (I'm not sure I picked the right one, I have an easier time finding my own answers so I'm biased.) But it doesn't matter much. – Gilles 'SO- stop being evil' Apr 12 '12 at 19:56

2 Answers2

16

There is a utility called sponge that is a part of the moreutils suite. It was made for this exact purpose.

grep -v ec2 ~/.ssh/known_hosts | sponge ~/.ssh/known_hosts
Shawn J. Goff
  • 46,081
  • Why not just piping to cat? Like: ... | cat > ~/.ssh/known_hosts ? – hek2mgl Jul 03 '14 at 13:47
  • 4
    cat can write to its output at any time; if it writes past the point at whatever is on the other side of the pipe has read, then you will read in what you wrote instead of the original file contents. That is what makes sponge special. It buffers its input until stdin closes, then writes. – Shawn J. Goff Jul 03 '14 at 13:54
  • 1
    This might be interesting: If sponge isn't available, you could emulate it using sed: ... | sed ':a;N;ba' – hek2mgl Jul 04 '14 at 11:49
  • 1
    sponge is also part of sbase – Sebastian Oct 16 '14 at 13:23
10

Redirections are done by the shell, before the command runs. This means that the shell is told to truncate the file before grep gets a chance to read it. There is no way around this if you are using shell redirection.

geekosaur
  • 32,047