33
> brew install moreutils                                                          
==> Downloading https://homebrew.bintray.com/bottles/moreutils-0.55.yosemite.bottle.tar.gz    
######################################################################## 100.0%               
==> Pouring moreutils0.55.yosemite.bottle.tar.gz       
  /usr/local/Cellar/moreutils/0.55: 67 files, 740K

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file. This allows constructing pipelines that read from and write to the same file.

I don't understand. Please give me some useful examples.

What does soaks up mean?

cuonglm
  • 153,898
Sybil
  • 1,823
  • 5
  • 20
  • 41

3 Answers3

47

Assume that you have a file named input, you want to remove all line start with # in input. You can get all lines don't start with # using:

grep -v '^#' input

But how do you make changes to input? With standard POSIX toolchest, you need to use a temporary file, some thing like:

grep -v '^#' input >/tmp/input.tmp
mv /tmp/input.tmp ./input

With shell redirection:

grep -v '^#' input >input

will truncate input before you reading from it.

With sponge, you can:

grep -v '^#' input | sponge input
cuonglm
  • 153,898
  • 5
    You actually can read and write a file at the same time safely as long as the bytes are only being transformed, using the <> operator. – Chris Down Jun 06 '15 at 09:14
  • @ChrisDown: Yes, I mean without making it corroupt – cuonglm Jun 06 '15 at 09:19
  • 1
    @ChrisDown: Let me remove that sentence to avoid confusing. I actually mean when using <>file, you open file for reading and writing but you actually don't write anything to file. – cuonglm Jun 07 '15 at 07:12
  • Did you try it? :-) <> has no problem writing to a file, try it for yourself: printf foobar >q; printf bar 1<>q results in barbar. – Chris Down Jun 07 '15 at 07:46
  • @ChrisDown: Of course, But I mean in the context of reading and writing to file, like <file >file. Continue with your above example, printf '\nfoo' >q; grep bar <>q make q un-changes. – cuonglm Jun 07 '15 at 08:02
  • @ChrisDown: Sorry, mis-typing printf '\nfoo' >>q; – cuonglm Jun 07 '15 at 08:23
  • Because grep bar there doesn't output anything :-) If it did, it would write. – Chris Down Jun 07 '15 at 08:23
  • @ChrisDown: Even if grep output anything, it never goes to q. – cuonglm Jun 07 '15 at 08:36
  • @cuonglm You need to explicitly specify the FD with1<>, not just <>. That's why your example doesn't work. – Chris Down Jun 07 '15 at 21:55
  • @ChrisDown: grep will ignore if using 1<> – cuonglm Jun 08 '15 at 01:18
  • 8
    I think the point that @ChrisDown is trying to make is that <> doesn't truncate a file, but merely replaces its existing bytes with the new output. If the new output is too short, you'll have leftover garbage at the end of the file. But if the new output is long enough, there's no risk. – BallpointBen Aug 02 '18 at 13:56
20

The moreutils home page itself documents a typical use case:

sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

Here, /etc/passwd is both being written to and read to, and is being modified. Without sponging up stdin before writing, /etc/passwd might be corrupted (as the file changed during reading).

Chris Down
  • 125,559
  • 25
  • 270
  • 266
  • 4
    And that would be a good example on the moreutils page, if it had explained the way you did :-) – Br.Bill Jan 24 '20 at 22:55
0

The easiest example of sponge I've ever read:

$ cat file1 
I
You
Me
We
Us

To sort it, you can simply

$ sort file1 > file1_sorted

However, with sponge, you can read from and write to the same file.

$ sort file1 | sponge file1

without requiring the temporary file file1_sorted. This is because sponge first read from STDIN, and then write it out to STDOUT, in separate steps, thus avoiding corrupting the file as it is changed during the reading.

Rubem Pacelli
  • 263
  • 1
    Using sort as an example is a bit awkward, as you would rather use sort -o file1 file1 to sort the file in place (i.e., sort already has this "built-in", so to speak). – Kusalananda Mar 12 '24 at 09:41