IO redirection and the head command

Question

I was trying to quickly edit an .hgignore file from the Cygwin bash shell today, and I added a line that was a mistake. I'm not sure if this was the best way to do it, but I quickly thought of using head -1 .hgignore to remove the offending line (I had previously only had one line in the file). Sure enough, when executed it gives the first line as the only output.

But when I tried to redirect the output and rewrite the file using head -1 .hgignore > .hgignore, the file was empty. Why does this happen? If I try appending instead, head -1 .hgignore >> .hgignore, it appends correctly but this is obviously not the desired result. Why does a truncating redirect not work in this case?

Answer 1

I think Bruce answers what's going on here with the shell pipeline.

One of my favorite little utilities is the sponge command from moreutils. It solves exactly this problem by "soaking" up all available input before it opens the target output file and writing the data. It allows you to write pipelines exactly how you expected to:

$ head -1 .hgignore | sponge .hgignore

The poor-man's solution is to pipe the output to a temporary file, then after the pipline is done (for example the next command you run) is to move the temp file back to the original file location.

$ head -1 .hgingore > .hgignore.tmp
$ mv .hgignore{.tmp,}

Answer 2

When the shell gets a command line like: command > file.out the shell itself opens (and maybe creates) the file named file.out. The shell sets file descriptor 0 to the file file descriptor it got from the open. That's how I/O redirection works: every process knows about file descriptors 0, 1 and 2.

The hard part about this is how to open file.out. Most of the time, you want file.out opened for write at offset 0 (i.e. truncated) and this is what the shell did for you. It truncated .hgignore, opened it for write, dup'ed the filedescriptor to 0, then exec'ed head. Instant file clobbering.

In bash shell, you do a set noclobber to change this behavior.

Answer 3

In

head -n 1 file > file

file is truncated before head is started, but if you write it:

head -n 1 file 1<> file

it's not as file is opened in read-write mode. However, when head finishes writing, it doesn't truncate the file, so the line above would be a no-op (head would just rewrite the first line over itself and leave the other ones untouched).

However, after head has returned and while the fd is still open, you can call another command that does the truncate.

For instance:

{ head -n 1 file; perl -e 'truncate STDOUT, tell STDOUT'; } 1<> file

What matters here is that truncate above, head just moves the cursor for fd 1 inside the file just after the first line. It does rewrite the first line which we didn't need it to, but that's not harmful.

With a POSIX head, we could actually get away without rewriting that first line:

{ head -n 1 > /dev/null
  perl -e 'truncate STDIN, tell STDIN'
} <> file

Here, we're using the fact that head moves the cursor position in its stdin. While head would typically read its input by big chunks to improve performance, POSIX would require it (where possible) to seek back just after the first line if it had gone beyond it. Note however that not all implementations do it.

Alternatively, you can use the shell's read command instead in this case:

{ read -r dummy; perl -e 'truncate STDIN, tell STDIN'; } <> file

Answer 4

The Real Man's solution is

ed .hgignore
$d
wq

or as a one-liner

printf '%s\n' '$d' 'wq' | ed .hgignore

Or with GNU sed:

sed -i '$d' .hgignore

(No, I'm kidding. I'd use an interactive editor. vi .hgignore GddZZ)

Answer 5

You can use Vim in Ex mode:

ex -sc '2,d|x' .hgignore

1. 2, select lines 2 until end

2. d delete

3. x save and close

Answer 6

For in-place file editing you may also use the open file handle trick as shown by Jürgen Hötzel in Redirect output from sed 's/c/d/' myFile to myFile.

exec 3<.hgignore
rm .hgignore  # prevent open file from being truncated
head -1 <&3 > .hgignore

ls -l .hgignore  # note that permissions may have changed