59

I'm writing shell scripts for my server, which is a shared hosting running FreeBSD. I also want to be able to test them locally, on my PC running Linux. Hence, I'm trying to write them in a portable way, but with sed I see no way to do that.

Part of my website uses generated static HTML files, and this sed line inserts correct DOCTYPE after each regeneration:

sed -i '1s/^/<!DOCTYPE html> \n/' ${file_name.html}

It works with GNU sed on Linux, but FreeBSD sed expects the first argument after -i option to be extension for backup copy. This is how it would look like:

sed -i '' '1s/^/<!DOCTYPE html> \n/' ${file_name.html}

However, GNU sed in turn expects the expression to follow immediately after -i. (It also requires fixes with newline handling, but that's already answered in here)

Of course I can include this change in my server copy of the script, but that would mess i.e. my use of VCS for versioning. Is there a way to achieve this with sed in a fully portable way?

Kusalananda
  • 333,661
Red
  • 1,442
  • 1
  • 15
  • 18

7 Answers7

63

GNU sed accepts an optional extension after -i. The extension must be in the same argument with no intervening space. This syntax also works on FreeBSD sed.

sed -i.bak -e '…' SOMEFILE

Note that on FreeBSD, -i also changes the behavior when there are multiple input files: they are processed independently (so e.g. $ matches the last line of each file). Also this won't work on BusyBox.

If you don't want to use backup files, you could check which version of sed is available.

# Assume that sed is either FreeBSD/macOS or GNU
case $(sed --help 2>&1) in
  *GNU*) set sed -i;;
  *) set sed -i '';;
esac
"$@" -e '…' "$file"

Or alternatively, to avoid clobbering the positional parameters, define a function.

case $(sed --help 2>&1) in
  *GNU*) sed_i () { sed -i "$@"; };;
  *) sed_i () { sed -i '' "$@"; };;
esac
sed_i -e '…' "$file"

If you don't want to bother, use Perl.

perl -i -pe '…' "$file"

If you want to write a portable script, don't use -i — it isn't in POSIX. Do manually what sed does under the hood — it's only one more line of code.

sed -e '…' "$file" >"$file.new"
mv -- "$file.new" "$file"
Gilles 'SO- stop being evil'
  • 829,060
  • 3
    GNU sed -i also implies -s. And the easiest way to check for a GNU sed is with the sed v command which is a valid noop for GNU but fails everywhere else. – mikeserv Sep 04 '14 at 19:11
  • Inspired by the above tips, here's an single-line (if ugly) portable version for those who really want one, though it does spawn a subshell: sed -i$(sed v < /dev/null 2> /dev/null || echo -n " ''") -e '...' "$file" If it's not GNU sed, it inserts a space followed by two singe quotes after -i so that it works on BSD. GNU sed gets only -i. – Ivan X Apr 26 '15 at 21:42
  • 2
    @IvanX I'm wary of using the presence of the v command to test for GNU sed. What if FreeBSD decided to implement it? – Gilles 'SO- stop being evil' Apr 26 '15 at 21:54
  • @Gilles Fair point, but the man page for GNU sed describes v as being exactly for that purpose (detecting that it's GNU sed and not something else), so one would hope that *BSD would honor that. I can't think of another test, offhand, that takes no action on GNU sed, while causing an error on BSD sed (or vice versa), other than using -i itself, but that would require creating a dummy file first. Your test for sed above is OK but unwieldy for inline. Avoiding -i entirely, as you suggest, certainly seems like the safest bet, but I'm ok with using sed v given that's its purpose for existing. – Ivan X Apr 26 '15 at 22:20
  • The trick with "$@" is very nice, but I wouldn't use it in a longer script. Unfortunately normal variables are not expanded in that way (and sh doesn't have arrays) so I couldn't find anything better. – lapo Mar 22 '16 at 10:45
  • 1
    @lapo An alternative is to define a function, see my edit. – Gilles 'SO- stop being evil' Mar 22 '16 at 12:04
  • For portability, perl in-place editing is a better answer than sed. The -i (GNU extension) is not POSIX as Gilles stated, and Perl doesn't need any special module or extension to perform this operation. – Daniel Liston Dec 01 '20 at 22:38
  • Not sure when it changed, but busybox 1.32 supports '-i.bak' just fine – someonewithpc Mar 21 '21 at 18:10
  • It's not GNU vs BSD, it's GNU/NetBSD/OpenBSD/busybox/toybox vs FreeBSD/macos. – Stéphane Chazelas Jul 16 '22 at 16:54
13

If you don't find a trick to make sed play nice, you could try:

  1. Don't use -i : 

    sed '1s/^/<!DOCTYPE html> \n/' "${file_name.html}" > "${file_name.html}.tmp" &&
  mv "${file_name.html}.tmp" "${file_name.html}"

  2. Use Perl

    perl -i -pe 'print "<!DOCTYPE html> \n" if $.==1;' "${file_name.html}"
cuonglm
  • 153,898
terdon
  • 242,166
9

ed

You can always use ed to prepend a line to an existing file.

$ printf '0a\n<!DOCTYPE html>\n.\nw\n' | ed my.html

Details

The bits around the <!DOCTYPE html> are commands to ed instructing it to add that line to the file my.html.

sed

I believe this command in sed can also be used:

$ sed -i '1i<!DOCTYPE html>\n` testfile.csv
slm
  • 369,824
  • I eventually resorted to Perl, but using ed is a good alternative that's not popular among Unix-like users as it should be. – Red Sep 30 '13 at 14:29
  • @Red - glad to hear you resolved you issue. Yeah I'd not seen that one before, googling turned it up and it actually seemed like the most portable, apt way to do this. – slm Sep 30 '13 at 14:30
8

You can also do manually what perl -i used to do under the hood:

{ rm -f file && { echo '<!DOCTYPE html>'; cat; } > file;} < file

Like perl -i, there's no backup, and like most solutions given here, beware it may affect the permissions, ownership of the file and may turn a symlink into a regular file.

With:

sed '1i\
<!DOCTYPE html>' file 1<> file

sed would overwrite the file over itself, so would not affect ownership and permissions or symlinks. It works with GNU sed because sed will typically have read a buffer full of data from file (4k in my case) before overwriting it with the i command. That wouldn't work if the file was more than 4k except for the fact that sed also buffers its output.

Basically sed works on blocks of 4k for reading and writing. If the line to insert is smaller than 4k, sed will never overwrite a block it has not read yet.

I wouldn't count on it though.

Beware that with all those solutions, if the file system is full or the system crashes in the middle, you may end up losing data.

Stéphane Chazelas
  • 544,893
  • Should be echo '<!DOCTYPE html>' or escaped without "" quotes. – A.D. May 22 '15 at 12:36
  • @A.D. Good point. I tend to forget about that bug^Wfeature of interactive bash/zsh as I generally disable it for myself. – Stéphane Chazelas May 22 '15 at 12:41
  • This is one of the very few answers here that doesn't have a wide open security hole of redirecting to a "temp file" with a static, predictable name without checking if it already exists. I believe this should be the accepted answer. Very nice demonstration of the use of group commands, also. – Wildcard Mar 11 '16 at 01:59
  • I really like this. As to losing permissions (which perl -i preserves), one could stat -c %a before the rm and then chmod before the close brace. But not only are you then getting into ugly code territory, you'd also need to consider that a setgid bit on the original file may not be desirable on the new one since the group may have changed. – laubster May 03 '21 at 15:56
  • If the system crashes (or the command gets prematurely terminated) while you’re running sed -i, do you totally lose your file? (It’s my understanding that it writes output to a new (temporary) file, and then** deletes the input file and renames the output file, as simulated in terdon’s answer.) The above command is vulnerable to interruption, unless I’m overlooking something. – G-Man Says 'Reinstate Monica' Sep 01 '22 at 21:55
  • @G-ManSays'ReinstateMonica', I've not checked all sed implementations, but yes, IIRC GNU sed tries to create a new file with the same metadata (which terdon's doesn't do, he's got missing --s which could end up being problematic as well (even ACE vulnerabiity in GNU sed), and could clobber an existing file.tmp) and replace the original in the end (but breaks symlinks IIRC). perl -i last I checked (many years ago, it might have changed since), behaves like my rm+cat, and yes could lose data. But then again, all sorts of things can be lost upon system crash. – Stéphane Chazelas Sep 02 '22 at 05:51
  • @G-ManSays'ReinstateMonica', from strace, it looks like perl now behaves like GNU sed and writes to a new temp file. It does copy some metadata (which my cat+rm doesn't even try to do either) – Stéphane Chazelas Sep 02 '22 at 06:03
3

FreeBSD sed, which is used on Mac OS X as well, needs the -e option after the -i switch to define & recognise the following (regex) command correctly & unambiguously.

In other words, sed -i -e ... should work with both FreeBSD & GNU sed.

More generally, omitting the backup extension after FreeBSD sed -i requires some explicit sed option or switch following the -i to avoid confusion on part of FreeBSD sed while parsing its command-line arguments.

(Note, however, that sed in-place file edits lead to file inode changes, see "In-place" editing of files).

(As a general hint, recent versions of FreeBSD sed have the -r switch to increase compatibility with GNU sed).

echo a > testfile.txt
ls -li testfile.txt
#gsed -i -e 's/a/A/' testfile.txt
#bsdsed -i 's/a/A/' testfile.txt  # does not work
bsdsed -i -e 's/a/A/' testfile.txt
ls -li testfile.txt
cat testfile.txt
carlo
  • 63
  • 9
    No, bsdsed -i -e 's/a/A/' is not in-place editing, it's editing with saving the original with a "-e" suffix (testfile.txt-e). – Stéphane Chazelas May 22 '15 at 11:27
  • 1
    note that GNU sed also supports -E (in addition to -r) for compatibility with FreeBSD. -E is likely to be specified in the next POSIX version, so we should all be forgetting about that -r non-sense and pretend it never existed. – Stéphane Chazelas May 22 '15 at 11:30
3

You can use Vim in Ex mode:

ex -s -c '1i|<!DOCTYPE html>' -c x file

  1. 1 select first line

  2. i insert text and newline

  3. x save and close

Zombo
  • 1
  • 5
  • 44
  • 63
  • There's nothing Vim-specific here. This is fully compliant with POSIX specifications for ex, except that implementations are not required to support multiple -c flags. For definite portability I would use printf '%s\n' 1i '<!DOCTYPE html>' . x | ex file – Wildcard Oct 16 '16 at 08:31
1

To emulate sed -i for a single file portably while avoiding race conditions as much as possible:

sed 'script' <<FILE >file
$(cat file)
FILE

By the way, this also handles the possible problem that sed -i introduces in that, depending on directory and file permissions, sed -i might enable a user to overwrite a file which that user does not have permissions to edit.

You might also do backups like:

sed 'script' <<FILE >file
$(tee file.old <file)
FILE
mikeserv
  • 58,310