1

I feel dumb as a sack of hammers asking this here, but it's been a long day and I simply CANNOT figure out what I'm doing wrong here.

I have a file; we'll call it textfile.txt. It's contents are (in this representative, but contrived example) as follows:

FILE CONTENTS

SOME=1
TRIED="THEIR BEST AT"
INSERTING='A VALUE'
BE=4

Now, assume I have two variables $KEY and $VAL, representing, oh, we'll say the key and value I want to update in the file (Brief aside: I CAN guarantee that neither will ever contain a quote ["] or apostrophe [']):

NEW VALUES

KEY="TRIED"
VAL="THEIR HAND AT"

Okay, great. So now I fire off what seems like it should be a bog-standard regex replace (note that I'm attempting to keep the replaced value enclosed in whatever optional delimiters it has encircling it presently):

sed -E "s/$KEY=([\"']?).*([\"']?)/$KEY=\1$VAL\2/g" textfile.txt > textfile.txt

EXPECTED RESULT

SOME=1
TRIED="THEIR HAND AT"
INSERTING='A VALUE'
BE=4

ACTUAL RESULT

(an empty file)

Okay, fine, so how about I target a NEW file instead of the one I'm reading from?

sed -E "s/$KEY=([\"']?).*([\"']?)/$KEY=\1$VAL\2/g" textfile.txt > textfile2.txt

NEW RESULT

SOME=1
TRIED="THEIR HAND AT
INSERTING='A VALUE'
BE=4

Capital!

...except now JUST the second delimiter (the double-quote trailing AT) is missing around the value. I can re-use \1, which works, but I feel this is brittle enough without figuring out where I'm dropping the ball here.

So... Question:

  1. Why does my first attempt purge the file outright?
  2. Why does my second omit the latter delimiter?

Note that I'm totally not married to this approach, and am fine with going another route, but if someone could PLEASE explain those two points AS WELL, I'd be much obliged. I woulda sworn up and down I knew RegEx if you'd asked me yesterday, but it's been a minute since I've used it in Shell.

I'm running GNU bash, version 5.2.12 on a 2021 MacBook M Chip Pro on Ventura 13.0 if that helps.

MC68020
  • 7,981
NerdyDeeds
  • 121
  • 5
  • It works for me. sed (GNU sed) 4.8 – Gilles Quénot Dec 05 '22 at 22:49
  • Really? When you cat textfile2.txt you get both? Oh, gods, now I'm REALLY baffled. – NerdyDeeds Dec 05 '22 at 22:51
  • @Quasímodo - Bro, you know how it is. I've been beating my head against a wall for close on an hour over this stupidity. And I KNOW it's gonna be something daft like that. I can't imagine (both files were created IN Bash), but... just to be sure, I ran both through dos2unix. No luck. Great suggestion, though! Didn't even occur to me. – NerdyDeeds Dec 05 '22 at 22:59
  • 2
    Empty file is explained here https://unix.stackexchange.com/a/36263 – don_crissti Dec 05 '22 at 23:02
  • I knew someone would have a reference to that part at least. Thanks @don_crissti! Although I'd swear there's a way to do that. Did I need to cat it out then pipe it into the write? – NerdyDeeds Dec 05 '22 at 23:03
  • 1
    Unless your sed supports -i here's the way to do it https://unix.stackexchange.com/a/505600 – don_crissti Dec 05 '22 at 23:05

2 Answers2

0

If you don't have -i switch and you have sponge from moreutils, you can do :

sed 's///g' file | sponge file

This tool is especially designed for that purpose

Gilles Quénot
  • 33,867
0

Okay, so this took some digging. Thanks to both don_crissti (for pointing me at a solid rationale for the output being purged) and Gilles Quenot (for sending me off down one helluva rabbit hole to try and learn about sponge).

In the end, however, the net result was as follows:

  1. The contents of the file were being purged because that's precisely the behavior that is intended when one is using the >/>> redirects. The redirects fire first, before the rest of the command, so i was wiping the file I was about to read to clean (by redirecting nothing to it), and then failing to write the results of the command to it on top of that (since both the file being read was now empty, and the redirect had already taken place regardless).

    The fix, in this case, was to briefly create a temporary file, then overwrite the source file with it immediately afterwards:

    sed -E "s/$KEY=([\"']?).*([\"']?)/$KEY=\1$VAL\2/g" textfile.txt > textfile.tmp && mv textfile.tmp textfile.txt

    This is an unfortunate consequence of MacOS using a less robust version of the sed binary that that found in GNU, and, while one can indeed brew install gsed to get GNU version, it cannot safely supplant the base version, which makes portability and future-proofing sketchy.

  2. ...but this still left me with the Case of the Vanishing String Delimiter weighing heavily on my mind. Through the course of my investigations, however, I stumbled onto an interesting factoid regarding the use of expanding variables in a sed regular expression on the Darwin version of the tool: I had to wrap the variable names in an expansion, escaping the $'s.

    So, instead of sed -E "s/$KEY=([\"']?)[...]etc. the necessary syntax was sed -E "s/\$(KEY)=([\"']?)[...]etc..

    Further exploration ultimately proved that, if I simply escaped the $ on each variable, as shown here: sed -E "s/\$KEY=([\"']?)[...]etc.

    ...I would also get the desired result, but a saltier Linux veteran than I intimated that the $(VARIABLE) syntax was safer in this case (though I've not the foggiest clue as to why).

Hope this followup helps someone else, should they find themselves similarly-stumped, and thanks again to don_crissti and Gilles Quenot for pointing me in the rightish directions! <3

NerdyDeeds
  • 121
  • 5