Using sed to clean up long paths in log files

Question

A script generates log files with tens of kilolines, and most lines are very long. It would be great to make this log human-friendly with a few edits.

At run time, there are helpful environment variables. Say, ${X} corresponds to /usr/projects/draco/vendors/spack.tt.developmental/spack.tt.2017-12-06.with-flags/opt/spack/cray-CNL-haswell/intel-17.0.4/gsl-2.4-zagfxq3p5iorh7viw4gc24bwzlx2tyk7

My goal is to use the environment variables to find lines like

Leaving directory /usr/projects/draco/vendors/spack.tt.developmental/spack.tt.2017-12-06.with-flags/opt/spack/cray-CNL-haswell/intel-17.0.4/gsl-2.4-zagfxq3p5iorh7viw4gc24bwzlx2tyk7

and supplant them with a convenient placeholder, here gsl-install

Leaving directory gsl-install

My efforts include an unsuccessful syntax like this

sed -i -e 's/${X}/gsl-install/g' tedious.log

How can I provide sed an environment variable and have corresponding strings replaced?

$ uname -a
Darwin ***.***.*** 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64

Where did that gsl-install come from? I don't see it in the origin string. — igal, Dec 07 '17 at 00:42
Not sure where you are getting gsl-install from, but you could quickly convert it to Leaving directory gsl-2.4-zagfxq3p5iorh7viw4gc24bwzlx2tyk7 — Tigger, Dec 07 '17 at 00:42
@Tigger: Great observation. There is an issue left out of the problem statement that precludes using your insight. I want to get rid off the darn hashes to allow me to use diff across these log files. — dantopa, Dec 07 '17 at 00:54

PiedPiper · Accepted Answer · 2017-12-07T01:03:29.197

3

You have two problems

your environment variable contains '/' characters
you are containing your sed argument in single quotes so ${X} is passed unsubstituted

You can use any character as a sed command delimiter. Choose something not likely to be in your search string. And use double quotes around your sed command..
For example:

sed -i -e "s;${X};gsl-install;g" tedious.log

edited Dec 07 '17 at 01:03

answered Dec 07 '17 at 01:00

PiedPiper

944

I have seen ; used as the delimiter as file names can still contain _ might be something to add to the answer. – thebtm Dec 07 '17 at 01:02
@PiedPiper: great answer and explanation. Works as promised. – dantopa Dec 07 '17 at 04:52

thebtm · Answer 2 · 2017-12-07T01:10:58.593

1

To use bash variables in a sed script, you need to break out of single quote, add the variable inside double quotes and go back into the single quote for sed.

Example below:

DAY1=$(date -d "+1days" +%m%d%Y)
DAY2=$(date -d "+2days" +%m%d%Y)
DAY3=$(date -d "+3days" +%m%d%Y)
sed -n -E -e '/\s\s\s*'"($DAY1|$DAY2|$DAY3)"'/p' $file
#or
sed -n -E -e ';\s\s\s*'"($DAY1|$DAY2|$DAY3)"';p' $file

edited Dec 07 '17 at 01:10

answered Dec 07 '17 at 00:44

thebtm

1,395

2

. . . but see What characters do I need to escape when using sed in a sh script? - especially since the variable in the OP's case includes forward slash path delimiters – steeldriver Dec 07 '17 at 00:51
1

I was trying to point the in the right direction. – thebtm Dec 07 '17 at 00:55
@steeldriver: Yes. Those forward slash delimiters are the crux of the problem. The post is basically: use an environment variable for a path to edit a log file. – dantopa Dec 07 '17 at 00:56
1

@thebtm: Good post. It adds to my knowledge. Yet it is a solution for a slightly different problem. – dantopa Dec 07 '17 at 00:57

Using sed to clean up long paths in log files

2 Answers2