remove exact line of string defined as string variable from a text file

Question

I have a sample text file called result.txt just to demonstrate:

{HEX}.A.Bs
{HEX}.A.Bss
{HEX}.A.Bsss
{HEX}.A.Bssss

but in real application, the content of the result.txt can have any characters line by line like:

/usr/local/abc.txt
/var/tmp/
png
OIPOP()_+(_)&*#)@IOJDNU*@Utest

and I want to delete a line of string via argument in a bash script. So, in my bash script called test.sh, I have the following code:

!/bin/sh
#test.sh
result=result.txt
del="$1"
sed -i "/$del/d" $result
cat result.txt

Example 1

So to delete "{HEX}.A.Bs", I run the following script with an argument:

./test.sh "{HEX}.A.Bs"

this should delete a string called "{HEX}.A.Bs" inside result.txt but all the contents inside result.txt were deleted. The expected output should be:

{HEX}.A.Bss
{HEX}.A.Bsss
{HEX}.A.Bssss

Example 2

If I want to delete "{HEX}.A.Bss":

./test.sh "{HEX}.A.Bss"

The above example left only a string "{HEX}.A.Bs" inside result.txt. It should delete only the string "{HEX}.A.Bss" . So, the expected output from the this example 2 should yield this:

{HEX}.A.Bs
{HEX}.A.Bsss
{HEX}.A.Bssss

May I know what is wrong with the code here?

Answer 1

If you want to delete lines that match a string then use a tool that understands literal strings, e.g. with GNU awk for inplace editing (just like GNU sed):

del="$1" awk -i inplace '($0"") != ENVIRON["del"]' "$result"

Answer 2

What's wrong with your code is that you're successfully matching the substring {HEX}.A.Bss on every line.

Assuming your string contains no characters that need escaping, you could instead use

sed "/^$string\$/d" filename

I.e., anchor the pattern at both the start and end of the line.

Since the pattern does contain a character that needs escaping (., which matches a single character), it would be safer to use grep -F, or awk with a string comparison.

If you still want to use sed, make sure that all dots in the string are matched as dot and nothing else:

pattern=$( printf '%s\n' "$string" | sed 's/[.]/[.]/g' )
sed "/^$pattern\$/d" filename

---

To delete lines matching a string exactly, use

grep -v -xF -e "$string" filename >newfile && mv newfile filename

The options used with grep here are

• -v: Negates the sense of the test, i.e. all lines not matching the pattern are extracted.
• -x: Require that the whole line matches the given pattern.
• -F: Treat the pattern as a string rather than as a regular expression.
• -e: "The next argument is the pattern". This is needed in case the value $string starts with a dash.

To retain as much meta data of the original file as possible (e.g. ownership, permissions etc.), make a copy of it, apply the grep operation to the copy, and redirect the result to the original file:

cp filename filename.tmp &&
grep -v -xF -e "$string" filename.tmp >filename
rm -f filename.tmp

Answer 3

With perl (that -i non-standard option supported by a few modern sed implementations was actually borrowed from perl):

LINE_TO_REMOVE='{HEX}.A.Bs' perl -i -nle '
  print if $_ ne $ENV{LINE_TO_REMOVE}' -- "$file"

To use sed (here assuming GNU sed or compatible), you'd need first to escape the regular expression operators in the string to match (that includes the . used in your example which unescaped would match any single character) and anchor it at beginning and end with ^ and $ respectively.

string='{HEX}.A.Bs'
regexp='^'$(printf '%s\n' "$string" | sed 's:[][\\/.^$*]:\\&:g')'$'
sed -i -e "/$regexp/d" -- "$file"

(it also assumes $string and the file contain valid text).

Answer 4

I believe \b is for word boundry , and just in case if the line contains "{HEX}.A.Bs anotherword" or "{HEX}.A.Bs filename" ( which is string contains space and "filename" as a filename ) , it will also delete that.

so if you want to delete only specific "{HEX}.A.Bs" its better to use:

$ sed '|^'"$del"'$|d' $result  

"$del" contains the string exactly you passed and single quotes ' in between them tell to match it exactly as it is and not as a pattern.

Perl:

perl -pe "s|^\Q$del\E$||g;" $result | sed '/^$/d'   

\Q => quote (disable) pattern metacharacters until \E

and if you want in-place edit ( change/edit the file directly ) you could use :

$ perl -pe "s|^\Q$del\E$||g;" $result | sed '/^$/d' > tmpfile && mv tmpfile $result  

And if you want to use awk :

$ awk -v d=$del  '$0 != d' $result > tmp && mv tmp $result

If you want to use grep:

$ grep -vxF "$del" $result > tmp && mv tmp $result

-F, --fixed-strings, --fixed-regexp Interpret PATTERN as a list of fixed strings
-v, --invert-match Invert the sense of matching, to select non-matching lines. (-v is specified by POSIX.)
-x, --line-regexp Select only those matches that exactly match the whole line. (-x is specified by POSIX.)

Answer 5

Assuming your result.txt file content is exactly as provided by you, and your search strings are as per your examples...

$ cat test.sh
#!/bin/sh
result="result.txt"
del="$1"
sed '/^'"$del"'$/d' "$result"
$

$ ./test.sh '{HEX}.A.Bs'
{HEX}.A.Bss
{HEX}.A.Bsss
{HEX}.A.Bssss
$ ./test.sh '{HEX}.A.Bss'
{HEX}.A.Bs
{HEX}.A.Bsss
{HEX}.A.Bssss
$ ./test.sh '{HEX}.A.Bsss'
{HEX}.A.Bs
{HEX}.A.Bss
{HEX}.A.Bssss
$ ./test.sh '{HEX}.A.Bssss'
{HEX}.A.Bs
{HEX}.A.Bss
{HEX}.A.Bsss
$

Answer 6

Here is one way if you don't want to be bothered with all this escaping business:

$ printf '%s\n' "$del" |
  sed -Ene '1h;G;/^(.*)\n\1$/!P' \
- "$result" > tmp && mv tmp "$result"

Assuming tmp file is not the same as result file. And no newline(s) in del variable. Gnu sed is being used.

We have essentially transferred the del variable in the hold and then we make a strict check on the current line with this hold content. For no match we print. Repeat for the whole file.

Answer 7

Sorry, it seems like I got this:

sed -i "/^$del\b/d" $result

Answer 8

I was looking to do something similar for some pam.d files in RedHat and kept bricking systems. But I only wanted to purge pam.d files of the word 'nullok' and NOT delete the line(s) on/in which 'nullok' was found.

The simplest method I found was to simply do a sed -i 's/texttobekilled//g' filename ,

Using the word 'nullok' and one of the pam.d files, the command looked like: sed -i 's/nullok//g' /etc/pam.d/system-auth