Possible to speed this while read bash script up?

Question

I'm still VERY new to the whole linux bash scripting and I've come up with this little bit of code, unfortunately, its execution time is kinda long for a small file (193 KiB)

real 0m7.234s user 0m6.772s sys 0m3.486s

If you could take a look at it, pass some improvements or tips, would be appreciated!

#!/bin/bash
#
while read line; do
    RNAME=$(echo $line | grep -w "ET CINS Active Threat Intelligence Poor Reputation" | sed 's/^.*\(ET CINS Active Threat Intelligence Poor Reputation.*\)/\1/g' | sed 's/".*//') 
    RSID=$(echo $line | grep -w "ET CINS Active Threat Intelligence Poor Reputation" | grep -o "sid:.*" | awk '{print $1}' | rev | cut -c 2- | rev | cut -c 5-)
    echo $line | grep -w "ET CINS Active Threat Intelligence Poor Reputation" | awk '{print  "'"$RSID"'" " " "\"[;][)]\"" " " "\"" "; fwsam: src[either], 1 hour;)\"; # " "'"$RNAME"'" }'#  >> /tmp/snortsam-rules.txt
    echo $line | grep -w "ET CINS Active Threat Intelligence Poor Reputation" | awk '{print  "'"$RSID"'" " " "\"\\(msg:\"\" \"(msg:\"[SNORTSAM] \"; # " "'"$RNAME"'" }'  >> /tmp/snortsam-rules.txt
done < /etc/snort.d/rules/emerging-threats/emerging-ciarmy.rules

Contents of the Input file:

alert tcp [1.11.244.148,1.119.129.16,1.119.133.214,1.119.144.196,1.163.25.190,1.170.159.97,1.173.65.136,1.177.142.203,1.177.220.170,1.177.251.214,1.180.189.18,1.180.208.131,1.180.208.132,1.180.233.23,1.186.176.220,1.186.176.246,1.186.220.92,1.186.235.187,1.192.123.218,1.192.145.246,1.202.225.53,1.202.65.39,1.215.230.46,1.221.225.138,1.228.102.199,1.230.44.160,1.232.113.151,1.234.1.70,1.234.4.14,1.239.35.88,1.24.156.110,1.245.107.90,1.247.184.111,1.251.177.206,1.253.135.172,1.254.20.189,1.254.47.75,1.28.202.11,1.28.202.16,1.31.87.35,1.32.200.123,1.32.216.88,1.32.47.74,1.33.73.100,1.34.113.192,1.34.158.177,1.34.209.99,1.34.21.27,1.34.2.152,1.34.28.244] any -> $HOME_NET any (msg:"ET CINS Active Threat Intelligence Poor Reputation IP TCP group 1"; flags:S; reference:url,www.cinsscore.com; reference:url,www.networkcloaking.com/cins; threshold: type limit, track by_src, seconds 3600, count 1; classtype:misc-attack; sid:2403300; rev:40471;)

Output Sample Code:

2403300 "[;][)]" "; fwsam: src, 1 hour;)"; # ET CINS Active Threat Intelligence Poor Reputation IP TCP group 1

Basically, what i want to do is, take some parts of the input contents, and save them into a file, like the above output sample code, if that makes sense?

You can eliminate the useless uses of echo with here strings. awk can search for strings as well so grep piped to awk is a useless use of grep. — jesse_b, May 11 '18 at 23:30
For each line read, you call 14 programs. Are you sure there's reason to complain? Please add your sample to your question to make it readable. — Gerard H. Pille, May 12 '18 at 00:29
@GerardH.Pille Not sure i understand how I'm calling 14 programs. but I've updated my post — dobzsock, May 12 '18 at 00:52
Related: https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice — Kusalananda, May 12 '18 at 06:27

score 2 · Accepted Answer · answered May 12 '18 at 06:09

2

Note:

Reading individual lines from a file and the running multiple programs on each line multiple times is going be to excruciatingly slow.
See above for each time you do echo $line | ....

Most of this can be done by awk itself:

awk -v fmt1='%s "[;][)]" "; fwsam: src, 1 hour;)"; # %s\n' -v fmt2='%s "\\(msg:"" "(msg:"[SNORTSAM] "; # %s\n' '/ET CINS Active Threat Intelligence Poor Reputation/ {
  rname = gensub(/.*(ET CINS Active Threat Intelligence Poor Reputation [^"]*).*/, "\\1", 1)
  rsid = gensub(/.*ET CINS Active Threat Intelligence Poor Reputation.*sid:(.*); .*/, "\\1", 1)
  printf fmt1, rsid, rname
  printf fmt2, rsid, rname
}' input-file

Notes:

Instead of doing complex escaping to get a particular output format, use printf with a format string. I Used two awk variables defined externally (-v fmt1=..., -v fmt2=...) as the format strings, thereby avoiding a lot of quoting.
You have used echo ... | grep ... | awk to print only if the grep pattern is matched. This is easily done in awk itself by using /pattern/ { action } to run actions only on lines matching the pattern.
The rev | cut | rev | cut and sed ... | sed commands' work is done easily enough in regex by keeping only the group of characters you need.

answered May 12 '18 at 06:09

muru

72,889

Isn't rname just set to the string ET CINS...? – Kusalananda May 12 '18 at 06:26
1

@Kusalananda there seems to be a variable part after the fixed string we see in the post, up to a double quote character. – muru May 12 '18 at 06:29
Ah, ok. That may well be the case. A bit difficult to see with only a sing line of input given. – Kusalananda May 12 '18 at 06:33
It's IP TCP group 1 in the example input (the part goes: ... any (msg:"ET CINS Active Threat Intelligence Poor Reputation IP TCP group 1"; flags:S; ...) – muru May 12 '18 at 06:35
@muru Thank you SO much! and thank you for so much info! i was reading into awk more, before your post, and i did see the /pattern/ { action } but couldn't wrap my head around it. After spending 40~ mins reading over the code you put forward, I'm getting the idea of how it works! again, thank you!
real 0m0.090s | user 0m0.076s | sys 0m0.006s compared to real 0m7.234s | user 0m6.772s | sys 0m3.486s
– dobzsock May 12 '18 at 20:48

Possible to speed this while read bash script up?

1 Answers1