Write shell script to analysis log file

Question

The log file is as below:-

Source=Mobile
IP=189.23.45.01
STATUS=SUCCESS
TIME=10 sec

Source=Desktop
IP=189.23.34.23
STATUS=FAIL
TIME=101 sec

Source=Mobile
IP=189.23.34.23
STATUS=FAIL
TIME=29 sec

File keep going so on.

Questions:

Find IP where status is FAIL?
Find Ave time taken by all request where status is "success"?
List how many logins were via Mobile and how much time did it took ?

If the entries of the log are consistent (four lines, third being status), then you can list only the failed ones by utilising greps context line controls. e.g. grep -B2 -A1 STATUS=FAIL — steeling, Jan 05 '19 at 17:51
@Machine Could you please clarify what you mean with "...and how much time did it took"? Do you want to know the overall time involved in Mobile-logins, successful and failed? An average? A overal or average calculation for only successful or only failed mobile logins? — ozzy, Jan 05 '19 at 18:16

ozzy · Answer 1 · 2019-01-05T18:05:25.633

1

In an unpolished version, a Bash-script could look like this, assuming that your data is contained in datafile:

#!/bin/bash

printf "IPs where status is fail:\n"
grep -z -oP 'IP=\K.*\n(?=STATUS=FAIL)' datafile

printf "Avg time taken by all requests where status is 'success':\n"
grep -z -oP 'STATUS=SUCCESS\nTIME=\K\d+' datafile | \
  awk '{ total += $1; count++ } END { print ( count == 0 ? "NaN" : total/count); }'

printf "Number of logins (successful and failed) via Mobile:\n"
grep -c 'Source=Mobile' datafile

A brief elucidation:

Q2) Calculation of the Average time: the grep command extracts the time values (which are assumed to be all in seconds). These values are piped into the awk command, which calculates their average, and then prints that average.

edited Jan 05 '19 at 18:05

answered Jan 05 '19 at 17:55

ozzy

845

@Machine What did you mean with "...and how time did they sent" ? – ozzy Jan 05 '19 at 17:56
can u explain what exactly u doing in second command to find Avg? – Machine Jan 05 '19 at 17:59
@Machine Of course, if you elucidate your question :-) – ozzy Jan 05 '19 at 18:01
1

@steeldriver Sorry. It seems I spoiled the fun... Perhaps we should leave it to Machine to figure out what the commands do precisely. Specific questions will be answered then... – ozzy Jan 05 '19 at 18:07

score 1 · Answer 2 · answered Jan 05 '19 at 18:01

1

You're really asking 3 questions - I'll get you started with the first one, and you should make an effort to solve the other two yourself using the same basic structure (there are plenty of examples on this site for using Awk to do numerical processing such as averaging):

Use Awk in paragraph mode (by unsetting the record separator, RS), splitting the record into fields using = and \n (newline):

$ awk -vRS= -F'[=\n]' '/STATUS=FAIL/{print $4}' file.log
189.23.34.23
189.23.34.23

answered Jan 05 '19 at 18:01

steeldriver

81,074

@Steekdriver, i need your help to understanding what u meant by unsetting the record separator, RS. – Machine Jan 08 '19 at 13:31
@Machine see for example 4.9 Multiple-Line Records in the GNU Awk User's Guide – steeldriver Jan 08 '19 at 13:42
@Steekdriver - awk -vRS= -F'[=\n]' '/STATUS=FAIL/ sum+=$8 {print sum}' test.log
When doing this , i get error as below awk: /STATUS=FAIL/ sum+=$8 {print sum} awk: ^ syntax error – Machine Mar 22 '19 at 05:25
@Steekdriver , below is what i get when i run : awk -F '=\n' '{/Source=Mobile/a[$2] += $8} END{for (i in a) print i, a[i]}' test.log awk: {/Source=Mobile/a[$2] += $8} END{for (i in a) print i, a[i]} awk: ^ syntax error – Machine Mar 22 '19 at 05:36
@Machine you seem to be misunderstanding the basic rule-action structure of Awk progams - have a look at the examples here – steeldriver Mar 22 '19 at 05:46
n=0;while read i;do n=$((n+i));done < <(awk -vRS= -F '[=\n]' '/Source=Mobile/ {print $8}' test.log);echo "MOBILE:-"$n; sh: syntax error near unexpected token `<' – Machine Mar 22 '19 at 05:52
@Steekdriver, any hints you can throw ? – Machine Mar 22 '19 at 06:19
Comments really aren't appropriate for this. It looks like you are trying to use process substitution in a shell that doesn't support it - see What is the bash '<(file contents)' syntax called? – steeldriver Mar 22 '19 at 06:34
Wow!!! Grt , Thanks a lot . This worked in Bash . n=0;while read i;do n=$((n+i));done < <(awk -vRS= -F '[=\n]' '/Source=Mobile/ {print $8}' test.log);echo "MOBILE:-"$n; MOBILE:-39 – Machine Mar 22 '19 at 10:20

score -2 · Answer 3 · edited Aug 06 '21 at 06:55

read -p "Lets Give File Name , placed in the same dir: " file ; 
echo " Ques : Find the Number of IP which failed "
echo "Ans: "

cat "${file}".txt | grep -i STATUS=FAIL -B1 | grep -i IP | awk -F '=' '{print $NF}'
#cat "${file}".txt | grep -i STATUS=SUCCESS -A1 | grep -i Time | awk -F '=' '{print $NF}' &> clear2.txt

echo "Ques : Find the avg of success time"
echo "Ans : "

cat "${file}".txt | grep -i STATUS=SUCCESS -A1 | grep -i Time | awk -F '=' '{print $2}' | awk '{print $1}' &> clear2.txt
avgtime=0
i=0
for x in `cat clear2.txt`
do  
    i=$(($i + 1))
    avgtime=$(($avgtime +$x))
    echo "avg time after ${i} iteration is :${avgtime}"
    y=$(($x))
done
#echo "${x}"
#echo "${y}"
avgtime=$(($avgtime/$i))

echo "THe avg time is : ${avgtime}"

echo "Ques : What is the Number of time mobile was tried to login"

echo "Ans :"

cat testfile.txt | grep -i Mobile | wc -l

Welcome to the site, and thank you for your contribution. Please keep in mind, though, that using shell loops for text processing is discouraged as it is very inefficient when compared to standard Unix/Linux tools such as awk. — AdminBee, Aug 06 '21 at 07:03

Write shell script to analysis log file

3 Answers3