Parsing a file using unix commands

Question

I am new to Unix and I have a log file which I need to analyze. Below is my sample log file:

Container:container_e182_1234
=============================
LogType:container-localizer-syslog
Log Upload Time :Thu Jun 25 12:24:45 +0100 2020
LogLength:0
Log Contents:
LogType:stderr
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:3000
Log Contents:
20/06/25 12:19:33 INFO datasources.FileScanRDD: Reading File path: hdfs://bpaiddev/dev/data/warehouse/clean/falcon/ukc/
20/06/25 12:19:39 ERROR Exception found
java.io.Exception:Not initiated
    at.apache.java.org.Exception(132)
20/06/25 12:19:40 INFO executor.EXECUTOR: Finished task 18.0 in stage 0.0 (TID 18),18994 bytes result sent to driver.
20/06/25 12:20:41 WARN Warning as the node is accessed without started
LogType:stdout
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:
Container:container_e182_1234
LogType:container-localizer-syslog
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:
LogType:stderr
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:3000
Log Contents:
LogType:stdout
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:

Expected output

stderr
Thu Jun 25 12:24:52 +0100 2020
3000
20/06/25 12:19:39 ERROR Exception found
java.io.Exception:Not initiated
    at.apache.java.org.Exception(132)
20/06/25 12:20:41 WARN Warning as the node is accessed without started

The output must contain only the ERROR and WARN and also the other details as mentioned above

Log file:

Container:container_e182_1234
=============================
LogType:container-localizer-syslog
Log Upload Time :Thu Jun 25 12:24:45 +0100 2020
LogLength:0
Log Contents:
LogType:stderr
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:3000
Log Contents:
20/06/25 12:19:33 INFO datasources.FileScanRDD: Reading File path: hdfs://bpaiddev/dev/data/warehouse/clean/falcon/ukc/masked_data/parquet/FRAUD_CUSTOMER_INFORMATION/rcd_crt_dttm_yyyymmdd=20200523/part-0042-ed52abc2w.c000.snapp.parquet, range:0-27899, partition values :[20200523]
20/06/25 12:19:39 ERROR Exception found
java.io.Exception:Not initated
    at.apache.java.org........
20/06/25 12:19:40 INFO executor.EXECUTOR: Finished task 18.0 in stage 0.0 (TID 18),18994 bytes result sent to driver.
20/06/25 12:20:41 WARN Warning as the node is accessed without started
LogType:stdout
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:
Container:container_e182_1234
LogType:container-localizer-syslog
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:
LogType:stderr
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:3000
Log Contents:
20/06/25 12:19:33 INFO datasources.FileScanRDD: Reading File path: hdfs://bpaiddev/dev/data/warehouse/clean/falcon/ukc/masked_data/parquet/FRAUD_CUSTOMER_INFORMATION/rcd_crt_dttm_yyyymmdd=20200523/part-0042-ed52abc2w.c000.snapp.parquet, range:0-27899, partition values :[20200523]
20/06/25 12:19:34 INFO executor.EXECUTOR: Finished task 18.0 in stage 0.0 (TID 18),18994 bytes result sent to driver.
LogType:stdout
Log Upload Time :Thu Jun 25 12:24:52 +0100 2020
LogLength:0
Log Contents:

How to do this ? Please help me to solve this issue. Thanks a lot!

Answer 1

You can use sed for the same purpose with the following one-liner (assuming your file is called file):

sed -n 's/^.*LogType:\(stderr\)$/\1/p; s/^.*Log Upload Time :\(.*\)/\1/p; s/^.*LogLength:\(.*\)$/\1/p; s/^.*\(ERROR\|WARN\).*$/\0/p' file

Then you can save its output somewhere using a redirection (>) to another file.

Split up to multiple lines for easier reading:

sed -n -e 's/^.*LogType:\(stderr\)$/\1/p' \
       -e 's/^.*Log Upload Time :\(.*\)/\1/p' \
       -e 's/^.*LogLength:\(.*\)$/\1/p' \
       -e 's/^.*\(ERROR\|WARN\).*$/\0/p' file

Update

The solution above doesn't exclude blocks which are not of the 'LogType:stderr' as requested by the OP; there is required non-local infomation (not in the same line), which is not amenable to treat with sed alone.

The following script, which uses both awk and sed, (with the awk part inspired by this post), does the job:

#!/bin/bash
file=$1
awk '{
  if($0 ~ /LogType/){
    if(hold ~ /LogType:stderr/){
      print hold;
    }
    hold=$0
  }else{
    hold=hold "\n" $0
  }
}END{
  if(hold ~ /LogType:stderr/){
    print hold
  }
}' $file | sed -n -e 's/^.*LogType:\(stderr\)$/\1/p' \
                 -e 's/^.*Log Upload Time :\(.*\)/\1/p' \
                 -e 's/^.*LogLength:\(.*\)$/\1/p'       \
                 -e 's/^.*\(ERROR\|WARN\).*$/\0/p'

Answer 2

I am able to do it using a short script. The original log is contained in the file logdata.

#!/bin/bash
tmpfile="/tmp/$0.$$"
sed -n '/stderr/,/^ $/p' logdata > "$tmpfile"
sed -n 's/^.LogType:(.)/\1/p
        s/^.Log Upload Time :(.)/\1/p
        s/^.LogLength:(.*)/\1/p' "$tmpfile"
grep -E "(ERROR|WARN)" "$tmpfile"
rm "$tmpfile"

First, we extract the stderr block into a temporary file. Then, take out the two fields and then, grep the error and warning. I was trying to connect the last two steps using tee but did not succeed.

I could do it without the temporary file by

sed -n '/stderr/,/^ *$/p' logdata | \
sed -n 's/^.*LogType:\(.*\)/\1/p
        s/^.*Log Upload Time :\(.*\)/\1/p
        s/^.*LogLength:\(.*\)/\1/p
        /ERROR/p
        /WARN/p'

Answer 3

With awk:

awk '
  /LogType:stderr/ || (p && /Log( Upload Time|Length)/){
    p=1                    # set flag for stderr block
    sub(/^[^:]+:/, "")     # replace content before `:` including `:`
    print                  # print (modified) line
  }
  p && / (WARN|ERROR) /{ 
    sub(/^[^0-9]*/, "")    # remove unknown prefix
    print
  }  
  /LogType:stdout/{ exit } # exit the script
' file

Answer 4

Using GNU sed and utilizing it's extended regex mode.

sed -Ee '
  /LogType:stderr/,/^\s*$/!d
  /Log Contents:/,/^\s*$/!{
    s/^[^:]*://;b
  }
  /\s(ERROR|WARN)\s/!d
' logfile

Explanation:

• We partition the file into range (log type to blank line) and then subdivide each range into (pre log contents and post)

• In the subrange pre block, strip away till the first colon character. But don't print it yet since we don't know as of now whether an error or warning is present in the post block of the subrange. So we hold it in the hold space.

• When we reach the post block in the subrange we detect the error or warning lines. Then retrieve the hold and print it now.

Results:

stderr
Thu Jun 25 12:24:52 +0100 2020
3000
20/06/25 12:19:39 ERROR Exception found
20/06/25 12:20:41 WARN Warning as the node is accessed without started

In case you need also the line numbers of the error / warning messages, then use the below sed commands which are modified from above:

sed -Ee '
  /LogType:stderr/,/^\s*$/!d
  /Log Contents:/,/^\s*$/!{
    s/^[^:]*://;b
  }
  /\s(ERROR|WARN)\s/!d
  p;=;d
' logfile |
sed -Ee '/\s(ERROR|WARN)\s/N;s/\n/ on line #/'

You can use other tools like awk n perl to do this job also: Note : remove trailing spaces from blank lines first.

awk '
  BEGIN {
    RS = "\n\n"              
    FS = "\nLog Contents:\n" 
    OFS = "\n"               
    ORS = OFS                
    spc = "[[:blank:]]" 
    str = "(ERROR|WARN)" 
    pat = spc str spc 
  }
  /^LogType:stderr/ &&
  NF == 2 {
    p = $1; gsub(/(^|\n)[^:]+:/, "\n", p);sub(/./, "", p) 
    N = split($2, a, /\n/)
    print p
    for ( i=1; i<=N; i++ ) 
      if ( a[i] ~ pat ) 
        print a[i]
  }
' logfile

perl -F'/^Log\hContents:$/m,$_,2' -00 -ne '
  next if ! /\ALogType:stderr$/m;
  (my $pre = $F[0])=~ s/.*?://gm;
  my $post = join "\n",
    grep { /\s(?:ERROR|WARN)/ }
    split /\n/, $F[1];
  print($pre,$post);
' logfile