Shell - Customize the color of each line of a log file based on a pattern

Question

I would like to change the way I display a big (~6000 lines) log file using wim, less or whatever, in order to simplify the problem checking.

I'd like to highlight some line of the log based on a patter (i.e. error, warning, info...) and/or hide some others.

Which tools could I use? Do I just need a shell script? It's important that after the process, I can read the output using less, vim, ... to perform search operations!

Edit: a little piece of the log:

2016/10/25 12:19:24.403355 INFO <ServiceManager.cpp#2614 TID#3> Security object has NOT been parsed
2016/10/25 12:19:24.403369 INFO <ServiceManager.cpp#1263 TID#3> Service object sequence started
2016/10/25 12:19:24.403372 DBG <ServiceManager.cpp#1276 TID#3> preinvoke succeeded

grep has a --color option you could work with. echo can also output colored text with -e option, see http://misc.flogisoft.com/bash/tip_colors_and_formatting . — Valentin B., Oct 25 '16 at 10:53
Also might I suggest that you edit your question and add a sample of said log file, seeing the formatting will help design an accurate solution. — Valentin B., Oct 25 '16 at 11:10
Thanks. How can I read the output using less, vim or another editor?? — Peppe, Oct 25 '16 at 11:40
See also: https://stackoverflow.com/questions/981601/colorized-grep-viewing-the-entire-file-with-highlighted-matches/981831#981831 — Melroy van den Berg, Mar 20 '19 at 16:56

score 7 · Accepted Answer · answered Oct 25 '16 at 12:19

7

I would recommend a shell script, based on awk like Valentin B. solution:

$ cat colorize
awk '
function color(c,s) {
        printf("\033[%dm%s\033[0m\n",30+c,s)
}
/error/ {color(1,$0);next}
/success/ {color(2,$0);next}
/warning/ {color(3,$0);next}
/INFO/ {color(4,$0);next}
/DBG/ {color(5,$0);next}
{print}
' $1

In order to be able to interactively view the colorized output, I would use less in raw mode, e.g.:

colorize mylog.txt | less -R

answered Oct 25 '16 at 12:19

jlliagre

61,204

Yes defining a function in your awk script seems appropriate to avoid being redundant. Also thanks for the raw mode in less, that is great ! Works with my solution too by piping the output. – Valentin B. Oct 25 '16 at 12:26
I didn't know that awk allows to use function like this!!! Thank you – Peppe Oct 26 '16 at 07:15
EDIT: See also: https://stackoverflow.com/questions/981601/colorized-grep-viewing-the-entire-file-with-highlighted-matches/981831#981831 What about sed command?
something like .. sed /INFO/s/^/<begin_color>/ and something with /$/<end_color>. Where begin would be\033[...and end will be\033[0m`.
– Melroy van den Berg Mar 20 '19 at 16:54

score 6 · Answer 2 · edited Apr 13 '17 at 12:36

awk, as mentioned in other answers, is definitely the tool to reach for first.

But it isn't the only tool, by a long shot.

A. L. Lambert's logtool is specifically designed for post-processing log files, and has a complex (and alas badly documented) configuration system that permits one to assign a file full of regular expressions to each one of 13 colours.

cat *.log | logtool

It has the distinction of understanding logs that have TAI64N timestamps.

To that, add:

problems with colourization

Alas, a few of these tools do colourization very wrongly. They hardwire control sequences, rather than using setaf/setab and so forth from terminfo.

Also note that colourization is tricky for subtler reasons, and almost no colourization program gets it right. To get it completely right, a colourizer has to deal with automatic margins and the DEC VT pending line wrap mechanism, which I have yet to see any colourizer do. GNU grep has a fairly famous colourization bug in this area, but this is not a problem that is limited to grep.

# output INFO lines in cyan
$3 == "INFO" {
    print "\033[36m"$0"\033[0m"
    next
}

# don't display DBG lines
$3 == "DBG" {
    next
}

# output WARNING lines in bright yellow
$3 == "WARNING" {
    print "\033[1;33m"$0"\033[0m"
    next
}

# output ERROR lines in bright red
$3 == "ERROR" {
    print "\033[1;31m"$0"\033[0m"
    next
}

# If you want to skip all other lines, comment next line
{print}

Then open a clean terminal window, check the display is unlimited (you can display all 6000 lines at once) and use it this way :

$ awk -f displayLog.awk log.txt

OR Like user jlliagre suggested in his solution, pipe it to raw mode less:

$ awk -f displayLog.awk log.txt | less -R

That should do the trick ! You can fiddle with the colors, and what lines you want to display based on the sample awk code in my answer. More on color coding here.

EDIT

If you want only one word to be colored instead of a whole line (for example you want ERROR to be in red only) do this:

$3 == "ERROR" {
    $3 = "\033[1;31m"$3"\033[0m"
    print
    next
}

score -1 · Answer 4 · edited Oct 25 '16 at 10:53

-1

If you want to to highlight some line of the log based on a pattern (i.e. error, warning, info...) then please use the below command :

grep -rn "pattern" <logfile>

This command will display all the complete lines of the logfile which will have the pattern as mentioned above.

edited Oct 25 '16 at 10:53

heemayl

56,300

answered Oct 25 '16 at 10:52

Sanghamitra Choudhury

1

OP wants colored output, not just to dump line numbers, along with the matched line. Also -r is absolutely redundant when working on a single file. – heemayl Oct 25 '16 at 10:54

Shell - Customize the color of each line of a log file based on a pattern

4 Answers4

problems with colourization

Further reading

Linked