Read n lines at once in a loop using bash script

Question

I have a file in this format:

Start tracing process 17134 17136 17137 17138 (/usr/sbin/nginx)...                                                                                    

[1465461983910514] pid:17136 GET /
    total: 244us, accept() ~ header-read: 19us, rewrite: 9us, pre-access: 13us, access: 9us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983911223] pid:17136 GET /sfi9876
    total: 110us, accept() ~ header-read: 12us, rewrite: 13us, pre-access: 20us, access: 13us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983911949] pid:17136 GET /lpt9
    total: 127us, accept() ~ header-read: 14us, rewrite: 18us, pre-access: 28us, access: 20us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983912121] pid:17136 GET /~sfi9876
    total: 127us, accept() ~ header-read: 11us, rewrite: 24us, pre-access: 37us, access: 26us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us

I want to sort to this log using the total time taken in every request so that it looks like this:

Start tracing process 17134 17136 17137 17138 (/usr/sbin/nginx)...                                                                                    

[1465461983911223] pid:17136 GET /sfi9876
    total: 110us, accept() ~ header-read: 12us, rewrite: 13us, pre-access: 20us, access: 13us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983911949] pid:17136 GET /lpt9
    total: 127us, accept() ~ header-read: 14us, rewrite: 18us, pre-access: 28us, access: 20us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983912121] pid:17136 GET /~sfi9876
    total: 127us, accept() ~ header-read: 11us, rewrite: 24us, pre-access: 37us, access: 26us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us
[1465461983910514] pid:17136 GET /
    total: 244us, accept() ~ header-read: 19us, rewrite: 9us, pre-access: 13us, access: 9us, content: 137us
    upstream: connect=0us, time-to-first-byte=0us, read=0us

So, please suggest some way to do this using a bash script.

Is each entry really taking 3 lines or it's just the way it looks in the copy? — Julie Pelletier, Jun 09 '16 at 11:47
I expect there is a very simple way to do this with a Perl one-liner. I definitely would recommend not using a shell loop; see Why is using a shell loop to process text considered bad practice? — Wildcard, Jun 09 '16 at 19:47

score 0 · Answer 1 · answered Jun 09 '16 at 12:04

You could loop until EOF, reading each entry (3 lines), extract the times with something like headTime=$(echo $line|cut -d':' -f2|cut -du) and add them together like total=$(expr $headTime + $rewriteTime + ...).

Then you could output each group of 3 lines as a single line starting with the total (for sorting) and put a line separator (ex: |) so that you can split them up easily after the sort. Then you'd obviously sort the temporary file, splitting the lines back: sort -n tempFile|sed "s/|/\n/g".

The main problem I have is how to loop on multiple lines. I know how to loop line by line, but how do I loop on 3 lines at a time? — tom, Jun 09 '16 at 16:51

score 0 · Accepted Answer · answered Jun 09 '16 at 19:33

0

You can use a normal while read loop, and then use read twice within the loop to read the 2nd and 3rd lines in each set.

while read -r line1; do
    read -r line2
    read -r line3
    ...
done

answered Jun 09 '16 at 19:33

Barmar

9,927

I am going to go ahead and select this as answer because that's exactly what I asked in this question, although right now I am doing this task in python because the actual task has a bit more work which is best handled using dictionaries. – tom Jun 10 '16 at 07:11

Read n lines at once in a loop using bash script

2 Answers2