3

How to get all the log lines between two dates ranges in Linux? I tried certain commands like

1)awk '$0>=from&&$0<=to' from=\"Wed 21 Mar 14:52:08\" to=\"Wed21 Mar 
14:53:08\" /home/db2inst1/logs/tracestart.log

but it gives me only those lines which has those exact dates in them.

2) sed -n '/Wed 21 Mar 14:52:00/,/Wed 21 Mar 14:53:08/p'  
/home/db2inst1/logs/tracestart.log  /home/db2inst1/logs/traceend.log

This one gives me correct data but date(Wed 21 Mar 14:52:00) should be an exact match, otherwise there is no output for nearest time also. For example, if Wed 21 Mar 14:52:01 is the start time then there is also no output.

log file sample::

2018-04-04 11:40:46 INFO  RestAssuredService:184 - some thing.......
2018-04-04 11:40:48 INFO  RestAssuredService:199 - some thing.......
2018-04-04 11:40:48 INFO  RestAssuredService:177 -
*********invokeService is 
2018-04-04 11:40:48 INFO  ProductInfoTest:57 - Response Map::::: 
{RESPONSE_TYPE=application/json, EXPECTED_RESPONSE={
"products": [
    {
        "id": 23001,
        "type": "SHIRT",
        "description": "Mens Wear Dresses",
        "price": 850,
        "brand": "PETER_ENGLAND"
    },
    {
        "id": 23002,
        "type": "KURTI",
        "description": "Womens Wear Dresses",
        "price": 899,
        "brand": "ALLEND_SOLEY"
    }
] }, 
ACTUAL_RESPONSE=com.jayway.restassured.internal.RestAssuredResponseImpl@7d48651a}
2018-04-04 11:40:48 INFO  ProductValidator:47 - EXPECTED_RESPONSE:::: {
"products": [
    {
        "id": 23001,
        "type": "SHIRT",
        "description": "Mens Wear Dresses",
        "price": 850,
        "brand": "PETER_ENGLAND"
    },
    {
        "id": 23002,
        "type": "KURTI",
        "description": "Womens Wear Dresses",
        "price": 899,
        "brand": "ALLEND_SOLEY"
    }
] }
2018-04-04 11:40:48 ERROR ProductInfoTest:65 - Exception occured::: null
2018-04-04 11:40:48 INFO  ProductInfoStepDefinations:27 - addProductDetailsApiTest Starting::::
2018-04-04 11:40:48 INFO  ProductInfoTest:53 - getAllProductsInfo starting
2018-04-04 11:40:48 INFO  RestAssuredService:170 -
*********invokeService is starting*********
2018-04-04 11:40:48 INFO  RestAssuredService:247 - Final uri:::::: rest/market/item/info
2018-04-04 11:40:48 INFO  RestAssuredService:258 - HeaderParametersMap :::::: {Accept=application/json, Content-Type=application/json
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Irfan
  • 31
  • 1
  • 3

3 Answers3

2

If your system is using systemd, then journalctl has options for time and date ranges to output from logs.

From man journalctl:

-S, --since=, -U, --until= Start showing entries on or newer than the specified date, or on or older than the specified date, respectively. Date specifications should be of the format "2012-10-30 18:17:16". If the time part is omitted, "00:00:00" is assumed. If only the seconds component is omitted, ":00" is assumed. If the date component is omitted, the current day is assumed. Alternatively the strings "yesterday", "today", "tomorrow" are understood, which refer to 00:00:00 of the day before the current day, the current day, or the day after the current day, respectively. "now" refers to the current time. Finally, relative times may be specified, prefixed with "-" or "+", referring to times before or after the current time, respectively. For complete time and date specification, see systemd.time(7). Note that --output=short-full prints timestamps that follow precisely this format.

Combine this with the --user option and some grep to filter out system messages to cut down on the clutter. If your system does not use systemd, or your program's messages are not caught by journald, then you may need something else.

Mio Rin
  • 3,040
1

Assuming a simplistic "time" environment (no timezone conversions, no daylight savings changes), you could tell awk your date ranges in seconds-since-the-epoch, then have awk convert each date to seconds-since-the-epoch and print only lines in that range:

awk -v from=$(date -d "2018-04-04 11:40:45" +%s) \
    -v   to=$(date -d "2018-04-04 11:40:47" +%s) \
 '{ "date -d \""$1 " "$2"\" +%s" | getline s; 
    if (from <= s && s <= to) print;
  }' < input
2018-04-04 11:40:46 INFO  RestAssuredService:184 - some thing.......

It's not particularly efficient, as it calls date for every line; it could be enhanced to cache lookups, if that becomes a concern.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
0

An other approach I am using frequently for log files from Java applications is to find the line number of the first occurencce of the time stamps

FROM_DATE="Wed 21 Mar 14:52:08"
TO_DATE=""Wed 21 Mar 14:53:08"

FROM_LINE=$(grep -n -m 1 ${FROM_DATE} ${FILE} | cut -d ":" -f 1)
TO_LINE=$(grep -n -m 1 ${TO_DATE} ${FILE} | cut -d ":" -f 1)

then give out the information between, i.e. 

tail -n "${FROM_LINE}" ${FILE} | head -n $(expr $TO_LINE - $FROM_LINE)

or via

sed -n -e "${FROM_LINE},${TO_LINE} p" -e "${TO_LINE} q" ${FILE}

This will catch up stack traces, REST API content, JSON structures, etc.

For certain applications like from the Hadoop framework, I have specific scripts to work with their log files. The from Jeff mentioned approach with date I am using to calculate the time between two events.

For further information (and reference):

U880D
  • 1,146