2

Here's an imaginary command line where "pushtolog" is a utility that accepts lines of text from a pipe and logs them into a file with a filename incorporating today's date:

tail -F mylatestlogfile.log | grep "ERROR" | pushtolog -d /tmp/myerrors/ -f yyyy-MM-dd -r errors -e log

When running on the 17th Feb 2017 the output would be logged to file:

/tmp/myerrors/errors.2017-02-17.log

At midnight on 18th Feb 2017, this file would be closed and any further output would be logged to file:

/tmp/myerrors/errors.2017-02-18.log

The switchover would happen automatically. Performance is fairly irrelevant due to a low volume of relevant log lines, so in the worst case it would be fine for the utility to check the date every time it wrote a log line.

The GNU/Linux server that this will run on has a minimal set of utilities installed. It has Python 2 installed, and I could write a utility myself. Before taking that step, it's sensible to ask whether there is a tool that will already the job, or whether there's an idiomatic alternative way of doing this.

logrotate is available, but I'd rather avoid doing a rotation every because it seems "simpler" to have it all done in step.

multilog seems to tackle a similar problem of switching target log file, but when the log file is full rather than when the date changes. I don't have root access and can't run a package manager to get hold of it; I haven't investigated whether I could build it from source successfully using the permissions that I have. If it's clearly the tool for the job then I'll pursue this option.

David B
  • 203
  • 3
  • 6
  • 1
    What about -f $(date +"%Y-%m-%d") – Lambert Feb 17 '17 at 11:11
  • I assume that's a comment about how I'd make the "pushtolog"'s -f argument contain the current date? Unfortunately the "pushtolog" command doesn't exist! I'm looking for a suggestion of how to implement the whole last piece of the pipeline; to replace this with something real: "pushtolog -d /tmp/myerrors/ -f yyyy-MM-dd -r errors -e log"

    In my imaginary example, I set the date format to year-month-day using the -f switch. I don't want that to be evaluated once today, because it needs to take a different value tomorrow. The utility needs the date format, not the current date.

     – David B Feb 17 '17 at 11:17
  • And what is the meaning for -r and -e? – Lambert Feb 17 '17 at 11:28
  • Sorry that it isn't clear. -r is "root of the filename", which in my example is "errors" and -e is "extension", which in my example is "log". Putting together the root, date and extension, you get "errors.2017-02-17.log" as the output filename. – David B Feb 17 '17 at 11:30
  • I think I'll end up doing something with AWK, like in this example: http://stackoverflow.com/questions/13077631/is-it-possible-to-print-different-lines-to-different-output-files-using-awk – David B Feb 17 '17 at 11:33
  • See http://unix.stackexchange.com/questions/340591/ for a more specific question. – JdeBP Feb 17 '17 at 12:58

2 Answers2

1

I used Awk's ability to evaluate construct a filename including today's date on each evaluation (for each line that orignates from the log) and ended up with:

tail -F mylatestlogfile.log | grep "ERROR" | awk '{print >> "/tmp/myerrors/errors." strftime("%Y-%m-%d") ".log"}'
David B
  • 203
  • 3
  • 6
0

You can try the following code, as a bash script:

#!/bin/bash

while getopts "d:f:e:r:a" opt; do
    case $opt in
        f) FDATE=$(awk -v fmt="$OPTARG" '{print strftime(fmt)}'<<<DUMMY) ;;
        r) PREFIX="$OPTARG" ;;
        e) SUFFIX="$OPTARG" ;;
        d) DEST="$OPTARG" ;;
        a) APPEND=true ;;
    esac
done
if [ ! $APPEND ]
then
    > ${DEST}/${PREFIX}.${FDATE}.${SUFFIX}
fi
while read line; do echo ${line} >> ${DEST}/${PREFIX}.${FDATE}.${SUFFIX}; done

You can also surround the body of the script with:

pushtolog()
{
...
}

and source it.

I also included a flag to be able to append to existing data. It should be easy to change the script to default to append and overwrite when a specific option is set.

Note 1: The date format should conform to strftime (see man strftime) so use "%Y-%m-%d"

Note 2: that variable APPEND is not actually a boolean, I just imagine that it is. Perhaps it is more nice to use APPEND="true" and if [ -z "$APPEND" ]

Lambert
  • 12,680