1

I have a directory that contains multiple directories each full of log files. Every log file is named in the format of YYYY-MM-DD.log. So it would be /path/to/logs/tokyo/2017-06-29.log, /path/to/logs/london/2017-06-29.log, etc.

A new log is opened every day at midnight with the new date, and the old ones will never be written to again.

There are literally thousands of these logs going back years, and while they're just text files they are starting to take up a non-negligible amount of space. So what I'm trying to do is write a script that looks in all of these log directories, finds any dated before today, and bzips them. Here is what I have so far:

#!/bin/sh
LOGDATE=$(date +%Y-%m-%d)
LOGPATH="/path/to/logs"

for i in $(ls $LOGPATH/*/*.log); do
    if [ "$i" != "$LOGDATE.log" ];then
        tar cfjv $i.tar.bz2 $i
        rm $i
    else
    fi
done

There are two problems I'm running into. First, the $(ls ...) in the for loop errors out saying the argument list is too long. Second, in the tar statement, I want the bzipped output file to be in the directory where the log file was, but it seems to instead drop them in the directory the script is being run from, and I can't find a way to make the loop use the correct directory for every matching file it encounters.

Presumably this would be better handled with a well written find and xargs command, but I've been unable to work one out that does what I need it to do.

The end goal of this script is to find all log files in the log directory structure, check that it is not today's log, bzip it in place, and delete the text log. It would be run once immediately to take care of all the old logs, and then added to crontab to run just after midnight every day.

What do I need to do to correct my script? Or, is there better way altogether?

The only real restriction is that it's a freebsd server so it needs to use either sh or csh, and use freebsd tools (meaning for example bsd's find instead of GNU find which has more extensive options).

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
Kefka
  • 477
  • 1
    It is generally a really bad idea to parse the output of ls. You should probably look into either using find or simple shell globbing to get your list of files to process. Extensive further reading on the subject can be found here. – DopeGhoti Jun 29 '17 at 16:40

2 Answers2

2

This could be made much simpler by combining find with the fact that you don't need to make a tarball if you're only compressing one file:

find $LOGPATH/ -name \*.log -mtime 1 -execdir bzip2 "{}" \;

If your find doesn't have -mtime, you can fake it by using touch to create a file (e. g. $DATEFILE) with a specific last-modified time of, say, "midnight today", and then use:

find $LOGPATH/ -name \*.log -not -newer /path/to/$DATEFILE -execdir bzip2 "{}" \;
rm /path/to/"$DATEFILE"
DopeGhoti
  • 76,081
  • I had tried something along those lines but I couldn't get -mtime to return exactly what I needed. In your example it only returns files modified today and yesterday. GNU find has a function to return results 'modified yesterday or earlier' but that feature doesn't exist in FreeBSD's find. – Kefka Jun 29 '17 at 16:50
  • @Matt is installing the findutils port illegal for some reason? – thrig Jun 29 '17 at 17:02
  • @thrig No, but I prefer to stick with stock tools whenever possible. – Kefka Jun 29 '17 at 17:04
  • You should probably use -mtime +0 rather than the unnecessarily precise -mtime 1. Or is this unavailable in FreeBSD's find? – Chris Davies Jul 09 '17 at 00:10
0

Surely the simplest way is to utilise the tool that FreeBSD already has for this - newsyslog.

Create a /etc/newsyslog.conf file, or edit the existing one. You give the name of the file (you can use wildcards although you may have to have multiple lines for different directories), how many back copies you want to keep (a very big nuumber in your case), when to roll over the file (there's a flag to allow you to do that yourself), which compression system you want to use, etc.

Use newsyslog -FvC path-to-your-log-file to start it off the first time.

Incidentally, you can just put the relevant lines in
/usr/local/etc/newsyslog.conf.d, in a file with any name you like. This makes remembering it after upgrades a little easier.

Bob Eager
  • 3,600