1

Context

I am required to use a poorly designed java application that logs A LOT of information while it is running. Under standard usage, it will create 100s of MB of logs per hour.

I don't need historical logs and it currently seems that the logrotate utility can't keep up with it as it doesn't run frequently enough. The application is closed source and rotates it's own logs at around 36MB.

My Linux distribution is RHEL7.

Question

I'd like to reduce wasted space by compressing and rotating the logs.

  1. As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?
  2. Is it possible to automatically delete all files in the format of assessor-cli.X.log where X is a digit greater than... say 5 (i.e. keep only the 5 most recent logs).

Here is my attempt at a logrotate file:

# cat /etc/logrotate.d/cis_assessor
/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {
        missingok
        notifempty
        compress
        rotate 5
        size 30M

This logrotate job would need to catch the log between the size of 30MB and 36MB to actually come into effect which might only be a 10 second period. That's why I'm asking about the manual path of compressing and deleting files without logrotate.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Crypteya
  • 494
  • 1
    You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron. – Haxiel Feb 12 '19 at 05:36
  • I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though – Crypteya Feb 13 '19 at 00:41

1 Answers1

1

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {
        missingok
        notifempty
        compress
        rotate 5
}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/
  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf
Crypteya
  • 494
Evan Carroll
  • 30,763
  • 48
  • 183
  • 315
  • I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it. – Crypteya Feb 12 '19 at 02:50
  • @Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files). – Evan Carroll Feb 12 '19 at 02:57
  • 1
    Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :) – Crypteya Feb 12 '19 at 03:03