8

I've got a legacy app that is generating a huge lot of logs message. I'd like to be able to rotate the file every minute (or so) but without loosing / truncating any message. The app is not able to change of file by itself so I need to do it externally. Surfing the net I've seen a lot of suggestion to use cp followed by a cat /dev/null but since my app is generating almost a continuous flow of messages some messages are lost during the process.

Do you have any ideas ?

Cerber
  • 235

2 Answers2

6

Make the log file a FIFO (man mkfifo) and put a process on the reading side which separates its input into files of limited size.

mkfifo /path/to/logfifo.app_xy
split ... </path/to/logfifo.app_xy &
/bad/app
Hauke Laging
  • 90,279
  • Nice start but I'm looking for a complete solution. Not that yours is bad, but I was very surpised that in 2013 the minimum rotation time for every one is daily like in logrotate. Everybody is talking about big data, auditing and so on, but I feel like the only one needing much frequent rotation ... – Cerber May 29 '13 at 16:54
  • @Cerber "complete solution" meaning what? logrotate can rotate by size, too, if I remember correctly. – Hauke Laging May 29 '13 at 17:32
  • ooooooops, you meant THE split unix core command. Sorry @HaukeLaging I've read your answer too fast. I'll check that on my app – Cerber May 29 '13 at 17:46
  • Hi Hauke Laging, I went on with a couple of tests but I can't get it right : split is not fit for time-based split, so I've thought to 'crontab' it. But I started with a simpler solution : I wrote a perl script that outputs in the fifo each numbers from 0 to 10000 with a 100ms sleep. I ran this in background and did a head on it to gather the first bytes just to check that two head calls would not break the sequence ... unfortunalty I got a [1]+ Broken pipe perl script.pl >log at the first head. Therefore I think split would not work. What am I missing ? – Cerber Jun 04 '13 at 18:14
  • @Cerber You probably did something like head log; head log. Indeed, the first head closes the FIFO then. You need a wrapper around the processes which read from the FIFO: {head log; head log;} or exec 3<log; head <&3; head <&3 – Hauke Laging Jun 04 '13 at 20:07
  • 1
    @Cerber I guess someone listened and now logrotate is hourly, and according to this answer you can set it to whatever interval you want, with a file in cron.d with this */10 * * * * root /usr/sbin/logrotate /etc/logrotate.conf – michaelok Nov 15 '18 at 21:06
  • Yes, time flies :) indeed logrotate has greatly improved – Cerber Nov 16 '18 at 15:40
2

Maybe the logrotate tool can handle this.

Spack
  • 1,997
  • 3
    logrotate needs at least a little support from the app, due to the way files work in Unix. If the log file is renamed without notifying the app in some way, the app will continue to write to the renamed file, rather than starting to write to a new file with the old name. man 8 logrotate and note the "postrotate" command in the examples. – Jander May 29 '13 at 08:43
  • beside, logrotate minimum is daily ... which is very far from what I want to achieve – Cerber May 29 '13 at 16:50
  • Actually you can force rotation depending on the file size. But it is true that you need either the application's help or the restart it for the log to be rotated. – Spack May 29 '13 at 17:50
  • 1
    @Cerber you can make it hourly – Sentient May 05 '19 at 18:03
  • 3
    Hi @Sentient, it was not the case back in 2013 (or at least neither of us figured it out). Thanks for the comment! for the others : hourly Log files are rotated every hour. Note that usually logrotate is configured to be run by cron daily. You have to change this configuration and run logrotate hourly to be able to really rotate logs hourly. – Cerber Jun 24 '19 at 15:31
  • To add to other historical comment about postrotate, there is an option called copytruncate which ensures the older file is not renamed, but content is copied to new file and old one is truncated. With this, no extra support or restart is needed from application. - https://access.redhat.com/solutions/646903 – Pavan Kumar May 17 '22 at 18:59