8

I'm really struggling to understand how logrotate works when running a command within a shell file of my own, and how it doesn't.

The command in question is:

rclone -L -vv --log-file "/home/mike/tmp/qqq.log"  sync "/media/mike/W10 D drive/My Documents/"  remote:MyDocuments_M17A_from_Linux

What I would like:

I would like this qqq.log file to be checked quite often, say every 10 min., to see whether it has exceeded a given size, say 1 MB. And if it has, to rotate the file. rclone with the -vv option produces copious output, deliberately (i.e. to try to understand to get logrotate working for this use case).

There's a moderately helpful tutorial here, but it still leaves me in the dark:

  1. How are you meant to configure rotation of logs from your own non-system processes? Are you meant to stick lines at the bottom of /etc/logrotate.conf? This is what I have done (under "#system-specific logs may be configured here"):

    /home/mike/tmp/qqq.log {
      notifempty
      size 1M
      daily
      create 0664 root root
      rotate 3
    }
    

    later: I now realise you can also put individual files in /etc/logrotate.d - no great complexity there.

  2. Is it the case that by default logrotate only runs once a day, by virtue of the file /etc/cron.daily/logrotate?

  3. Does that mean that if I want much more frequent checking I should perhaps set up a cron job to do this, running (in this example) every 10 minutes?

  4. Are you meant to run logrotate <config file> as root? I ask this question because I have tried both as root and as user. User didn't seem to work.

A link to a beginner's guide to this sort of setup, which can't be that uncommon, would be helpful. I have searched but most of what I've found doesn't give you a step-by-step guide.

muru
  • 72,889
mike rodent
  • 1,132
  • 1
    Of course, one can challenge the whole idea of using logrotate in the first place, when (with suitable application of things like /dev/stderr if necessary) one can just do the task directly in the script. https://unix.stackexchange.com/a/340631/5132 https://unix.stackexchange.com/a/326166/5132 – JdeBP Jan 17 '20 at 12:39
  • Thanks, I'm at best a low-intermediate Linuxer. At this stage I'd just like to find a way of configuring logrotate so my disks don't get jammed up with monstrous log files. There's also the "right tools for the job" argument, which I'm not qualified to answer... although I assume logrotate is probably meant to do what it says on the tin. – mike rodent Jan 17 '20 at 12:54
  • Plus the other thing is, with rclone you set an option, --log-file... the process just carries on regardless, spewing out log messages. Do those other tools you reference run as cron jobs? (As I presume logrotate does...)? – mike rodent Jan 17 '20 at 12:59

2 Answers2

10

logrotate can be run as an ordinary user, without administrative privileges, to rotate logs for that user.

# Create a local per-user configuration file
cat >.logrotate.conf <<'X'
/home/mike/tmp/qqq.log {
    notifempty
    missingok
    size 1M
    rotate 3
}
X

# Run logrotate with that configuration file
/usr/sbin/logrotate -v -s .logrotate.state .logrotate.conf

I've removed your daily criterion because you wanted purely a size-based check, and this would have limited any possible action to just once a day (the first time each day that logrotate is run, as it happens). I've replaced create with missingok so that it's up to your actual rclone job to create the output file rather than logrotate.

Then put the logrotate command into your user's crontab file:

# Capture any existing crontab entries
crontab -l >.crontab

# Append ours to the list
echo '0 * * * * /usr/sbin/logrotate -s .logrotate.state .logrotate.conf >>.crontab.log 2>&1' >>.crontab

# Reload crontab
crontab .crontab

Using this example, output from the command will be written to .crontab.log, and you'll probably want a logrotate entry to cycle or reset it monthly.

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • Thanks, this is very helpful. In fact I have just managed to run sudo logrotate <config file> at the CLI... and my qqq.log file became qqq.log.1 (and a new qqq.log was created). Hurrah. So a cron job is indeed needed for this use case: you'd be surprised how few "logrotate how to guides" out there actually make that clear! – mike rodent Jan 17 '20 at 13:41
  • Pleased to hear that. You don't need the sudo with my example – Chris Davies Jan 17 '20 at 14:08
  • 1
    I've tried this and am a little puzzled: changing the date back one day in the .conf file did the trick... but without this the size (I also tried maxSize) directive on its own had no effect (naturally I ran it several times). ... AHA... if I omit any time frequency (daily, etc.) it does do a rollover... – mike rodent Jan 18 '20 at 11:59
4

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

logrotate on the other hand, checks the logfiles when it is run, and usually systems are setup to run logrotate (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/logrotate.d and yes, it usually runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.

[edit 2] @mike rodent

I'm not familiar with rclone, whether it writes to stdout or stderr and what if anything it writes to either, but there are ways around this, even if the --log-file=/dev/stdout didn't work.

This StackOverflow Q/Answer, amongst others explains redirection and piping. But to summarise, it's possible to redirect stderr only or merged with stdout into a pipe.

Bash example:

Merge stderr with stdout into pipe

FirstCommand 2>&1 | OtherCommand

Only stderr into pipe, stdout to otherfile (which could be /dev/null or even just a hyphen - which closes stdout).

FirstCommand 2>&1 1>otherfile.log | otherCommand

These is examples use the older bash syntax, bash v4 has a modern less verbose variant.

X Tian
  • 10,463
  • This is a great idea, but you probably need -n 3 to stop rotatelogs creating files with a seconds-since-the-epoch datetime suffix, e.g. aaa.log.1579268239. – Chris Davies Jan 17 '20 at 13:38
  • You've got me scratching my head over this answer: "works by reading stdin" ... "rotatelogs ... rotatelogs on the other hand". But I'll have a look at the utility, thanks. – mike rodent Jan 17 '20 at 13:45
  • @roaima Tks, Yes -n number to limit no of log files and have a circular rotation (which also alters naming), -p <script or program> to add a post-processor which could possible archive or compress are also useful options to use. – X Tian Jan 17 '20 at 15:35
  • Thanks. Much clearer now. Yes, I was going to doubt this, due to the fact that by default rclone prints to sterr, I think, but then I saw your last line: being a low-level Linuxer I didn't know you could do that to direct to stdout. I've learnt something new. – mike rodent Jan 17 '20 at 16:45
  • 1
    This is the sort of challenge to the whole idea of using logrotate in the first place that I mentioned in a question comment, even down to the use of /dev/stdout//dev/stderr. There are in fact a lot of tools for doing this. https://unix.stackexchange.com/a/340631/5132 https://unix.stackexchange.com/a/326166/5132 – JdeBP Jan 17 '20 at 17:19
  • 1
    @JdeBP if you provided answers instead of spending your time criticising we could all benefit – Chris Davies Jan 17 '20 at 18:14
  • That's a very silly thing to say, given that I quite evidently provided answers to this. And you are the one criticizing. I on the other hand merely pointed out that this was the very thing that I was talking about. – JdeBP Jan 17 '20 at 19:00
  • 2
    @JdeBP you provided a comment (and didn't feel like answering my further questions). Any reason why you don't in fact make your own explicit answer, singling out your tool of choice and giving clear ideas of how it works and why it's good? I mean, given that you seem to be an expert in this area. – mike rodent Jan 17 '20 at 21:18
  • This is interesting as an alternative approach to the problem ... but if you're capturing stdin doesn't that mean you will capture all stdin, from whatever source? – mike rodent Jan 18 '20 at 12:16
  • @mikerodent, see edit2 in answer above. – X Tian Jan 18 '20 at 13:00
  • Thanks. I've got this running now (rclone -vv sync "/media/mike/W10 D drive/My Documents/" remote:MyDocuments_M17A_from_Linux 2>&1 | rotatelogs /home/mike/tmp/qqq.log 200K common). It's wonderful. It appears to be working asynchronously, meaning that rotation happens during the rclone process. Clearly this makes it, on the face of it, a superior solution to logrotate for this sort of use case. Hopefully you'll appreciate that I don't intend to change the accepted answer: because my question was quite specifically about how to get logrotate to work. – mike rodent Jan 18 '20 at 13:48
  • One caveat: the function of the -n option (e.g. -n 5) makes things a bit unusual: the active log file is then not necessarily qqq.log. It may be qqq.log.1 / .2 / .3 or .4: find this out by sorting by "last modified". – mike rodent Jan 18 '20 at 14:01