0

I want to use syslog(3) to write application logs to a central recorder. The application runs on a collection of computers on the same subnet.

Not interested (at present) in capturing system (non-application) logs to the central server. Not interested in having the application logs captured into the systemd journal of each local system.

Aim is to have syslog(3) with facility of "local0"-"local7" skip the systemd journal, and instead be written to the recorder via UDP (port 514 likely).

This seems to be the inverse of the usual question (how to redirect application logs into syslog).

Currently puzzling out what makes sense for the Centos systemd/rsyslog lashup. To be clear, I expect to have this figured out in the next few days.

Putting up the question for the next guy, who like me searched for an answer, without immediate luck. :)

Paulo Tomé
  • 3,782
Preston L. Bannister
  • 203
  • You might want to look at imfile module which can be used to forward logs. https://rsyslog-doc.readthedocs.io/en/latest/configuration/modules/imfile.html – annahri Jan 08 '20 at 21:37
  • Thanks, but that is a rather different case. :)

    The imfile is reading a file via rsyslog and injecting into the system syslog. In my case the applications are not writing into a file, but instead will be calling syslog(3) directly.

     – Preston L. Bannister Jan 08 '20 at 22:06
  • Should add I found interesting answers to this: https://unix.stackexchange.com/questions/205883/understand-logging-in-linux – Preston L. Bannister Jan 08 '20 at 22:14

1 Answers1

1

Aim is to have syslog() with facility of "local0"-"local7" skip the systemd journal

You will not achieve it. The syslog() library function logs to the systemd journal. The server listening on the well-known /dev/log socket that the library talks to is not rsyslogd. It is systemd-journald. rsyslogd is attached to the other side of systemd-journald, and the library function is not talking to it. No amount of fiddling with rsyslog configuration will alter this.

The only way to not send stuff via systemd-journald is to use some other route to the destination, not the syslog() library function talking to the well known socket

Ironically, that could be the proper and simpler place for application logs: standard error. Connect standard error up to a pipe that talks to some program that transports it over the network to a remote log collector.

My preferred setup for this sort of stuff is to run the applications as "main" services under a daemontools-family service manager.

  • The applications log to standard error.
  • The usual daemontools-style service management has log services whose standard inputs are connected via pipes to the applications' standard errors. Service management handles setting these pipes up between the "main" and "log" services.
  • These log services run cyclog, multilog, s6-log, svlogd, or some such and dump logs into local (strictly size-capped, automatically rotated, insulated from the application) log directories.
  • A further service watches these log directories and feeds new information across the network to a centralized log server running logstash or some such. My export-to-rsyslog tool was designed for this.

The benefits to this are:

  • that there are local, immediately accessible, per-application, individual (not mixed with, or flooded by, other things), recent logs;
  • that flaky network connectivity to the log server does not exert back-pressure upon applications (as it would in a more direct model where application standard error talks over the network, or directly to a process that immediately relays over the network);
  • that the network transport is entirely non-invasive with respect to the local logging, can be set up later (when volume grows enough to need something like a centralized logstash), and can be reconfigured (adding/removing what applications' logs are sent over the network) without affecting it; and
  • that in extremis one can even arrange to replay recent log data to a flaky/accidentally wiped log server.

Further reading

JdeBP
  • 68,745