8

I've recently learned that journalctl is occupying a big chunk of my 16GB SD card (Raspberry Pi):

$ journalctl --disk-usage
Archived and active journals take up 312.1M in the file system.

I don't feel that journalctl and journald are pulling their weight in my use case for this machine. It's an old-ish RPi, and rsyslog is also running. I estimate my need for and use of journalctl is maybe "once in a blue moon". Consequently, I decided I would disable journald - which "feeds" journalctl. I assumed this would be straightforward, using systemctl to stop, or perhaps just disableing the systemd-journald.service so that it would not start on the next boot.

Before pulling the plug, I decided to do somee research. Instead of finding thousands of references that offered "how-to" advice, there were remarkably few results addressing my specific search term: "how to disable journald". Instead, the results mostly offered advice on reducing journald's resource consumption. I did find a couple of references that gave me pause:

An old thread in the ArchLinux forum suggested it was not possible to disable journald without repercussions; i.e. "Masking systemd-journald causes all kinds of dependency failures and drops you at an emergency prompt." But this post is now 10 years old...

The systemd-journald.service manual says, "stopping it [systemd-journald.service] is not recommended.". The documentation proceeds from there to discuss namespaces?

I've learned that the usual command to prevent systemd units from starting has no effect; i.e. it starts normally:

$ sudo systemctl disable systemd-journald.service
$ sudo reboot

... and after boot & login:


$ systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Fri 2022-06-03 07:30:29 UTC; 1min 59s ago
TriggeredBy: ● systemd-journald-audit.socket
             ● systemd-journald.socket
             ● systemd-journald-dev-log.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 134 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 1598)
        CPU: 820ms
     CGroup: /system.slice/systemd-journald.service
             └─134 /lib/systemd/systemd-journald


...
$

How can journald be disabled? ... or can it be disabled?

If not, why would the systemd developers force this on users? (OK, yeah - asking for an opinion, so forget that part of the question.)

Seamus
  • 2,925
  • 1
    Please try disabling all the TriggeredBy's as well (systemd-journald-audit.socket systemd-journald.socket systemd-journald-dev-log.socket). This should work. I'm not going to test this on my working PC ;-) – Artem S. Tashkinov Jun 03 '22 at 09:12
  • 1
    I'd like to configure a system (an RPi Zero) to be "read only" and minimize all CPU activity that's not necessary for what I'm doing. I'm here because I was thinking disabling system services, like logging, might be a good approach... at a minimum, an interesting exercise. Sounds like I might need to seek an embedded distro. –  Nov 29 '22 at 21:47
  • 1
    @Seamus See my answer. I just spent half a day ripping systemd's benighted journaling from a bunch of systems because it couldn't handle the load of a system running with debug logs set. Loads that caused systemd to use 3+ entire CPUs while losing messages are now handled by a single rsyslogd process that takes about 17% of a single CPU. And it doesn't lose any log messages. – Andrew Henle May 26 '23 at 16:37

3 Answers3

5

The reason not to want to eliminate journald when you are using rsyslogd is that rsyslogd can get it's information from journald, and they play very well together this way.

That is sort of contra some of the Q&A's linked in comments here. Some observations about this on a current linux system (checked this on Fedora and Debian) which is configured to have journald use only volatile (in memory) storage, with rsyslog writing actual log files:

  • /var/run/log is a directory with one subdirectory, journal. This contains files written to by systemd-journald and read by rsyslogd. Note this directory is on a tmpfs partition, hence it counts as in memory volatile storage.
  • /dev/log is a symlink to /run/systemd/journal/dev-log.

The /etc/rsyslog.conf on these systems includes this (which is not the stock rsyslog.conf):

# Input
module (
        load="imjournal"
        ...
)

Fedora and Debian differ in the stock config here; the former does use imjournal as well as imuxsock (userspace socket).1 Debian uses imuxsock and imklog (kernel log), presumably because without imjournal it needs to listen to the kernel itself. I also presume this is a bit anachronistic.

Part of my point here is that completely ripping out journald may not be impossible but it is a bad idea. It is a core component of contemporary systemd based systems. It's deficit is that it potentially uses much more disk space than rsyslog, but this is easy to configure out by using Storage=volatile and (eg) RuntimeMaxUse=64M in /etc/systemd/journald,

Having rsyslogd fed from journald is a back-end/front-end relationship; rsyslog is great because it provides plain text logging as well as much more fine grained control over what gets logged where and how, while journald provides better integration with system services.

  1. The rsyslog recommendation is to use imuxsock as imuxjournal is less efficient and may suffer from other issues; see also Andrew Henle's answer about this and comments below.
goldilocks
  • 87,661
  • 30
  • 204
  • 262
  • A well-reasoned argument - thank you. I feel like this is the correct answer, and the answer I expected as there's virtually nothing in the way of how-to-recipes explaining how journald might be removed or disabled. And since this is a comment, I'll express an opinion in closing: journald is to Linux what Siri is to Apple: You might not like it, but you're "forced" (more or less) to accept something you revile in order to get something you want or need. – Seamus Jun 05 '22 at 20:32
  • 1
    Journald could certainly use some work on that front end part for contexts where it is the only logger (which it probably is now on most linux systems). It's only features are --unit to get service specific stuff, and of course it's colourful. BTW, I added a footnote that I realized might be important if you are putting this information into practice. – goldilocks Jun 05 '22 at 21:36
  • my servers struggle with postfix on Debian stable with systemd: some messages that postfix logs correctly into journald do not get forwarded to the log-server via rsyslogd. with journalctl -feu "postfix*" they can be observed on the postfix-system while on log-server, tail -f /var/log/mail.info only provides some. I seems to me that on the postfix-server either rsyslogd or journald consider the sequences of log-messages to exhaust the burst-limits. Tests with either or both imuxsock as well as imjournal have not helped. Any hints? – luke Apr 09 '23 at 07:32
  • @luke Have you tried tinkering with the rate limit? Imuxsock has a similar feature: https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#input-rate-limiting Although it is noted there that "the journal tends to drop messages when it becomes busy instead of forwarding them to the system log socket". I'd presume that doesn't apply to the journal, although maybe that is optimistic. – goldilocks Apr 10 '23 at 15:30
  • they play very well together this way Not according to the rsyslog documentation: "Note that this module reads the journal database, what is considered a relatively performance-intense operation. As such, the performance of a configuration utilizing this module may be notably slower than when using imuxsock." and then (bolding is original to rsyslog): "Warning: Some versions of systemd journal have problems with database corruption, – Andrew Henle May 26 '23 at 16:06
  • (continued) ...which leads to the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resources get exhausted. This can be somewhat mitigated by using proper rate-limiters, but even then there are spikes of old data which are endlessly repeated. By default, ratelimiting is activated and permits to process 20,000 messages within 10 minutes, what should be well enough for most use cases. If insufficient, use the parameters described below to adjust the permitted volume. – Andrew Henle May 26 '23 at 16:07
  • (continued) ... It is strongly recommended to use this plugin only if there is hard need to do so." That bolding is from the rsyslog documentation, and that sure doesn't sound like "they play very well together at all. I just spent several hours ripping journald out of a bunch of systems because it does not play well at all, even by itself. Logging loads that took close to 300% using systemd's journal kludge are how handled by a single rsyslogd process that takes all of about 1/6th of a single CPU. – Andrew Henle May 26 '23 at 16:09
  • Hiya. I've read all that, it's on the same page as the link in my previous comment WRT "tinkering with the rate limit". The deal with corruption is I believe historical, "some versions" being ones prior to a fix, but I can't give you a reference. What I can tell you is that I've been working on a network logging/messaging app that reads from an rsyslog output socket in order to exploit it's relatively simple field markup, and that's applied on various cloud servers that use ForwardToSyslog in journald.conf... – goldilocks May 26 '23 at 17:05
  • ...and the rsyslog imuxsock module, which is what is recommended instead of the imjournal module in the same paragraph as one you've quoted above, and there have been no issues with this for me in the year or so I've been working with it. More recently I've done something using the journald C API to put stuff into SQLite databases for processing.... – goldilocks May 26 '23 at 17:05
  • Because of that, I've gotten into the issue I think you mentioned earlier WRT not all journald output making it into rsyslog. This is definitely true; for starters there are piles of metadata it keeps that aren't in the plain messages, and stuff like stderr output captured from systemd services (which appears in systemd status output)... – goldilocks May 26 '23 at 17:06
  • However, except for kernel messages and some highly integrated services like auditd that stuff does not come from applications feeding the logger; more detail about that in man systemd.journald.fields. WRT it taking 300% CPU share, methinks the real issue there is upstream with the clients doing the logging, but I suppose that is a matter of context and opinion. – goldilocks May 26 '23 at 17:06
  • @goldilocks WRT it taking 300% CPU share... When you have to find an intermittent issue with one client system among hundreds using your service, you pretty much have to log everything you do. systemd's journaling is not up to that. But a simple, single rsyslogd process doesn't even break a sweat. – Andrew Henle May 26 '23 at 17:17
  • TBH I believe applications shouldn't use syslog exclusive of anything else if they tend to do lots of logging; that might as well be in its own file somewhere (although rsyslog does provide a means to do that, I dunno how great it would be if everything required an explicit if-then action in rsyslog), beyond that it seems like abuse not use of syslog (a la auditd). But I won't claim that is a canonical opinion. – goldilocks May 28 '23 at 12:39
0

Disabling this service won't stop your device from functioning but logging will cease obviously. I see no other issues.

Artem S. Tashkinov
  • 29,025
  • 1
    "logging will cease obviously" All logging - or just journald logging? – Seamus Jun 03 '22 at 06:45
  • In modern Linux distros most services log to the system journal (/dev/log). Some don't use it but they are rare. The kernel keeps its own log in a memory buffer (dmesg) but it's limited and gets rotated. – Artem S. Tashkinov Jun 03 '22 at 06:46
  • Right - but what I'm asking is if rsyslog will continue "logging" as it does now. I don't want to stop rsyslog from logging - only journald because of its resource usage. – Seamus Jun 03 '22 at 06:49
  • rsyslog itself is a logging server but if you had it installed with journald it doesn't log anywhere but redirects its messages to journald. You need to reconfigure it to make it actually log to /var/log. – Artem S. Tashkinov Jun 03 '22 at 06:52
  • Please read this: https://unix.stackexchange.com/questions/205883/understand-logging-in-linux – Artem S. Tashkinov Jun 03 '22 at 06:55
  • Thanks - a good answer here I thought. But here's a line from that answer: Bad things happen system-wide if this program crashes, or the service is stopped. If systemd-journald.service isn't started at boot (because it's disabled), will "bad things" happen?? – Seamus Jun 03 '22 at 07:12
  • The "move to chat" nagging has begun, but one more comment: It doesn't seem that the disable systemd-journald.service does *anything at all*. When I reboot, journald is started, and systemctl status systemd-journald.service reports it is running, loaded and active?! So - I do not see how this answers my question. What are your thoughts? – Seamus Jun 03 '22 at 07:44
  • "only journald because of its resource usage": Sounds like you've found a journald bug? Journald is normally pretty OK in resource usage. Nothing to log, no load at all. The logging itself is fairly efficient. Is there maybe a problem you're trying to solve that might not actually have the "solution" of disabling a central piece of your operating system, but changing the way it works? – Marcus Müller Jun 03 '22 at 08:17
  • To be honest I see no issue with trying to disable logging altogether. There are multiple cases when it's completely redundant. – Artem S. Tashkinov Jun 03 '22 at 09:36
  • "rsyslog itself is a logging server but if you had it installed with journald it doesn't log anywhere but redirects its messages to journald" -> You have this backward. Have a look at where most of rsyslogd's open fd's point to -- files in /var/log/journal/[machine-id]/ (the exact dir may be distro dependent). And those are read to by it, not written by it; the writer is journald (see man systemd-journald FILES). – goldilocks Jun 05 '22 at 14:00
  • ...See also man journald.conf and the options to send to, not receive from, syslog. Also note that rsyslog by default is not configured to "redirect its messages to journald" (and AFAICT, can't directly). – goldilocks Jun 05 '22 at 14:47
0

How can journald be disabled?

First, this is what the Rsyslogd documenation says about the imjournal module used to get log messages from systemd's journal (bolding is original, italics are my emphasis):

Note that this module reads the journal database, what is considered a relatively performance-intense operation. As such, the performance of a configuration utilizing this module may be notably slower than when using imuxsock. The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data. Only if that structured data is needed, imjournal must be used. Otherwise, imjournal may simply be replaced by imuxsock, and we highly suggest doing so.

We suggest to check out our short presentation on rsyslog journal integration to learn more details of anticipated use cases.

Warning: Some versions of systemd journal have problems with database corruption, which leads to the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resources get exhausted. This can be somewhat mitigated by using proper rate-limiters, but even then there are spikes of old data which are endlessly repeated. By default, ratelimiting is activated and permits to process 20,000 messages within 10 minutes, what should be well enough for most use cases. If insufficient, use the parameters described below to adjust the permitted volume. It is strongly recommended to use this plugin only if there is hard need to do so.

Do you have a "hard need" to feed "structured data" to rsyslog? And go a lot slower while using significantly more CPU cycles while doing that?

Replacing systemd's journaling with straight-to-rsyslog is easy:

  1. Edit /etc/rsyslog.conf. Remove all references to imjournal and imuxsock.

  2. Replace the above lines with

    module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
    SysSock.Use="on") # Turn on message reception via local log socket;

  3. Run the following command:

    systemctl stop systemd-journal-flush.service systemd-journald.service systemd-journald-dev-log.socket systemd-journald.socket

  4. Run

    systemctl stop systemd-journal-flush.service systemd-journald.service systemd-journald-dev-log.socket systemd-journald.socket

  5. Run

    systemctl restart rsyslog

  6. Verify messages are appearing in /var/log/messages et al

You may need to tweak some of those commands - especially by disabling more systemd services.

Andrew Henle
  • 3,780
  • this is what the Rsyslogd documenation says about the imjournal module used to get log messages* from systemd's journal* -> No, that's a module used to do such, and the documentation you quote recommends instead the imuxsock module, which does not suffer from these problems and is much more the commonplace norm, I would guess (in fact I think it is the default on debian and derivations currently). – goldilocks May 26 '23 at 17:10
  • @goldilocks And without that "structured data", what does the entire systemd/journald pipeline add to logging? While it sucks up stupendous amounts of CPU cycles to do a simple thing like log data? Pitch it into a black hole where it belongs, never to bother the rest of the universe again. – Andrew Henle May 26 '23 at 17:14
  • If it isn't of interest or use to you, then you might as well not use it, of course. Conversely, though, I'd say on the vast majority of systemd based systems the journal doesn't "suck up stupendous amounts of CPU cycles", so most people might as well leave it be. The issue in the original question actually has to do with it being a storage pig, which is easy enough to limit in places where it becomes a problem. – goldilocks May 26 '23 at 17:19