How do I analyze journalctl logs outside of journalctl?

Question

I need to collect some stats from my systemd log files, like how many users pass in a certain query string parameter to my service. I know I can view this information in journalctl but I can't figure out how to do so in Go (or Python) so that I can aggregate the data. I've been looking for a library for this but can't seem to find anything and I can't figure out where the log files are actually stored.

Answer 1

You can use the Journal API to access the systemd journal programatically.

Alternatively, you might want to consider running the systemd-journal-gatewayd service on your host, which exports journal data through an HTTP server, in which case you can use an HTTP client implementation from another language to query the journal directly.

Journal API

The official Journal API is provided as part of systemd itself and is used to implement journalctl. It's a C API.

This API accesses the files written by systemd-journald under /var/log/journal or /run/log/journal directly, so you need appropriate permissions to read from those locations, which typically means you need to run your code as root.

This API also requires that you link to the systemd libraries and have those available at runtime. You need to install the libsystemd-dev package on your Ubuntu 18.04 system, in order to be able to compile and link against it.

sudo apt-get install libsystemd-dev

You can look up the man page for sd-journal(3) for a general overview of this API. For the more specific case of querying the journal, see sd_journal_next(3) (which includes a nice example with a complete simple program), also sd_journal_add_match(3) to restrict the matches you get (effectively implementing a journal query.) You'll also need sd_journal_get_data(3), sd_journal_open(3), etc. but hopefully these pointers get you started with accessing the journal directly.)

Python bindings

There are Python bindings for systemd from the official systemd project, which include bindings for the Journal API.

These modules are also available as Ubuntu 18.04 packages, which you can install with:

sudo apt-get install python3-systemd

...for Python 3 (recommended), or if you still need to use Python 2 you can use:

sudo apt-get install python-systemd

See documentation for the systemd.journal module's Reader class on the code docstrings. This class gives you read access to the journal, which is probably what you're interested on.

Go bindings

There are Go bindings for the systemd libraries in the github.com/coreos/go-systemd module.

Take a look at the sdjournal submodule, in particular the JournalReader type, for reading from the journal, including matches for filtering. You can find more about it from the source code

Journal Gatewayd

An alternative to using the Journal API to access the journal files directly is to run a service which exports them through an HTTP server. Systemd provides such a service through systemd-journal-gatewayd.

This program is available on Ubuntu 18.04 as part of the systemd-journal-remote package, so you can install and enable it using:

sudo apt-get install systemd-journal-remote
sudo systemctl enable --now systemd-journal-gatewayd

Which will export an HTTP server on port 19531, which you can use to browse and query the journal.

See more details on the man page for systemd-journal-gatewayd.service(8), which includes some examples of how to access it using curl. You can get it to export the data using its own native Journal Export Format, as JSON or as syslog-like plain text.

If you go the systemd-journal-gatewayd route, make sure you understand the security implications of exporting your log data through an HTTP server. At the very least, consider exposing the port on localhost only.

Answer 2

I think journalctl grabs a subset of /var/log/syslog, that subset being things it knows about. This is just from observation. If you run journalctl with no parameters you'll see all the logs. Exit by pressing q then run tail /var/log/syslog. You'll notice the same output, but also that the information in the syslog file will contain more than the journalctl output.

So in light of the comment below, you could ensure you have syslog installed, and just parse /var/log/syslog for what you need. You could also make your service (unclear as to what that is) write to its own log in /var/log/.