I am trying to upload my journald logs from a server to a centralized server. I am using systemd-journal-remote and systemd-journal-upload for this. The initial sync of existing logs goes perfectly. After a while, when uploading new logs, systemd-journal-remote starts claiming that the log is corrupt.
# journalctl -u systemd-journal-remote
Dec 07 11:59:41 syslog systemd-journal-remote[1459]: /var/log/journal/remote//remote-192.168.100.85.journal: Journal header limits reached or header out-of-date, rotating
Dec 07 11:59:44 syslog systemd-journal-remote[1459]: /var/log/journal/remote//remote-192.168.100.85.journal: Journal header limits reached or header out-of-date, rotating
Dec 07 13:25:55 syslog systemd-journal-remote[1459]: File /var/log/journal/remote//remote-192.168.100.85.journal corrupted or uncleanly shut down, renaming and replacing.
Dec 07 14:14:04 syslog systemd-journal-remote[1459]: File /var/log/journal/remote//remote-192.168.100.85.journal corrupted or uncleanly shut down, renaming and replacing.
Dec 07 22:06:10 syslog systemd-journal-remote[1459]: File /var/log/journal/remote//remote-192.168.100.85.journal corrupted or uncleanly shut down, renaming and replacing.
Dec 08 07:14:29 syslog systemd-journal-remote[1459]: File /var/log/journal/remote//remote-192.168.100.85.journal corrupted or uncleanly shut down, renaming and replacing.
It moves the current file to a journal~ file.
# ls -l /var/log/journal/remote
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Dec 8 09:08 remote-192.168.100.85.journal
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 83886080 Dec 7 13:25 remote-192.168.100.85@0005ef3c05a61f38-d9e2719124309c81.journal~
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Dec 7 14:14 remote-192.168.100.85@0005ef3cb1d4ed0a-4409c128d6c7996f.journal~
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Dec 7 22:06 remote-192.168.100.85@0005ef434a3c6e81-cd917ee1470654d7.journal~
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 8388608 Dec 8 07:14 remote-192.168.100.85@0005ef4af3270982-8d69c39c835093b4.journal~
-rw-r----- 1 systemd-journal-remote systemd-journal-remote 75497472 Dec 7 11:59 remote-192.168.100.85@20cd77d5c97e47119c33de6c4ca2c0d2-0000000000000001-0005c3d7d8a4d2a4.journal
The strange part, is that these files are perfectly readable by journalctl, and verify returns PASS.
# journalctl --file="/var/log/journal/remote/*.journal~" --verify
PASS: /var/log/journal/remote/remote-192.168.100.85@0005ef4af3270982-8d69c39c835093b4.journal~
PASS: /var/log/journal/remote/remote-192.168.100.85@0005ef3c05a61f38-d9e2719124309c81.journal~
PASS: /var/log/journal/remote/remote-192.168.100.85@0005ef434a3c6e81-cd917ee1470654d7.journal~
PASS: /var/log/journal/remote/remote-192.168.100.85@0005ef3cb1d4ed0a-4409c128d6c7996f.journal~
Nothing is logged on the systemd-journal-upload server.
Both source and destination servers are running Ubuntu 20.04 (Focal) server with systemd version 245. I have tried using both http and https as transport. I also tried to upgrade the systemd-journal-remote server to Ubuntu 22.04 (Jammy) (systemd 249). All resulting in the same behaviour.
Why is it doing this? This is filling up my diskspace, as the journal~ files are always at least 8MB, but sometimes only contain 2 log lines.
