308

I have created a simple systemd service file for a custom application. The application works well when I run it manually, but my CPU gets maxed out when I run it with systemd.

I'm trying do track down where my problem is, but I don't know where to find the output (or how to configure systemd to put the output somewhere).

Here is my service file:

[Unit]
Description=Syncs files with a server when they change
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/filesync-client --port 2500
WorkingDirectory=/usr/local/lib/node_modules/filesync-client
Restart=always


[Install]
WantedBy=multi-user.target

Throughout the application, I output to stdout and stderr.

How can I read the output of my daemon?

Edit:

I found man systemd.exec, which mentioned the StandardOutput= option, but I'm not sure how to use it. From the man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, syslog, kmsg, kmsg+console, syslog+console or socket.

If set to inherit the file descriptor of standard input is duplicated for standard output. If set to null standard output will be connected to /dev/null, i.e. everything written to it will be lost. If set to tty standard output will be connected to a tty (as configured via TTYPath=, see below). If the TTY is used for output only the executed process will not become the controlling process of the terminal, and will not fail or wait for other processes to release the terminal. syslog connects standard output to the syslog(3) system logger. kmsg connects it with the kernel log buffer which is accessible via dmesg(1). syslog+console and kmsg+console work similarly but copy the output to the system console as well. socket connects standard output to a socket from socket activation, semantics are similar to the respective option of StandardInput=. This setting defaults to inherit.

Does this mean that these are my only options? I would like, for example, to put output in /dev/shm or something. I suppose I could use a Unix domain socket and write a simple listener, but this seems a little unnecessary.

I just need this for debugging, and I'll probably end up removing most of the logs and change the output to syslog.

Community
  • 1
beatgammit
  • 7,583
  • Have you tried checking /var/log/syslog for output? Most systems will log stuff into /var/log/ so I'd start by checking there. You can use grep to search for text if you know the output: grep "my output" /var/log should do the trick. – sbtkd85 Sep 09 '11 at 19:10
  • @sbtkd85 - Well, I don't have /var/log/syslog, but /var/log/messages does the trick. The problem is, according to the logs, my daemon crashes on start, yet I can tell that it is still running because it has an HTTP server, and I can query it. It seems the rest of the logs are getting lost... – beatgammit Sep 10 '11 at 03:50
  • Why not try setting StandardOutput=tty so you can see what is happening when you launch your daemon. It should output the terminal (you may have to use ttyS0 or similar to get the output on your screen). – sbtkd85 Sep 12 '11 at 13:46
  • 5
    Shouldn't standard IO redirection operators work in this context. Something like ExecStart=/usr/local/bin/filesync-client --port 2500 2>/tmp/filesync.log – Deepak Mittal Apr 10 '12 at 18:30
  • What is actually abusing your CPU? Is it systemd, your service or system (e.g. by spawning new copies of the service because systemd went crazy)? – peterph Nov 13 '12 at 16:03
  • @peterph - It was a problem with my service, which is why I wanted to look at the output. The problem is now solved though. – beatgammit Nov 14 '12 at 19:59
  • Appears that it is possible now with newer versions of systemd, see https://stackoverflow.com/a/48052152/32453 – rogerdpack May 18 '19 at 02:12

2 Answers2

308

Update

As mikemaccana notes, the systemd journal is now the standard logging device for most distros. To view the stdout and stderr of a systemd unit use the journalctl command.

sudo journalctl -u [unit]

Original Answer

By default stdout and stderr of a systemd unit are sent to syslog.

If you're using the full systemd, this will be accesible via journalctl. On Fedora, it should be /var/log/messages but syslog will put it where your rules say.

Due to the date of the post, and assuming most people that are exposed to systemd are via fedora, you were probably hit by the bug described here: https://bugzilla.redhat.com/show_bug.cgi?id=754938 It has a good explanation of how it all works too =) (This was a bug in selinux-policy that caused error messages to not be logged, and was fixed in selinux-policy-3.10.0-58.fc16)

Matt
  • 8,991
  • 5
    Note that using the standard logging mechanism like this will not create persistent logs by default. To do that, you'll need to create /var/log/journal, and then run sudo systemctl restart systemd-journald – mlissner May 08 '15 at 02:21
  • 1
    what syslog facility and priority? – jrwren Jul 07 '15 at 16:31
  • 2
    This worked for me: StandardOutput=syslog+console and StandardError=syslog+console after that all output from my unit appeared in journalctl. The default setting was wrong apparently. (Such as DefaultStandardOutput in /etc/systemd/system.conf) – gregn3 Jul 22 '15 at 12:23
  • @gregn3 That does not help for me. It just keeps outputting stderr, no matter what I do :( – Joakim Apr 12 '16 at 22:29
  • @Joakim I have no idea right now... (but I would first check that the issue is not with the unit itself, that it works correctly when started from the command line (as a normal script)... then I would check systemd and journalctl settings... good luck :) ) – gregn3 Apr 13 '16 at 13:11
  • 2
    -f was helpful for me. Followed the log as changes occur (use case was following minecraft server which was running as a daemon) – blaughw Nov 21 '16 at 04:55
  • 3
    This is driving me crazy... On a standard Debian stretch journalctl won't show me any of the standard output whatsoever. I even use /usr/bin/stdbuf -oL <cmd> and an explicit StandardOutput=journal. Still nothing. – jlh Dec 18 '18 at 09:14
  • 1
    use sudo journalctl -eu [unit] to go the end of the logging – ignacio Jun 09 '20 at 08:26
  • I'm very confused, on OpenSUSE Leap 15.2 journalctl -u example.service does not show the stdout at all. It shows the times when the service was started and stopped but no stdout or stderr. – Parker Gibson Feb 24 '21 at 06:20
  • "Note that using the standard logging mechanism like this will not create persistent logs by default" what is does this means? – Viren Nov 04 '22 at 09:14
  • @Viren: It's clear what it means (logs disappear on reboot) but is it right or not? – Joshua Jan 17 '24 at 14:39
127

Shorter, simpler, non-legacy answer:

sudo journalctl -u [unitfile]

Where [unitfile] is the systemd .service name. Eg, to see messages from myapp.service, 

sudo journalctl --unit=myapp

To follow logs in real time: 

sudo journalctl -f -u myapp
mikemaccana
  • 1,793
  • 1
  • 12
  • 16