0

There are a large variety of questions floating around the 'net from people who want to watch the output of dmesg in real time. All the suggestions and answers involve using dmesg -w, tail, journalctl -f, configuring systemd and/or syslog to output dmesg info to a specific console, etc.

I'm stuck in an unusual situation: I too need to view dmesg (kernel debug) output in real time, but on a machine with frozen userspace. (See details below)

If I fire up a kernel with a very barebones init=/bin/bash, Alt+SysRq+h will print the magic SysRq help text immediately, even though no syslog or similar process is running. This example proves that the kernel can print dmesg info straight onto the screen without any sort of help from userspace.

Unfortunately, a component of my Arch Linux system is eating messages printed by the kernel before they get to the screen, even though I have console=tty1 loglevel=9 in my kernel commandline. Pressing Alt+SysRq+h while at tty1 appears to do nothing until I manually go fishing with dmesg | tail.

(Edit:) A comment suggested using setterm to enable messages. I was surprised to see this caused no appreciable effect.

I'm guessing systemd's journald needs reconfiguring. I have no idea where to start.

Details/context:

My laptop (an old but still useful ThinkPad T400) sometimes arbitrarily decides to lock up on resume from suspend. ACPI wakeup works, resuming video and the backlight works, the keyboard works, switching between consoles (as long as I carefully avoid the one X is on) works, even scrolling up the tty works... but userspace remains frozen.

My understanding of sleep is that...

# echo mem > /sys/power/state
(...)
t400 kernel: Freezing user space processes ... (elapsed 0.047 seconds) done.
(...)
t400 kernel: smpboot: CPU 1 is now offline
<system is asleep>
<I open the lid!>
t400 kernel: ACPI: Low-level resume complete
(...)
t400 kernel: Restarting tasks ... done.
(...)
# echo ohi
ohi
# _

...for quite a large amount of the sleep->resume process, userspace/tasks are frozen so nothing interferes with the suspend->sleep->wakeup->resume cycle.

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
i336_
  • 1,017
  • You might want to rewrite the question without systemd sarcasm. You may hate it (I couldn't say) but this doesn't make for a good question. – Stephen Harris Jan 31 '19 at 02:05
  • systemd certainly unifies a lot of otherwise distro-specific management tasks. But in many cases it does this in ways that aren't universally intuitive. I share a small amount of the frustration that systemd's introduction imposed on sysadmins anywhere due to the huge cognitive load of relearning that it required. – i336_ Jan 31 '19 at 02:41
  • This being said, my comment was an attempt to share this frustration while being dramatic/lighthearted and take the mickey out of things a bit. – i336_ Jan 31 '19 at 02:42
  • Oh, trust me, I'm not a systemd fan. It breaks 20+ years of expected behaviour and my bug reports have been closed "who cares?". But that's not my point; Unix SE is a Q&A site, where we aim to create a series of good questions and answers. Your joke detracts from that. I strongly suggest removing it. – Stephen Harris Jan 31 '19 at 02:50
  • Thanks for the consideration. systemd has helped me understand one of the UNIX fundamentals - that ad-hoc, duct-tape solutions that are designed "in the moment" can actually work and legitimately solve real-world problems... and that what looks like seemingly-arbitrary design doesn't always have to produce PHP, when architects know what they're doing! Unfortunately, I have been taught this because of systemd's refusal to take responsibility for the thousands of tiny edge-cases that their product has eliminated any viable alternatives for. – i336_ Jan 31 '19 at 03:08
  • Seems they want Linux in their image, not Linux in the people's image. Which I thought was the point. Bureaucracy is the worst kind of drug... heh. Anyway, ; this was a nice conversation and helped me figure out/articulate those points for the first time. I also edited my question to be a bit more objective (it used to look like this). – i336_ Jan 31 '19 at 03:10
  • The unix philosophy is "small tools doing small things". But there was also an implicit Postel's Principle "Be conservative in what you do, be liberal in what you accept from others" in that tooling. systemd kinda follows the first (a gazillion processes all doing something) but definitely breaks the latter. But, yeah, this isn't the place for this rant :-) – Stephen Harris Jan 31 '19 at 03:13
  • And, yes, I think the revised question is a lot more objective :-) – Stephen Harris Jan 31 '19 at 03:16
  • I have been Debian without systemd for years... – Rui F Ribeiro Jan 31 '19 at 06:02

2 Answers2

1

systemd's systemd-sysctl.service service disables all "Magic SysRq" functions except for s (sync). You need to provide your own overriding setting somewhere in /etc/sysctl.d/ with a higher precedence.

console=tty1 only outputs to the first kernel virtual terminal, which is not necessarily the active KVT. To always output to the active KVT whatever it happens to be at the time, use console=tty0.

Further reading

JdeBP
  • 68,745
0

setterm can be used to control the console log level:

setterm -msg on -msglevel 8

Hopefully your message-eater just changes these settings once during boot, and doesn't mess with them again during suspend/resume.

  • Wow, TIL about setterm, thanks. Unfortunately... even after running it, the message-eating actually continues! I verified that the kernel was getting my Alt+SysRq+h input with dmesg -w. Now to find the next size up systemd sledgehammer... – i336_ Jan 31 '19 at 02:38