Virtual server stops responding and uses 100% CPU, how to diagnose?

Question

We are facing weird issue with a CentOS 7.4 virtual server (VMware ESXi), the server suddenly gets 100% usage in few seconds resulting in ssh won't work and a specific program running is not responding. The only solution is to force restart the server through vSphere. We can't figure out the source of such high usage so my question is, how to diagnose such sudden high usage? Is there a way to log some processes information to be investigated after restart?

EDIT: ssh was completely not working, in fact I did ssh -vvv and the verbose output got stuck at "Connecting to host port 22" and the shell never returns. It seems to be waiting for the connection to be established. Regarding ping, our IT engineers block ICMP traffic to the servers hence I cannot verify ping operation.

Answer 1

When I faced similar issue I've created a small script like this (it writes every second the date and list of process running with its CPU and RAM usage):

#!/bin/sh
while true
  do 
    date
    ps faux
    sleep 1
  done >> /a/log/file

and I let it run as a background program. This would help you to understand what and when a process become crazy.

After that you will have to look at /var/log/messages and other logs (the one from the crazy program maybe) to identify precisely your issue.
You could also install atsar which provide amazing binary log with a log of stats such as IO, network activity, cpu and so on...

/!\ Warning:
If your run this script for long enough the log could become big. Place the log file in a spot where there's enough disk space or this could become a major problem.

/!\ Warning 2:
I have no idea what is your esxi setup. However, if for any reason disk become laggy esxi wide; this could cause major latency and a dramatic CPU usage on the virtual machine if this one rely on IO.

edit 2:

As mention by @sourcejedi you could add sync to the script to ensure the log will be written in case of hard reboot (I'm not sure it's necessary though but two security nests are better than one:

#!/bin/sh
    LOGFILE="a/log/file"
    echo "" > $LOGFILE
    while true
      do 
        date
        ps faux
        sync $LOGFILE
        sleep 1
      done >> $LOGFILE

Answer 2

On a system with N CPUs, you can run N userspace threads using 100% of a CPU each. However when you try to use ssh, the kernel would give ssh a "fair" share of CPU time and allow you to log in.

When you see 100% CPU usage in VMware, and ssh is not responding, you might have a busy-loop inside the kernel.

Make sure your server is not running a graphical interface on its local console. You want to be in text mode, so you can see any messages printed by the kernel. Now read:

• Documentation/lockup-watchdogs.txt
• Documentation/sysctl/kernel.txt: nmi_watchdog
• Documentation/sysctl/kernel.txt: soft_watchdog

The docs say the hard lockup detector, aka NMI watchdog, is enabled by default. Except it is disabled when the kernel runs in a VM. So in this case, the default is to only detect soft lockups.

    /*
     * Hard lockup detection is enabled by default. Disable it, as guests
     * can get false positives too easily, for example if the host is
     * overcommitted.
     */
    hardlockup_detector_disable();

-- arch/x86/kernel/kvm.c: kvm_guest_init()

I am confused by the rationale and history of this. I don't know why it considers the soft-lockup detector to be "safer" than the hard-lockup detector. Also the original change was justified with a different reason. "The guest PMU is still flushing out bugs. The idea is once the KVM PMU is stable enough, the default switches to hardlockup enabled by default". Lastly, it is mentioned that on some versions of hypervisors, it might not be possible to enable the NMI watchdog at all.

Assuming you are not massively over-committing CPUs in your hypervisor, you could see if it's possible to enable the NMI watchdog as well. You can use the sysctl linked above, or the doc for the sysctl says you could also use the kernel boot option nmi_watchdog=1.

Then test that you can see messages printed by the kernel:

Is the "local console" logged or at least persistent, so that you would have noticed if a kernel crash was printed on it? It really should be, but I am not sure familiar with how vSphere etc. would work if you used an emulated serial console. If you are just using an emulated video display, then it will be persistent already.

This VMWare article appears to rely on the same assumption.

Make sure you have not disabled console logging. Run this command:

sudo sh -c "echo '<3>test' >/dev/kmsg"

It should show "test" on the console.

If this is an emulated video display, part of a crash message might scroll off the top of the screen. And if the kernel has crashed, then you cannot use shift+PageUp to scroll up. In principle, it can be more useful to have an emulated serial console which implements scrollback.

For kernel crashes, there are a couple of other crash dump suggestions in the VMWare link above.

-- Debian Stretch VM becomes quasi-unresponsive every few days

Most of the other instructions in the linked answer are about hung task messages. You will not necessarily see those if you have a lockup.

That said, it does also mention sysrq. sysrq+L might be an alternative way to get useful information, if you have a soft lockup. That will generate a kernel backtrace for each CPU. But the root cause might only be visible for one of the CPUs, so you really need to be able to capture quite a lot of messages there. It would be best if you have a serial console. If you have a video console, shift+PageUp might just work, assuming you don't have too many CPUs.