5

kworker process consumes 75% of one CPU. The offending kworker thread is related with ACPI stuff:

sudo cat /proc/THE_PID_OF_KWORKER_PROCESS/stack
[<ffffffff85c0c705>] acpi_ns_evaluate+0x1bc/0x23a
[<ffffffff85bffe09>] acpi_ev_asynch_execute_gpe_method+0x98/0xff
[<ffffffff85be4e30>] acpi_os_execute_deferred+0x10/0x20
[<ffffffff8588dc21>] process_one_work+0x181/0x370
[<ffffffff8588de5d>] worker_thread+0x4d/0x3a0
[<ffffffff85893f1c>] kthread+0xfc/0x130
[<ffffffff8588de10>] process_one_work+0x370/0x370
[<ffffffff85893e20>] kthread_create_on_node+0x70/0x70
[<ffffffff858791ba>] do_group_exit+0x3a/0xa0
[<ffffffff85e6a2b5>] ret_from_fork+0x25/0x30
[<ffffffffffffffff>] 0xffffffffffffffff

so I started debugging by rebooting with some acpi related kernel parameters, such as: 

acpi=off : Completely solves the high cpu usage, but computer no longer suspends. 
acpi=ht : no effect, still high cpu usage
pci=noacpi : not booting at all
pnpacpi=off : no effect, still high cpu usage
noapic : worse, 100% cpu usage 
nolapic : worse, 100% cpu usage

uname -a: Linux 4.13.0-1-amd64 #1 SMP Debian 4.13.4-1 (2017-10-01) x86_64 GNU/Linux

My root folder disk layout is: BTRFS over LVM over LUKS.

How can I find the root of the problem?

Update

I wasn't using my external hard drive which uses a DVD enclosure to be attached to the laptop.

enter image description here Today I reconnected the drive and kworker consumed that excessive amount of CPU again. Note that I didn't mount partition from the external drive, only attaching caused that cpu usage.

ceremcem
  • 2,351

4 Answers4

12

When I checked ACPI interrupts, I noticed that gpe6F had a very high trigger count:

root@HOST:~# grep . -r /sys/firmware/acpi/interrupts/

/sys/firmware/acpi/interrupts/ff_gbl_lock: 0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe15:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe43:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe71:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe05:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe33:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6D:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe61:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe2F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe23:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe51:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe13:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe41:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe03:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe31:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe21:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe78:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe11:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pwr_btn:  0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/ff_slp_btn:  0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0D:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe68:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe01:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_pmtimer:  0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe58:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe48:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe76:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe38:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0B:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe66:       4  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe28:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe56:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe18:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe46:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe74:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe08:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci:  819678
/sys/firmware/acpi/interrupts/gpe36:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe64:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe26:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/error:       0
/sys/firmware/acpi/interrupts/gpe54:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe16:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe44:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe72:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe06:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/ff_rt_clk:             disabled     unmasked
/sys/firmware/acpi/interrupts/gpe34:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6E:  262969  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe62:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe24:       0  EN     enabled      unmasked
/sys/firmware/acpi/interrupts/gpe5E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe52:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe14:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe42:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe70:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe04:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe32:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe60:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe22:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe50:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe79:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe12:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe40:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0E:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe69:       0         disabled     unmasked
/sys/firmware/acpi/interrupts/gpe02:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe30:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe59:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe20:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe49:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe77:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe10:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe4A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe39:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0C:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe67:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe00:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe3A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe29:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe57:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe2A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe19:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe47:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe1A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe75:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe09:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe37:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe0A:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe65:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe27:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe55:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe17:       0     STS invalid      unmasked
/sys/firmware/acpi/interrupts/gpe45:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe7F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/sci_not:     101
/sys/firmware/acpi/interrupts/gpe73:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe07:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe35:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe6F:  560719     STS enabled      unmasked
/sys/firmware/acpi/interrupts/gpe63:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe25:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe5F:       0         invalid      unmasked
/sys/firmware/acpi/interrupts/gpe_all:  823692
/sys/firmware/acpi/interrupts/gpe53:       0         invalid      unmasked

I disabled it:

root@HOST:~# echo "disable" > /sys/firmware/acpi/interrupts/gpe6F

and everything went back to normal:

Linux 4.9.0-6-amd64 (HOST)  05/01/2018  _x86_64_    (4 CPU)

12:30:27 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:30:30 PM     all      6.88      0.00      1.26      0.17      0.00     91.69
12:30:33 PM     all      6.45      0.00      1.17      0.17      0.00     92.20
12:30:36 PM     all      7.15      0.00      1.01      0.34      0.00     91.51
Average:        all      6.83      0.00      1.15      0.22      0.00     91.80
Michael Mrozek
  • 93,103
  • 40
  • 240
  • 233
DEBFIX
  • 121
  • 1
  • 3
  • 4
    Could you please explain what you are doing here and why? Also it is difficult to see what of the information you have posted is important. – Raphael Ahrens May 01 '18 at 12:22
  • 1
    For me it was gpe61. I wish I knew what these were, but my idle CPU usage went down from 11-15% to around 1%. – TheWanderer Jun 16 '18 at 04:40
  • Thanks! This solved the issue for me. Should be accepted as best answer – Kyrol Jun 10 '19 at 09:33
  • This solved it for me as well. I had these 2.5' sata HDDs connected via USB C to SATA adapter bought from Amazon. Whenever it's connected one CPU core went to 100% utilised by kworker. Thanks! – retromuz Sep 04 '19 at 09:23
  • Same interrupt here: gpe6F (Manjaro, 3 different drives). It would be nice if some kernel hacker could tell us what this int is for, or how could we possibly dig deeper (is it from a specific module / device?). For now I'm enyoing the extra CPU core I got back. – Tomasz Gandor May 21 '20 at 23:07
2

kworker is used by encryption (LUKS) so you will see it consume CPU on disk IO (which can be background FS maintenance). You should run :

cryptsetup benchmark

to see if you are missing hardware acceleration.

user1133275
  • 5,574
  • How can I interpret the results? – ceremcem Oct 17 '17 at 22:12
  • @ceremcem a small aes-xts 512b comparison : https://forum.odroid.com/viewtopic.php?f=55&t=18002&start=200#p193600 if your numbers for that sub test are in the low end you can expect slowness/high cpu use. – user1133275 Oct 18 '17 at 06:22
  • The greatest score in the list is 1393 MB/s and mine is 1494 MB/s. So this is not the encryption case I suppose. – ceremcem Oct 18 '17 at 07:44
  • @ceremcem it still might be depending on IO, you could unmount all but a small partition, stop all services, to see if the issue persists.... or start checking logs maybe 'iotop -o -b -n 1' – user1133275 Oct 18 '17 at 07:55
  • Coincidentally I needed to remove my expansion hard disk from the laptop which was recently installed in place of DVD drive and installed the DVD drive temporarily. Now there is no kworker consuming that CPU resource. I'll try somethings more and let you know the results. – ceremcem Nov 01 '17 at 10:43
  • I'm having a similar issue in AWS EC2 but the installed version of cryptsetup doesn't support a "benchmark" option. I'm running v1.5.1 on this instance. Is this option only support by newer versions? – SaxDaddy Jan 09 '18 at 01:24
2

I had a similar problem running Lubuntu 20.10, 5.8 kernel, in an old Macbook 2006. In my case the solution was to mask gpe17 using boot parameter

I added boot parameter acpi_mask_gpe=0x17 to GRUB and rebooted.

Add acpi_mask_gpe=0x17 to /etc/default/grub line GRUB_CMDLINE_LINUX_DEFAULT, then run update-grub and reboot.

That solved the high CPU utilization.

I found the problem with this command: grep . -r /sys/firmware/acpi/interrupts/

The issue: /sys/firmware/acpi/interrupts/gpe17: 3491042 EN enabled unmasked

The solution was provided by Peter Nowee in his answer: Disable GPE ACPI interrupts on boot?

ofbarea
  • 71
0

Apparently it was a hardware related problem. The kworker process was related with the external hdd enclosure where the enclosure had a switch to use for different brands:

enter image description here

Toggling the switch solved the issue.

ceremcem
  • 2,351