I run iotop and vmstat on a CentOS server. Under some load the iotop swapin column starts to display values around 5% to 20% for some processes. But vmstat si field remains at zero. I thought that vmstat si and iotop swapin would both reflect swapping. How can si be zero while iostat shows significant swapin?
Side note:
vmstat si: Amount of memory swapped in from disk (/s)
If you take a look at the diagram I posted on this U&L Q&A titled: Diagram of Linux kernel vs. performance tools? it shows what subsystem of the Linux kernel each of those tools is showing performance data from.
iotop primary purpose is in showing stats from the perspective of the block device interface. So this would be your HDD, for example. While vmstat shows stats from the kernel's virtual memory subsystem.
This paragraph from iotop's man page also explains what it's up to:
iotop displays columns for the I/O bandwidth read and written by each process/thread during the sampling period. It also displays the percentage of time the thread/process spent while swapping in and while waiting on I/O. For each process, its I/O priority (class/level) is shown.
Total DISK READ : 0.00 B/s | Total DISK WRITE : 36.93 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
303 be/3 root 0.00 B/s 29.54 K/s 0.00 % 4.42 % [jbd2/dm-1-8]
29559 be/4 saml 0.00 B/s 7.39 K/s 0.00 % 0.00 % google-chrome-stable [Chrome_HistoryT]
So in the above example jdb2 has spent 4.42% of its time waiting for I/O resources. This is over it's life, not at this particular moment.
The output from vmstat is showing the ongoing stats from virtual memory, indicating any pages that are either dirty or need to be pushed/pulled from swap, so you're seeing these pages being shuttled around there. You have no indications from vmstat what process these pages belong to, only that there was pages swapped in/out.
$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 2990956 1429112 45664 712416 8 15 50 149 7 33 34 5 60 2
0 0 2990948 1385392 45696 725096 0 0 2009 36 833 1556 6 1 92 1
0 0 2990948 1380700 45704 726516 0 0 0 16 700 1423 3 1 95 1
2 0 2990944 1386960 45760 724400 3 0 3 264 1075 1837 5 1 91 3
The si & so columns are the swap related stats.
The value being reported for si is actually a rate of change value (amount moved per second). See this article titled: Use vmstat to Monitor System Performance:
The swap section reports the rate that memory is sent to or retrieved from the swap system. By reporting “swapping” separately from total disk activity, vmstat allows you to determine how much disk activity is related to the swap system.
The si column reports the amount of memory that is moved from swap to “real” memory per second. The so column reports the amount of memory that is moved to swap from “real” memory per second.
Given this it's likely that even though a process is reporting that it's using swap through iostat, that usage of swap is probably not enough to get picked up over the course of a second that would show up in vmstat's "si" measurement.
