What files process or what uses tons of IO?

Question

I uses iotop and this is what I got

Total DISK READ :       0.00 B/s | Total DISK WRITE :      15.78 M/s
Actual DISK READ:      19.08 K/s | Actual DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
16583 be/4 nobody      0.00 B/s    3.82 K/s  0.00 %  0.00 % httpd -k start
16850 be/4 mysql       0.00 B/s    7.04 M/s  0.00 %  0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --lo~te.com.err --open-files-limit=10000 --pid-file=/var/lib/mysql/host.hostgreatsite.com.pid
16317 be/4 nobody      0.00 B/s    3.82 K/s  0.00 %  0.00 % httpd -k start
16859 be/4 mysql       0.00 B/s    3.04 M/s  0.00 %  0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --lo~te.com.err --open-files-limit=10000 --pid-file=/var/lib/mysql/host.hostgreatsite.com.pid
 1024 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % NetworkManager --no-daemon [gdbus]
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % systemd --switched-root --system --deserialize 21
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
 5124 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u12:0]
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
  518 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible recursive locking detect~ption: simd exception: iret exception: /var/log/messages -- /usr/bin/abrt-dump-oops -xtD
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    8 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_bh]
    9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/1]
   11 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/2]
   12 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/3]
   13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/4]
   14 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuob/5]
   15 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_sched]
   16 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/0]
   17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/1]
   18 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/2]
   19 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/3]
   20 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/4]
   21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcuos/5]
   22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
   23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
   24 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
   25 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
15898 be/4 nobody      0.00 B/s    0.00 B/s  0.00 %  0.00 % httpd -k start
   27 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/1:0H]
   28 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
   29 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
   30 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
  517 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % abrtd -d -s
   32 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/2:0H]
   33 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
   34 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
   35 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
16420 be/4 nobody      0.00 B/s    0.00 B/s  0.00 %  0.00 % httpd -k start
   37 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/3:0H]
   38 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
   39 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
   40 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
   42 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/4:0H]
   43 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/5]
   44 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]

Looks like mysql is the problem.

root@host [~]# iostat -xk 5

12/05/2016 _x86_64_ (6 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.35    0.05    8.48    6.11    0.17   49.84

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.26     9.95   24.20   33.63   480.30  1125.98    55.55     7.50  129.63  102.41  149.22   5.55  32.08
xvdap2            0.21     0.83    0.04    0.21     0.99     4.18    41.88     0.00   18.50   10.36   19.88   3.67   0.09
xvdap3            0.16    21.65   16.33   19.33    90.31   230.11    17.97     0.57   15.87   10.18   20.68   1.63   5.82
xvdap4            0.01     0.06    1.26    0.13    15.21     3.25    26.67     0.03   20.28   17.04   52.31   3.19   0.44

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          69.67    0.00   14.01    2.17    0.27   13.88

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00    11.40   10.40   39.80    56.80  1005.60    42.33     4.84   96.41   47.13  109.28   4.16  20.86
xvdap2            0.00     3.60    0.00    2.20     0.00    23.20    21.09     0.01    2.27    0.00    2.27   1.73   0.38
xvdap3            0.00     0.00    0.60    0.00     2.40     0.00     8.00     0.00    0.67    0.67    0.00   0.67   0.04
xvdap4            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          60.56    0.00   13.50    1.07    0.23   24.63

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvdap1            0.00    10.60    1.00  147.80     8.80  6284.00    84.58    10.33   69.45    6.40   69.88   0.67   9.96
xvdap2            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
xvdap3            0.00     0.00    1.00    0.00    32.80     0.00    65.60     0.02   15.80   15.80    0.00   5.00   0.50
xvdap4            0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

It shows that only the first drive xvdap1 that has heavy load.

Is this a correct conclusion?

Yes. Note that questions should not normally be answered by a simple yes or no. — Julie Pelletier, Dec 06 '16 at 04:55
Write-heavy mysql indicates poorly optimized queries creating temporary tables. — jordanm, Dec 06 '16 at 05:01
Yes, but if the temporary tables are kept in memory there's no I/O on disk. Check the values of max_heap_table_size and tmp_table_size in the MySQL config, and increase them if necessary. — dr_, Dec 06 '16 at 08:00

What files process or what uses tons of IO?

0 Answers0