4

When I run a command in a container, I'm getting

inotify_init1() failed: Too many open files

How can I see how many inotify watchers I have open and what's holding them open?

Evan Carroll
  • 30,763
  • 48
  • 183
  • 315

2 Answers2

4
#!/bin/bash -e

cols=$(stty size | awk '{print $2}')

printf "%-13s%-8s%-6s%-5s%s\n" COUNT PPID PID UID CMD

grep ^inotify /proc/*/fdinfo/* 2>/dev/null |
    cut -d/ -f3 | sort | uniq -c | sort -n |
    while read -r count pid; do
        printf "%-10s" "$count"
        ps -o ppid=,pid=,uid=,cmd= -p "$pid"
    done |
    awk '{sum += $1; print} END {print sum}' |
    cut -c-"$cols"

echo
sysctl fs.inotify.max_user_watches

Sample output:

COUNT        PPID    PID   UID  CMD
1            1303  112217  1000 xfce4-terminal --maximize
1               1    1169   100 /lib/systemd/systemd-timesyncd
1            1238    1255  1000 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
1            1187    1292  1000 xfce4-panel
1               1    1614     0 /usr/lib/clightd/clightd
1            1085    1672  1000 /usr/libexec/gvfs-udisks2-volume-monitor
1            1085    1810  1000 /usr/libexec/gvfs-afc-volume-monitor
1            1085    1903  1000 /usr/libexec/evolution-calendar-factory
1            1085    1958  1000 /usr/libexec/evolution-addressbook-factory
1               1     738     0 /usr/sbin/acpid
1               1     760     0 /lib/systemd/systemd-logind
2            1085    1092  1000 /usr/bin/pulseaudio --daemonize=no --log-target=journal
2            1085    1112  1000 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
2            1085    1278  1000 /usr/bin/gpg-agent --supervised
2            1085    1866  1000 /usr/libexec/evolution-source-registry
3               1    1675     0 /usr/lib/udisks2/udisksd
3               1     742   106 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
6            1241    1885  1000 /usr/libexec/gvfsd-trash --spawner :1.8 /org/gtk/gvfs/exec_spaw/0
8               1   17770  1000 /usr/lib/firefox/firefox
10           1292    1303  1000 /usr/lib/x86_64-linux-gnu/xfce4/panel/wrapper-2.0 /usr/lib/x86_64-linux-gnu/xfce4/panel/plugins/libwhiskermenu.so 6 12
12              1     332     0 /lib/systemd/systemd-udevd
14              1    1268     0 /usr/lib/policykit-1/polkitd --no-debug
18              1    1085  1000 /lib/systemd/systemd --user
21           1187    1419  1000 xfdesktop
36              1    1291  1000 xfsettingsd
46              0       1     0 /sbin/init
196

fs.inotify.max_user_watches = 2048
Irfan Latif
  • 448
3

To find out how many inotify_init files are opened for a certain user, use the above snippet (replace user) :

find /proc/*/fd -user "<user>" -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev/null | xargs cat | grep -c '^inotify'

You can also use inotify-consumers and add it to you aliases in your profile for next uses to find out the number of inotify_init opened files for the current user :

inotify-consumers

They can be increased by setting :

echo <new-number> > /proc/sys/fs/inotify/max_user_instances

But it's not permament ; you can use /etc/sysctl.conf to do so.

echo fs.inotify.max_user_watches=<new-number> | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
Reda Salih
  • 1,754