279

I'm currently facing a problem on a linux box where as root I have commands returning error because inotify watch limit has been reached.

# tail -f /var/log/messages
[...]
tail: cannot watch '/var/log/messages': No space left on device
# inotifywatch -v /var/log/messages
Establishing watches...
Failed to watch /var/log/messages; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via '/proc/sys/fs/inotify/max_user_watches'.`

I googled a bit and every solution I found is to increase the limit with:

sudo sysctl fs.inotify.max_user_watches=<some random high number>

But I was unable to find any information of the consequences of raising that value. I guess the default kernel value was set for a reason but it seems to be inadequate for particular usages. (e.g., when using Dropbox with a large number of folder, or software that monitors a lot of files)

So here are my questions:

  • Is it safe to raise that value and what would be the consequences of a too high value?
  • Is there a way to find out what are the currently set watches and which process set them to be able to determine if the reached limit is not caused by a faulty software?
rozcietrzewiacz
  • 39,269
Ultraspider
  • 3,073
  • You've probably checked this by now since this is 8 months old, but is your drive full? "tail: cannot watch '/var/log/messages': No space left on device" – froggythefrog Oct 02 '19 at 18:33

2 Answers2

390

Is it safe to raise that value and what would be the consequences of a too high value?

Yes, it's safe to raise that value and below are the possible costs [source]:

  • Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]
  • This comes out of kernel memory, which is unswappable.
  • Assuming you set the max at 524288 and all were used (improbable), you'd be using approximately 256MB/512MB of 32-bit/64-bit kernel memory.
    • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.

To check the max number of inotify watches:

cat /proc/sys/fs/inotify/max_user_watches

To set max number of inotify watches

Temporarily:

  • Run sudo sysctl fs.inotify.max_user_watches= with your preferred value at the end.

Permanently (more detailed info):

  • put fs.inotify.max_user_watches=524288 into your sysctl settings. Depending on your system they might be in one of the following places:
    • Debian/RedHat: /etc/sysctl.conf
    • Arch: put a new file into /etc/sysctl.d/, e.g. /etc/sysctl.d/40-max-user-watches.conf
  • you may wish to reload the sysctl settings to avoid a reboot: sysctl -p (Debian/RedHat) or sysctl --system (Arch)

Check to see if the max number of inotify watches have been reached:

Use tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg: - If all is well, it will show the last 10 lines and pause; abort with Ctrl-C - If you are out of watches, it will fail with this somewhat cryptic error:

tail: cannot watch '/var/log/dmsg': No space left on device

To see what's using up inotify watches

find /proc/*/fd -lname anon_inode:inotify |
   cut -d/ -f3 |
   xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
   uniq -c |
   sort -nr

The first column indicates the number of inotify fds (not the number of watches though) and the second shows the PID of that process [sources: 1, 2].

Community
  • 1
tshepang
  • 65,642
  • 6
    I guess very few codes need the values higher than the default Dropbox may require a higher limit, depending on how many files you have. Iv'e raised mine with no issues. in fact, the dropbox notification (that occurs when it reaches its limit) explicitly tells you to raise it. – Falmarri May 25 '11 at 20:55
  • Isn't there also a dnotify that works the same, but on directories? Maybe consolidate multiple files you are inotifying into a single directory? – LawrenceC Jun 16 '11 at 13:36
  • 2
    @ultrasawblade- inotify replaced dnotify. dnotify was slow and buggy. inotify can be used on directories, and a directory will be "changed" when one of the files in that directory (one level deep) is modified. Directories are just files anyway. – beatgammit Jul 10 '11 at 05:14
  • Other people might run into this, too. The ps command above says error: process ID list syntax error with procps-ng version 3.3.9 on Ubuntu. – CodeGnome Apr 19 '15 at 18:23
  • 7
    "Permanently: Replace the value within the /proc/sys/fs/inotify/max_user_watches" <-- this is incorrect. To make this permanent you need to change /etc/sysctl.conf – Merc May 07 '16 at 04:38
  • You should uniq before you run ps otherwise this can take very long. – user3467349 Oct 17 '16 at 15:00
  • Note that what you describe as "Debian/RedHat" and "Arch" are more precisely SysV and systemd systems, respectively. Debian-based systems now use systemd, so the "Arch" method is preferrable for them. – stackexchanger Jan 06 '17 at 23:46
  • 1
    @stackexchanger if you very sure that these are similar (e.g. you've tested), then please submit and edit to the Question. I myself haven't checked, and I know that distros tend to patch packages, so behavior could be different for the same packages. This is not to mention differences in package versions as well. – tshepang Jan 08 '17 at 22:55
  • 1
    +1 Debian-Version works for current Ubuntu-Based (for its based on Debian) Systems. I needed the higher value for indexing a big git repo via gitkraken (git-gui) – Tobias Gaertner Oct 23 '17 at 09:32
  • Did you mean -f1 instead of -f3? – Karel Vlk May 12 '21 at 18:33
  • I recursively watched 50000 files in a directory in Ubuntu 20 and the was far less than 1kb per file as suggested above. It used about 5MB. – Jonathan Aug 01 '22 at 08:14
  • Worth mentioning you need to run sysctl -p also for the temp solution, for changes to apply – Mugen Jan 04 '24 at 12:59
5

From this other topic how to get the number of inotify watches in use there is a reference to this script inotify-consumers which I found to be very helpful because it will show you exactly who is hogging the watchers.

TaborKelly
  • 206