2

Reading up on an answer to find out which process registered inotify watchers, I executed the following commands 

echo 1 | sudo tee /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
sudo cat /sys/kernel/debug/tracing/trace

The resulting output from the trace file had a header that indicated that the first column should have the name of the process (task?), along with the PID:

#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
           gmain-1715  [004] .... 23200.386116: sys_inotify_add_watch -> 0xfffffffffffffffe

Surprisingly, none of the PIDs listed in the many lines of output exist when I grep for them using ps -ef | grep $THE_PID (where $THE_PID would be 1715). Also, the task name is unknown to me and does not exist in the ps output either.

Further, the entire list is all tuples like gmain-<some number>. Not the expected postgres or node. So what is this gmain process? And what are these PIDs that do not exist in /proc/? Are they historic by the time I try accessing them?

I see gmain is a thing in these kernel docs by Theodore Ts'o on tracing, so it's something.

oligofren
  • 1,150
  • 1
    this is not answering your actual question, but the original problem: that answer you link to is not the best approach; info about watches set by inotify is present since linux 3.8 in /proc/PID/fdinfo/FD, so you can adapt my answer to a similar question: find /proc/*/fd -lname anon_inode:inotify -printf '%hinfo/%f\n' 2>/dev /null | xargs grep -c '^inotify'. There are better answers (using fdinfo) even to the question you're linking to. –  Feb 22 '19 at 16:03
  • Ooh, nice oneliner! I even learned a new thing with the -c option. Genius. – oligofren Feb 22 '19 at 16:17
  • @mosvy I cobbled togheter this utility script from your comment: https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers. Thanks! – oligofren Feb 22 '19 at 17:07

2 Answers2

3

These are not processes. These are tasks. Linux works in terms of tasks. You do not see their IDs in a process list because these tasks happen to be threads within a process. They are the worker threads of some multi-threaded GIO process somewhere. You'll find them in the task/ subdirectory of the /proc/<process> subdirectory (example /proc/860/task/926).

oligofren
  • 1,150
JdeBP
  • 68,745
1

As suspected, this gmain process had something to do with GTK or Gnome, but the thing to note is that it wasn't a process at all, but a thread (the gtk main loop)! That's also why it didn't show when grepping ps.

I understood this when using the -q option to ps, which lets you list the pids you are interested in. The pid that came up wasn't the one I passed as an option at all, but still showed when I did pstree -p, which made me think this was perhaps some kind of thread.

Using my newfound knowledge, I found that I could list all the threads, that also have PIDs (is this the right name?) of their own by supplying ps with the -L option.

Example: sudo ps -efL -q 906.

This (and pstree) led me to find out that the thread belonged to NetworkManager).

oligofren
  • 1,150