How to find out which file is currently written by a process

Question

My situation is that from time to time a specific process (in this case, it's Thunderbird) doesn't react to user input for a minute or so. I found out using iotop that during this time, it writes quite a lot to the disk, and now I want to find out which file it writes to, but unfortunately iotop gives only stats per process and not per open file(-descriptor).

I know that I can use lsof to find out which files the process has currently open, but of course Thunderbird has a lot of them open, so this is not that helpful. iostat only shows statistics per device.

The problem occurs only randomly and it might take quite some time for it to appear, so I hope I don't have to strace Thunderbird and wade through long logs to find out which file has the most writes.

See also Logging outgoing connections as they happen for network connections — Gilles 'SO- stop being evil', Feb 12 '12 at 12:43

score 6 · Accepted Answer · answered Feb 10 '12 at 13:08

6

If you attach strace to the process just when it's hung (you can get the pid and queue the command up in advance, in a spare terminal), it'll show the file descriptor of the blocking write.

Trivial example:

$ mkfifo tmp
$ cat /dev/urandom > tmp &
[1] 636226
  # this will block on open until someone opens for reading

$ exec 4<tmp
  # now it should be blocked trying to write

$ strace -p 636226
Process 636226 attached - interrupt to quit
write(1, "L!\f\335\330\27\374\360\212\244c\326\0\356j\374`\310C\30Z\362W\307\365Rv\244?o\225N"..., 4096 <unfinished ...>
^C
Process 636226 detached

answered Feb 10 '12 at 13:08

Useless

4,800

Thanks, I didn't know I can attach strace while the process is running. I will try this. – Philipp Wendler Feb 10 '12 at 13:36
1

You can cross this information with lsof -p $PID, in order to know where the file descriptor points – Coren Feb 10 '12 at 14:56
1

or ls -l /proc/pid/fd on Linux – Useless Feb 10 '12 at 15:53
Using strace I could actually find out exactly what I wanted to know, so thanks again! – Philipp Wendler Feb 13 '12 at 09:30

score 3 · Answer 2 · edited May 23 '17 at 12:40

3

I you have root access, I think the best tool would be the audit subsystem. There isn't much literature about it (but more than about loggedfs); you can start with this tutorial or a few examples or just with the auditctl man page. Here, it should be enough to make sure the daemon is started, then run auditctl as root:

auditctl -a exit,always -F pid=1234 -F dir=/home/philipp

This will write to logs in /var/log/audit/audit.log every time the process with pid 1234 writes somewhere under /home/philipp. The overhead is fairly small, a lot smaller than strace.

edited May 23 '17 at 12:40

Community

1

answered Feb 12 '12 at 12:43

Gilles 'SO- stop being evil'

829,060

1

Thanks for pointing out the audit subsystems as an interesting alternative to strace. However, it doesn't seem to help here as it seems to logs only when the file is opened, and thus I can't find out how much is written into each file. – Philipp Wendler Feb 13 '12 at 09:29
@PhilippWendler Ah. Hmmm. Try adding -S read -S write (untested). – Gilles 'SO- stop being evil' Feb 14 '12 at 06:50

How to find out which file is currently written by a process

2 Answers2

Linked