31

A specific file on our production servers is being modified at apparently random times which do not appear to correlate with any log activity. We can't figure out what program is doing it, and there are many suspects. How can I find the culprit?

It is always the same file, at the same path, but on different servers and at different times. The boxes are managed by puppet, but the puppet logs show no activity at the time the file is modified.

What kernel hook, tool, or technique could help us find what process is modifying this file?

lsof is unsuitible for this, because the file is being opened, modified and closed very quickly. Any solution that relies upon polling (such as running lsof often) is no good.

  • OS: Debian testing
  • Kernels: Linux, 2.6.32 through 3.9, both 32 and 64-bit.
Wayne Conrad
  • 592
  • 6
    http://www.la-samhna.de/library/audit.html – goldilocks Nov 06 '13 at 13:37
  • @goldilocks I'd give this the checkmark, if it were an answer – Wayne Conrad Nov 06 '13 at 15:06
  • Hopefully someone comes along who can write one then, lol. I'm aware of auditd and that looks like a decent introduction, so it seemed a good suggestion -- but I haven't made use of it myself, so I'm not really qualified to elaborate. – goldilocks Nov 06 '13 at 15:10

4 Answers4

42

You can use auditd and add a rule for that file to be watched:

auditctl -w /path/to/that/file -p wa

Then watch for entries to be written to /var/log/audit/audit.log.

Jonathon Reinhart
  • 1,861
Stéphane Chazelas
  • 544,893
  • 2
    +1 (again!). That is much better than using "lsof", which would need to catch the exact moment the modification occurs (very unlikely). You can have mode details in the top answer on this unix.se post, which mentions the use of monitoring facilities such as "auditctl", and how to set them up and use them. That is probably the safest way to ensure you know what happen(s)(ed)! – Olivier Dulac Nov 06 '13 at 16:59
  • This was very easy to set up, and quickly found the process that was modifying the file. – Wayne Conrad Nov 07 '13 at 12:52
  • 3
    To install: apt install auditd – Jonathon Reinhart Jan 15 '17 at 17:51
  • how do we remove the auditctl on the fle ? – Siddharth Dec 08 '18 at 07:45
  • To install on CentOS: yum install audit – alphaGeek Aug 29 '19 at 08:47
  • On Archlinux, core/audit will give you auditctl and you can consult man auditd as well as man auditd.conf for background reading. auditctl -h is useful if you actually want to do things. – Cbhihe May 30 '20 at 20:24

  • Then watch for entries to be written to /var/log/audit/audit.log.

    more convenient way is: sudo ausearch -f /path/to/that/file

    Also you may watch for directories and symlinks

     – Igor Tverdovskiy Jul 23 '20 at 22:12
  • I get Error - audit support not in kernel Cannot open netlink audit socket in Unraid (Slackware) – mgutt Oct 14 '20 at 22:17
  • 1
    Complementing this answer, auditctl -a always,exit -F dir=/path/to/that/file -F perm=wa -F uid!=<whitelisted_userid> -k mykey where it will ignore changes made my <whitelisted_userid> – Sandeep Kanabar Dec 16 '20 at 11:07
  • To remove the auditctl rule once you finished the task you need to do auditctl -W path/file. You can see the auditctl rules with auditctl -l – NetVicious Jan 16 '24 at 07:53
4

in case the program(s) you are looking for still have the file opened, you can use the following:

 sudo lsof /path/to/file/being/modified

you could also call this in a small loop, suing the following script getfileusers.sh:

 #!/bin/sh

 FILE=$1

 while true; do
    lsof "${FILE}"
 done > /tmp/fileusers.log

and then call it:

 sudo ./getfileusers.sh /path/to/file/being/modified

and eventually inspect /tmp/fileusers.log to see who touched the file...

umläute
  • 6,472
  • I'm tempted to -1... the loop will be very heavy on CPU load (especially as it invoke "true" on each passage, and then "lsof"), and still have a great chance of missing the time the modification occurs [unless it's long enough, which is not always the case]! See StephaneChazelas answer for a much better solution, with more chance to actually catch the event(s). – Olivier Dulac Nov 06 '13 at 17:00
  • this was helpful – vsingh Nov 19 '23 at 00:21
4

SystemTap can do this, using the inodewatch script .

Mark Plotnick
  • 25,413
  • 3
  • 64
  • 82
2

If you can catch it in the act, that is if the file is being accessed at the moment, you can find the process by running lsof:

$ while :; do echo foo >> bar.txt; done &
$ sudo lsof bar.txt
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    25723 terdon    1u   REG    8,6  5015796 16647496 bar.txt
terdon
  • 242,166