Is there an easy way to log all commands executed, including command line arguments?

Question

I am trying to find how to log a specific instantiation of rrdtool to see whether the path it is receiving is incorrect.

I know I could wrap the executable in a shell script that would log the parameters, but I was wondering if there was a more kernel-specific way to monitor for that, perhaps a filesystem callback that sees when a particular /proc/pid/exe matches a given binary?

Answer 1

Yes, there is a kernel facility: the audit subsystem. The auditd daemon does the logging, and the command auditctl sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve system call:

auditctl -a exit,always -S execve

To specifically trace the invocation of a specific program, add a filter on the program executable:

auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool

The logs show up in /var/log/audit.log, or wherever your distribution puts them. You need to be root to control the audit subsystem.

Once you're done investigating, use the same command line with -d instead of -a to delete a logging rule, or run auditctl -D to delete all audit rules.

For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.

Answer 2

You could use snoopy.

Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload.

Disclosure: I am current snoopy maintainer.

Answer 3

The Linux kernel "audit" subsystem can do what you need.

e.g. if you run these commands:

auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve

Then every execution event is logged, and a lot of information is provided around that

e.g. this is the output of me running tail /var/log/audit/audit.log

exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64):  cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67

There's some interesting values that can be seen; e.g. "auid" is 500, which is my login ID, even though "uid" is zero ('cos I'm running under su). So even though the user may have switched accounts with su or sudo we can still track back to their "audit ID"

Now those auditctl commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/ directory, on CentOS 7). The exact location will depend on your OS version. The auditctl manual page should help here.

Beware, though... this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!

If necessary the rules can be limited to a specific user, or a specific command.

And also beware; if a user puts the password in the command execution (e.g. mysql --user=username --password=passwd) then this will be logged.