How can I copy a /tmp/ directory that is created & deleted by a process?

Question

I am investigating the behavior of a binary on Oracle Linux 9 (XFS filesystem). This binary, when called by a process, creates a directory under /tmp and copies some files to it. This directory gets a randomized name each time the process runs (a keyword + a GUID).

Immediately after, it deletes the directory. I want to access the files contained in this directory before it is deleted, but the whole process ends too fast for any of my commands.

Is there any way I could "intercept" and copy this directory before it is deleted?

Answer 1

You could always run the application under:

gdb --args /path/to/your/your-program and its args

Then add breakpoints on unlink(), unlinkat(), rmdir() functions or syscalls:

catch syscall unlink
catch syscall unlinkat
catch syscall rmdir
run

Then each time a breakpoint is reached, check that it's about deleting files in that directory and inspect the files in there or copy them elsewhere. Enter cont in gdb to resume execution (until the next breakpoint).

Example with rm -rf:

$ gdb -q --args rm -rf /tmp/tmp.HudBncQ4Ni
Reading symbols from rm...
Reading symbols from /usr/lib/debug/.build-id/f6/7ac1d7304650a51950992d074f98ec88fe2f49.debug...
(gdb) catch syscall unlink
Catchpoint 1 (syscall 'unlink' [87])
(gdb) catch syscall unlinkat
Catchpoint 2 (syscall 'unlinkat' [263])
(gdb) catch syscall rmdir
Catchpoint 3 (syscall 'rmdir' [84])
(gdb) run
Starting program: /bin/rm -rf /tmp/tmp.HudBncQ4Ni
Catchpoint 2 (call to syscall unlinkat), 0x00007ffff7eb6fa7 in __GI_unlinkat () at ../sysdeps/unix/syscall-template.S:120
120     ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) info registers
rax            0xffffffffffffffda  -38
rbx            0x555555569830      93824992319536
rcx            0x7ffff7eb6fa7      140737352789927
rdx            0x0                 0
rsi            0x555555569938      93824992319800
rdi            0x4                 4
rbp            0x555555568440      0x555555568440
rsp            0x7fffffffda48      0x7fffffffda48
r8             0x3                 3
r9             0x0                 0
r10            0xfffffffffffffa9c  -1380
r11            0x206               518
r12            0x0                 0
r13            0x7fffffffdc30      140737488346160
r14            0x0                 0
r15            0x555555569830      93824992319536
rip            0x7ffff7eb6fa7      0x7ffff7eb6fa7 <__GI_unlinkat+7>
eflags         0x206               [ PF IF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0
(gdb) x/s $rsi
0x555555569938: "test"
(gdb) info proc
process 7524
cmdline = '/bin/rm -rf /tmp/tmp.HudBncQ4Ni'
cwd = '/export/home/stephane'
exe = '/bin/rm'
(gdb) !readlink /proc/7524/fd/4
/tmp/tmp.HudBncQ4Ni
(gdb) !find /tmp/tmp.HudBncQ4Ni -ls
  1875981      4 drwx------   2 stephane stephane     4096 Aug  8 09:30 /tmp/tmp.HudBncQ4Ni
  1835128      4 -rw-r--r--   1 stephane stephane        5 Aug  8 09:30 /tmp/tmp.HudBncQ4Ni/test

Here, the breakpoint was on the unlinkat() system call for the test entry inside /tmp/tmp.HudBncQ4Ni on a x86_64 Linux system where the first two arguments of the syscall are in the rdi and rsi registers.

strace can inject signals to a process when a syscall is called (strace -e inject=unlink,unlinkat,rmdir:signal=STOP to suspend for instance), but AFAICT it always does it after the syscall returns, so once the file has already been removed.

You can however delay the entry so you can suspend by hand with Ctrl+Z for instance:

$ strace -e inject=unlink,unlinkat,rmdir:delay_enter=5s -e unlink,unlinkat,rmdir rm -rf /tmp/tmp.HudBncQ4Ni
unlinkat(4, "test", 0^Z
zsh: suspended  strace -e inject=unlink,unlinkat,rmdir:delay_enter=10s -e  rm -rf

Or, as suggested by @PhilippWendler, you can use:

strace -e inject=unlink,unlinkat,rmdir:retval=0 -e unlink,unlinkat,rmdir ...

or:

strace -e inject=unlink,unlinkat,rmdir:error=EACCES -e unlink,unlinkat,rmdir ...

To hijack the syscalls and pretend they succeed (with retval=0) or fail (with EACCES here meaning Permission denied) without actually calling them.

Both gdb and strace can attach to an already running process with --pid <the-process-id> / -p <the-process-id> respectively. They can also be told to follow forks and execs and trace the children as well so you can attach to the parent and watch for or hijack unlinks in the children (see -f in strace and the follow-* settings in gdb)

Answer 2

I found this shell script that uses inotify-tools, and it did exactly what I was looking for (author: https://unix.stackexchange.com/a/265995/536771):

#!/bin/sh
TMP_DIR=/tmp
CLONE_DIR=/tmp/clone
mkdir -p $CLONE_DIR
wait_dir() {
  inotifywait -mr --format='%w%f' -e create "$1" 2>/dev/null | while read file; do
    echo $file
    DIR=dirname "$file"
    mkdir -p "${CLONE_DIR}/${DIR#$TMP_DIR/}"
    cp -rl "$file" "${CLONE_DIR}/${file#$TMP_DIR/}"
  done
}
trap "trap - TERM && kill -- -$$" INT TERM EXIT
inotifywait -m --format='%w%f' -e create "$TMP_DIR" | while read file; do
  if ! [ -d "$file" ]; then
    continue
  fi
echo "setting up wait for $file"
  wait_dir "$file" &
done

The simpler solution that worked for me even better than the script: chattr +a /tmp

This is because the script fails if the binary creates a single file under /tmp instead of a folder. It also fails if the binary creates more than one folder under /tmp.

Edit: an even simpler solution that worked was to run:

cp -rp /source /clone

chattr interfered with what I was checking, and the first script works fine for directories created under /tmp, but not for files created under /tmp

Answer 3

I had a similar situation in the past. I vaguely remember running something similar to chattr -R -a /tmp, essentially making /tmp append-only. Processes can create files/directories but not delete them. Please double-check the command before running in and make sure you undo the attributes as soon as you can.

Answer 4

The solution, I once used for a different function (listen) was to create a simple dynamic library where you redefine functions of interest (e.g. unlink or fopen).

Compile, link it with -fPIC to create a dynamic library and then inject it to the binary with something like

LD_PRELOAD=/path/to/mylib.so ./binary