36

I have a long running bash instance (inside a screen session) that is executing a complex set of commands inside a loop (with each loop doing pipes, redirects, etc).

The long command line was written inside the terminal - it's not inside any script. Now, I know the bash process ID, and I have root access - how can I see the exact command line being executed inside that bash?

Example

bash$ echo $$
1234
bash$ while true ; do \
    someThing | somethingElse 2>/foo/bar | \
    yetAnother ; sleep 600 ; done

And in another shell instance, I want to see the command line executed inside PID 1234:

bash$ echo $$
5678
bash$ su -
sh# cd /proc/1234
sh# # Do something here that will display the string  \
   'while true ; do someThing | somethingElse 2>/foo/bar | \
    yetAnother ; sleep 600 ; done'

Is this possible?

EDIT #1

Adding counter-examples for some answers I've got.

  1. About using the cmdline under /proc/PID: that doesn't work, at least not in my scenario. Here's a simple example:

    $ echo $$
8909

$ while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done

    In another shell:

    $ cat /proc/8909/cmdline
bash

  2. Using ps -p PID --noheaders -o cmd is just as useless:

    $ ps -p 8909 --no-headers -o cmd
bash

  3. ps -eaf is also not helpful:

    $ ps -eaf | grep 8909
ttsiod    8909  8905  0 10:09 pts/0    00:00:00 bash
ttsiod   30697  8909  0 10:22 pts/0    00:00:00 sleep 30
ttsiod   31292 13928  0 10:23 pts/12   00:00:00 grep --color=auto 8909

    That is, there's no output of the ORIGINAL command line, which is what I'm looking for - i.e the while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done.

Gilles 'SO- stop being evil'
  • 829,060
ttsiodras
  • 2,371
  • 1
  • 21
  • 26

3 Answers3

50

I knew I was grasping at straws, but UNIX never fails!

Here's how I managed it:

bash$ gdb --pid 8909
...
Loaded symbols for /lib/i386-linux-gnu/i686/cmov/libnss_files.so.2
0xb76e7424 in __kernel_vsyscall ()

Then at the (gdb) prompt I ran the command, call write_history("/tmp/foo") which will write this history to the file /tmp/foo.

(gdb) call write_history("/tmp/foo")
$1 = 0

I then detach from the process.

(gdb) detach
Detaching from program: /bin/bash, process 8909

And quit gdb.

(gdb) q

And sure enough...

bash$ tail -1 /tmp/foo
while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done

For easy future re-use, I wrote a bash script, automating the process.

ttsiodras
  • 2,371
  • 1
  • 21
  • 26
  • 1
    +1 very nice sleuthing.Be sure to mark this as the excepted A in 2 days. – slm Oct 03 '14 at 08:28
  • @slm: Thanks! I'll write a blog post about it - it was fun, hunting this down. – ttsiodras Oct 03 '14 at 08:36
  • 1
    With readline enabled you can also do this in gdb: print (char *)rl_line_buffer. The current command in a sequence is print (char *)the_printed_command. You may also call history_builtin(), but that will output on the tty of the bash process, so it might be less useful. – mr.spuratic Oct 03 '14 at 10:16
  • 12
    @slm: Couldn't resist - I blogged about it here: http://users.softlab.ece.ntua.gr/~ttsiod/bashheimer.html – ttsiodras Oct 03 '14 at 13:59
  • 2
    Is bash thread-safe? You'll have to hope you're not modifying some internal state that upsets the currently-executing code when you execute something from gdb. In practice, bash will almost certainly be just waiting for its child process to finish whenever you happen to suspend it with gdb. – Adrian Pronk Oct 04 '14 at 20:43
  • read this post, read your blog, your commandlinefu is epic! – kvantour Jan 30 '21 at 10:22
5

Since the command is still running in screen, its parent bash has not reread any history so:

  • reattach to screen
  • press ^Z then up arrow
  • bonus: wrap the command in single quotes (navigating with ^A^A - because screen(1) - and ^E) and echo + redirect into a file
  • fg to pursue command execution

There are caveats, but this is useful enough, most of the time.

Lloeki
  • 225
  • Yes, even killing it and pressing up. Though you have to be confident that it will start again without any problem, but if it doesn't then you'd have the same problem after a reboot anyway. You just have to choose your moment when a downtime wouldn't be a problem. – Matthieu Napoli Oct 04 '14 at 09:16
1

I know you found your own answer, but is there a reason you can't do something like this:

(set -x; for f in 1 2 3 4 ; do  echo "$f"; sleep $f; done)

Perhaps you can't co-mingle the output of the actual job and the output from bash showing the currently executing line.

Also, FWIW, if you prefer verbosity, set -o xtrace.

crimson-egret
  • 322