How to view the output of a running process in another bash session?

Question

I have left a script running on a remote machine from when I was locally working at it. I can connect over SSH to the machine as the same user and see the script running in ps.

$ ps aux | grep ipcheck
myuser  18386  0.0  0.0  18460  3476 pts/0    S+   Dec14   1:11 /bin/bash ./ipchecker.sh

It is simply outputting to stdout on a local session (I ran ./ipchecker.sh form a local terminal window, no redirection, no use of screen etc).

Is there anyway from an SSH session I can view the output of this running command (without stopping it)?

So far the best I have found is to use strace -p 18386 but I get hordes of text flying up the screen, its far too detailed. I can stop strace and then sift through the output and find the text bring printed to stdout but its very long and confusing, and obviously whilst it's stopped I might miss something. I would like to find a way to see the script output live as if I was working locally.

Can anyone improve on this? The obvious answer is to restart the script with redirection or in a screen session etc, this isn't a mission critical script so I could do that. Rather though, I see this as a fun learning exercise.

Answer 1

You can access the output via the proc filesystem.

tail -f /proc/<pid>/fd/1

1 = stdout, 2 = stderr

(or like @jmhostalet says: cat /proc/<pid>/fd/1 if tail doesn't work)

Answer 2

If all you want to do is spy on the existing process, you can use strace -p1234 -s9999 -e write where 1234 is the process ID. (-s9999 avoids having strings truncated to 32 characters, and write the system call that produces output.) If you want to view only data written on a particular file descriptor, you can use something like strace -p1234 -e trace= -e write=3 to see only data written to file descriptor 3 (-e trace= prevents the system calls from being loged). That won't give you output that's already been produced.

If the output is scrolling by too fast, you can pipe it into a pager such as less, or send it to a file with strace -o trace.log ….

With many programs, you can divert subsequent output with a ptrace hack, either to your current terminal or to a new screen session. See How can I disown a running process and associate it to a new screen shell? and other linked threads.

Note that depending on how your system is set up, you may need to run all these strace commands as root even if the process is running under your user with no extra privileges. (If the process is running as a different user or is setuid or setgid, you will need to run strace as root.) Most distributions only allow a process to trace its children (this provides a moderate security benefit — it prevents some direct malware injection, but doesn't prevent indirect injection by modifying files). This is controlled by the kernel.yama.ptrace_scome sysctl.

Answer 3

In BSD, you can use watch which snoops a given tty, e.g.

watch /dev/pts/0

In Linux, it won't be possible if the process wasn't run under multiplexer before such as screen or tmux. See also: Reptyr: Attach a Running Process to a New Terminal

It seems the only way is to debug the process (e.g. strace, dtrace/dtruss, gdb, lldb, etc.).

Since you've used strace, to fetch any meaningful output, you need to filter by a qualifying expression (such as file), then parse the output. Here is example:

strace -e trace=write -s1000 -fp 18386 2>&1 | grep -o '".\+[^"]"'

What it does it prints write operation of the process (1000 length) specified by PID (use pgrep to find it by name), redirects standard error into output (to be filtered), and prints double-quoted string.

If you're dealing with binary output, you may parse escape sequences characters by using read (with -r) and printf (with %b), e.g.

while read -r -t1 line; do printf "%b" $line; done

Check help read for more parameters (e.g. -n to print after certain amount of characters, rather than newline).

Here is more complete example:

strace -e trace=write -s1000 -fp 18386 2>&1 \
| grep --line-buffered -o '".\+[^"]"' \
| grep --line-buffered -o '[^"]\+[^"]' \
| while read -r line; do
  printf "%b" $line;
done

For examples using any process, please check: How to parse strace in shell into plain text? at stackoverflow

Answer 4

Parsing the output of strace:

I used the top answer (with my process ID of 28223) ...

> sudo strace -p28223 -s9999 -e write
...
write(9, "Info\nI\nCare\nabout", 55) = 55
...

To determine that I care about write(9. (The 9 is used below, it's probably a file handle and might be different for your process.) Then I wrote a quick Ruby script to parse and display them.

Paste the following into /usr/bin/parse_strace.rb

#!/usr/bin/ruby

num = ARGV[0]
STDIN.each { |line|
  if (line.match(/write\(#{ num },\s*"(.*?)"/)) then
    puts $1.split('\x').map { |s| s.to_i(16).chr }.join()
  end
}

Don't forget chmod a+x /usr/bin/parse_strace.rb

I invoke strace -xx (outputs hex, so the regex matches properly), piping (including STDERR) to my script with 9 as the first arg.

sudo sh -c 'strace -xx -p28223 -s9999 -e write 2>&1 | parse_strace.rb 9'

And, voila, it outputs the process's original STDOUT, newlines, color, and all!

Answer 5

If you want to get stderr and stdout you can just run this:

tail -f /proc/<pid>/fd/*

Answer 6

You always can launch a process whith nohup and &

nohup rsync source_file dest_file &

Then, you'll can check the progress from any tty with:

tail -f nohup.out

This works fine to me.

Answer 7

I'd advise to make a named pipe (mkfifo) and then write to that file. Then, read from it. You can always do that with things like tail, to minimize output, etc. Whenever you clear the pipe (read from it), it gets cleared, so the output is not preserved.

The other option would be to write everything to a file (much like a logfile) and then analyze it an any time. This would be the preferred action, if you want to preserve all output.

Answer 8

1. You might be able to peek at the remote screen using ssh localhost 'DISPLAY=:0.0 xwd -root' | xwud -scale where localhost is to be replaced by your remote server login credentials and :0.0 with the display number of your GUI.

2. Use x11vnc, which is a VNC server for your on screen X-session.

3. When running on one of the 6 virtual consoles try sudo setterm -dump 2 -file /dev/stdout, where you replace 2 with the appropriate vc.

Answer 9

A very simple to get the output would be capturing your output to a file and tailing that file.

IF DO:

./ipcheck

INSTEAD DO:

./ipcheck > [replacewithyourfilename]

This would create an output file where your script is located. Then from any other bash shell you can simply tail the file:

tail [replacewithyourfilename] -f

Answer 10

I wrote catp to do this.

It's basically strace with only tracee's output extracted.

Usage: catp $PID

Answer 11

In bash 1:

any_command_you_want > my_output.txt&

In bash 2:

cat my_output.txt

Answer 12

If you have the commercial LSF from IBM as job scheduler, then you have also the option

bpeek displays the stdout and stderr output of an unfinished job

bpeek [-f] [-q queue_name | -m host_name | -J job_name | job_ID |
"job_ID[index_list]"]

Answer 13

Simple solution using strace, pgrep, grep and sed filtering.

• create bash script

  nano ~/.local/bin/strace_cout
  

• copy paste

  #/usr/bin/bash
  PID_CUR=$(pgrep $1)
  STREAM=1 # 1 = stdout 2 = stderr
  strace -p$PID_CUR -s9999 -e write 2>&1 \
   | grep --line-buffered -E "^write\($STREAM," \
   | sed -u -e 's/^[^"]*"//' | sed 's/\\n"[^"]*$//'
  

• Example use:

  ./strace_cout name_of_process