How to redirect a running process output when the output device is deleted

Question

I was trying to inspect a disowned curl process, which was started in an earlier login. I can see that it (PID 17193) is still running:

$ ps aux | grep curl
17193 jack 7700 3692 S      1:00 20:48  curl https://examples.com/file.ext -O
18124 jack 3576  876 S      0:00 21:18  grep curl

And I was trying to see how far the downloading is going, but the output of it has been deleted:

$ ls -l /proc/17193/fd
total 0
lrwx------    1 jack users           64 May 19 20:50 0 -> /dev/pts/1 (deleted)
lrwx------    1 jack users           64 May 19 20:50 1 -> /dev/pts/1 (deleted)
lrwx------    1 jack users           64 May 19 20:50 2 -> /dev/pts/1 (deleted)
lrwx------    1 jack users           64 May 19 20:50 3 -> socket:[2343571]
l-wx------    1 jack users           64 May 19 20:50 4 -> /dir/file.ext

What I should do to see the output? Thanks!

After reading @JdeBP accepted answer, I am not sure anymore :) However, this answer in another post shows a way to do that with sudo strace -p17193 -s9999 -e write (tested). Although your process must be death by now, it may help others reaching out. — rellampec, Oct 23 '22 at 12:21

score 2 · Answer 1 · answered May 19 '18 at 14:11

2

Seeing is easy:

cat /proc/17193/fd/1 ; tail -n 0 -f /proc/17193/fd/1

Real redirection is complicated. You could attach a debugger to the running process, close the fd and open it for a new file. But with active networking I guess this has to be done quickly in order to avoid connections being broken.

answered May 19 '18 at 14:11

Hauke Laging

90,279

Does that actually work? If you re-open a /dev/pts/ device for reading like that, do the reads give you the stuff written to the terminal (by an application), or the stuff coming from the terminal (typed by the user)? – ilkkachu May 19 '18 at 14:18
Thanks for answering! I tried this method, but got three lines of errors: 1. cat: can't open '/proc/19927/fd/1': Input/output error; 2. tail: can't open '/proc/19927/fd/1': Input/output error; 3. tail: no files. (19927 is the PID of curl in this time). – jackxujh May 19 '18 at 14:22
@ilkkachu I have to admit that I did not notice that those were ptys. How do you delete a pty? If the controlling terminal of a process disappears then the kernel kills the process... But it probably just sends SIGHUP which may be ignored. – Hauke Laging May 19 '18 at 14:29
@HaukeLaging The system I am using is running BusyBox. And the active ssh session is pty/0 according to $ who. – jackxujh May 19 '18 at 16:55

score 2 · Answer 2 · answered May 20 '18 at 03:47

2

As JdeBP says, this is impossible without attaching as a debugger. Luckily, there exist programs to do exactly that in a fully-automated way. Reptyr is one such program. Just run reptyr 17193, and it will be reconnected to your terminal.

answered May 20 '18 at 03:47

Joseph Sible-Reinstate Monica

3,323

Thanks for the recommendation! I will try this as soon as I get back to the device! – jackxujh May 21 '18 at 02:37

score 1 · Accepted Answer · answered May 19 '18 at 22:23

This is not possible without a debugger.

Your program's standard output and standard error are going to the slave side of a pseudo-terminal whose master side has been closed. There is no way, short of extreme measures such as debugging the process and intercepting system calls, to access its output.

These are the semantics of terminals.

If you hangup the terminal session (i.e. hangup a real terminal's serial device, exit a login session on a kernel virtual terminal device, hangup the master side of a pseudo-terminal as used by SSH/user-space virtual terminals/TELNET/rlogin/X11 terminal emulators) the open file descriptors for processes that still reference that terminal are intended not to cause I/O nor be connected to anything any more. It is the Unix approach to combatting trojan horse programs that used to hang on to open file descriptors to terminal devices and use them to present fake login prompts to the next user of the terminal.

This has long since begotten programs such as tmux, GNU Screen, mosh, or console-terminal-emulator which sit holding the master side of a pseudo-terminal open, emulate a terminal's display and input, and allow disconnectable and reconnectable client/realizer processes (i.e. the "client" parts of tmux and GNU Screen; mosh clients; or console-fb-realizer, console-termio-realizer, or some other such) to render that onto real devices.

So use tmux, GNU Screen, mosh, a nosh toolset's user-space virtual terminal, or some other such system if you want reconnectability.

How to redirect a running process output when the output device is deleted

3 Answers3

Further reading