1

How do you open a file descriptor and echo it to a terminal while it's being written to from a process?

I have a backup program, Duplicity, that writes its logs to a file descriptor specified by the --log-fd=16 parameter.

Sure enough, if I run lsof -p <duplicity PID> I see:

python2 9224 myuser    0r      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    1w      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    2w      CHR                1,3      0t0         6 /dev/null
python2 9224 myuser    3u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    4u     unix 0x0000000000000000      0t0    158199 type=STREAM
python2 9224 myuser    5u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    6u  a_inode               0,11        0      7005 [eventfd]
python2 9224 myuser    7r      DIR                8,3     4096  22414346 <some random file being accessed during the backup>
python2 9224 myuser    8r      CHR                1,9      0t0        11 /dev/urandom
python2 9224 myuser   15r     FIFO               0,10      0t0    157054 pipe
python2 9224 myuser   16w     FIFO               0,10      0t0    157054 pipe

However, if I try to open the file descriptor in Python, I get an error:

>>> import os
>>> os.fdopen(16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor

Why is this? How do I read the the file descriptor?

Cerin
  • 1,678
  • 2
    File descriptors are local to a process. Every process has their own fd 0, 1, and 2, and so on. It's just an index into a kernel structure in the process' execution context. – Kusalananda May 24 '17 at 19:55
  • You could possibly use some kernel tracing tool e.g. SystemTap to snoop on the file descriptor calls of that process (but why?) – thrig May 24 '17 at 20:32
  • @thrig, I'm using Deja-Dup for backups, which calls duplicity behind the scenes. Unfortunately, Deja-Dup outputs zero progress information, even though duplicity supports it, so I'm trying to view duplicity's logs to track progress. – Cerin May 25 '17 at 00:01
  • So, this is basically an issue relating to micro-managing a backup? Don't you trust duplicity to do its job? – Kusalananda May 25 '17 at 18:21
  • @Kusalananda, No, I want to see progress, so I know if there's 1 hour or 8 hours remaining... – Cerin May 25 '17 at 22:04

2 Answers2

1

I believe duplicity's --log=fd option is intended for complex pipelines where you want to separate stderr and stdout from your log.

This answer to this question gives an example. Here's a simple example:

#!/bin/sh
# Generate output on three different fds
echo hello >&3
echo world >&2
echo today >&1

And when executed like this,

./foo 2> 2.log 3> 3.log 1> 1.log

Results in

$ cat 1.log 2.log 3.log
today
world
hello
rrauenza
  • 734
0

Use strace (trace system calls and signals).

Usage:

sudo strace -p <PID of writing process> -s 9999 -e write=<corresponding FD>

From manual page:

       -p pid      Attach to the process with the process ID pid and begin tracing.  The trace may be terminated
                   at any time by a keyboard interrupt signal (CTRL-C).  strace will respond by detaching itself
                   from  the  traced process(es) leaving it (them) to continue running.  Multiple -p options can
                   be used to attach to many processes in addition to command (which is optional if at least one
                   -p option is given).  -p "`pidof PROG`" syntax is supported.

   -s strsize  Specify the maximum string size to print (the default is 32).  Note that  filenames  are  not
               considered strings and are always printed in full.

   -e read=set
          Perform a full hexadecimal and ASCII dump of all the data read from file descriptors listed in the
          specified set.  For example,  to  see  all  input  activity  on  file  descriptors  3  and  5  use
          -e read=3,5.   Note  that  this  is independent from the normal tracing of the read(2) system call
          which is controlled by the option -e trace=read.

   -e write=set
          Perform a full hexadecimal and ASCII dump of all the data written to file  descriptors  listed  in
          the  specified  set.   For  example,  to  see  all output activity on file descriptors 3 and 5 use
          -e write=3,5.  Note that this is independent from the normal tracing of the write(2)  system  call
          which is controlled by the option -e trace=write.

Ref: https://man7.org/linux/man-pages/man1/strace.1.html

Toran Sahu
  • 1
  • The question is asking how to read from a (known) file descriptor open in another process. strace can’t help with that... – Stephen Kitt Nov 06 '20 at 21:50
  • What's blocking someone from using strace to read data from a FD if PID# and FD# are already known? – Toran Sahu Nov 07 '20 at 04:04
  • You can’t ask strace to read on its own; all it can do is show you the data that’s read by the process being traced, it won’t issue extra reads by itself. In the context of this question, the process under consideration isn’t reading, it’s writing. -e write might do what the OP is after, but -e read won’t. – Stephen Kitt Nov 07 '20 at 08:02
  • Got it. The answer could have been more precise and descriptive. I thought the provided description of read and write would be intuitive and enough. I'll update the answer. Thanks for the correction/suggestion. – Toran Sahu Nov 08 '20 at 10:09