2

The problem with reading /proc file system is that it is not atomic. I tried using a library (procfs in Python) that wishes to read it, but it has glitches about 5% of the time because the process that I am after is now gone. So, I wish to copy everything in /proc to another location, say /proc_clone/<GUID>/. A GUID can be generated using the uuidgen command.

The naive approach would be trying to use cp -r, but I am getting a whole bunch of 'permission denied' errors and it is taking more than a second. I am only after a subset /proc/<pid> and even then, only the read-only stuff that can be turned into a string (ultimately I wish to build a single JSON file that contains all of this).

I am looking for a script that would do that for me and is also short, simple, very fast, can deal with processes dying mid-flight by skipping them, however, I would like to avoid writing C code - I would much rather stick to bash, existing utilities and maybe Perl / Python. The goal here is just to take a snapshot of a subset of /proc, not produce the JSON file itself.

I have heard opinions that "one should not ever try to copy /proc but just read from it". Well, the non-atomic nature of it means that one has to throw a bunch of try/catch all over the code when trying to do very simple operations. Using a high level language like Python (what I ultimately want to use) to iterate over /proc is a slow method, which involves both IO and CPU time. This increases the risk of seeing a process die on you (I certainly see it often; I have a script that interrogates /proc every minute and I run into plenty of exceptions). I would like to build a library that outputs a single JSON file that contains cpu and memory usage information about processes in a human-readable format (e.g. using seconds and not jiffies, bytes and not pages, have units in addition to values. I am not concerned that it would take some time to create that file out of a directory dump; I just want to make sure that the snapshot is as accurate as possible. So. if I should not just copy /proc over, then which other method should I use?

Leonid
  • 151
  • 2
    What exactly do you want to do? Copying a complete process will take a lot of time... – vonbrand Apr 26 '13 at 23:57
  • @vonbrand, I believe I stated what I want to do. Say I want to copy reliably and fast all of the /proc/*/cmdline and all of the /proc/*/statm and /proc/*/status files. Can you suggest a tool/script that can do that for me? – Leonid Apr 28 '13 at 16:41
  • I understand the problem. But why can't you catch the exception where the file being read is gone (and terminate whatever you wanted to do there)? – Nils Apr 29 '13 at 13:45
  • @Nils, A) Because I want to minimize the window during which I interact with the /proc file system, B) Initially I was using a nice high-level library for Python called procfs. I would prefer to use that library, except hack it to point to a new root. – Leonid Apr 29 '13 at 17:04
  • So you should catch the exception. I looked into some mount-options (mount --make-private) that looked as if you could do a snapshot of a FS-mount, but this does not seem to work. – Nils Apr 30 '13 at 06:50

1 Answers1

4

You need to have a look at the man pages for proc. Look at /proc/[pid]/* entries and pick which files you want.

You do not want to copy everything. As an example you have /proc/[pid]/mem which is all the virtual memory for a process including shared etc. – i.e. size of all your memory. Further you are not able to read it unless the process own mem or it is ptraced and read with higher privileges. (And then only Virtual Size - /proc/pid/status -> VmSize) – and so on.

Or /proc/PID/exe which simply is a link to the executable, /proc/PID/fd/* which holds file descriptors for stdin, stdout, stderr and other files, pipes etc.

There is a lot of other special files as well, and a lot of links (If you were to follow them all you'll never be done). Read the man mentioned above, documentation from kernel.org, etc.

Further the files are not block-device files. They reside in memory as proc is a memory based filesystem. In one way you can call it a window into the kernel arranged for easy access.

Thus this is also a big advantage when it comes to the reads it looks like you want – as the information reside in memory there is no disk I/O and therefore very fast. Off course if you choose to write the data to disk this is lost – but is still much faster then disk to disk.

Look at it as shared memory and try to not duplicate more then necessary. For your task you would probably also look at files at the root level of /proc. Look at the source code for ps, top, free etc. to get a quick introduction to how other tools do it.

Have written a few other answers that might more or less touch some of the aspects. You might find some of it useful:

Edit:

As for comment. You could do something like:

save_to=/some/path

pushd /proc
for d in [0-9]*; do
    cp "$d/cmdline" "$save_to/$d.cmdline"
    cp "$d/status" "$save_to/$d.status"
done
popd

But I would have used some lower level like e.g. C to parse the files as the method above quite possible not would be quick enough for what one want.

If it is a one time copy every hour or so, OK, but not for e.g. seconds or even a few minutes interval.

Community
  • 1
Runium
  • 28,811
  • so say I want to copy over only /proc/<pid>/cmdline, /proc/<pid>/statm, /proc/<pid?/status reliably and fast. Is there a command-line utility that would help me do that? – Leonid Apr 28 '13 at 16:40