3

I am running a command with nohup on one of my servers, the output file has become too big and I predict that the process would need an extra month of execution before it completes. Is there a way to change the nohup output file while the process is running?

Zaid Amir
  • 131
  • Maybe you could enlarge the file system. Are you using LVM or raid? – rudimeier Oct 18 '16 at 07:56
  • The process holds an open file descriptor, so modifications to the filesystem won't affect the file to which the file descriptor points (so mounting over it, moving, deletion,... are out of the question). The only things that might work are modifying the file in-place (rewriting the file as empty). Relevant link: http://stackoverflow.com/questions/980283/truncating-a-file-while-its-being-used-linux – orion Oct 18 '16 at 08:06
  • @orion writing the file in-place made matters worse. nothing is being written to the file anymore – Zaid Amir Oct 18 '16 at 08:44

2 Answers2

1

The idea given in one of the comments is generally what's done, even if the link to another question has an erroneous (and unaccepted) answer.

To truncate the file in a POSIX shell use: 

cat /dev/null > _name_of_file_

Not cp as one answer in the link suggests. Of course, you might want to copy the old file to a different filesystem or to a remote location if you want to save it first.

In low-level terms, the difference is that you are opening the existing file and with its inode and truncating it. See http://man7.org/linux/man-pages/man2/open.2.html and the O_TRUNC option.

Some other things that work in special cases. I sometimes check /etc/logrotate.d (or some other suitable log rotation program) to see how that swaps out files. Some commands like apache2 have a reload command that cause it to load a configuration file and in the process reopen log files.

rocky
  • 1,998
  • The problem is when I did that the process stopped writing to the file completely – Zaid Amir Oct 18 '16 at 15:31
  • I suspect you used cp, not `cat > /dev/null. Or in more low-level POSIX libc terms you are doing an open in truncate mode. See http://man7.org/linux/man-pages/man2/open.2.html . From your running processes standpoint it won't know the difference if on the nohub you used ">". – rocky Oct 18 '16 at 16:29
  • @ZaidAmir Another thing you should try to convince yourself this works is to run the thing again (maybe in a test scenario) and see that it works. If not feel free to add more details. The key idea here is to think about and try the remedies before you need them. – rocky Oct 18 '16 at 18:20
  • Well I AM trying to convince my self but its not, as I said now the process stopped writing completely while I can see it working and generating new entries in the database. I cannot restart the task now as that will make me lose about 40 days of processing. – Zaid Amir Oct 19 '16 at 09:14
  • So try it on another computer or in the same computer in a slightly different environment with reduced/test data. If you are spending 40 days of processing, then definitely this is the kind of stuff you should have been considering beforehand. But it is never too late to learn from this and start. – rocky Oct 19 '16 at 10:15
0

Just found this neat little utility that enabled me to redirect the output to another file and everything is working fine now. The utility is called reredirect and can be found in: https://github.com/jerome-pouiller/reredirect/

A simple command like: reredirect -m FILE PID gets the job done.

Zaid Amir
  • 131