0

My situation is:

I have many files with a name timestep1_i for many values of i and I am trying to combine all the files for timestep1 with a program called combinefiles. I want to do it in parallel (for timestep2 timestep3 ...) using find | xargs.

To do so I have a bash function called combine_and_move:

function combine_and_move {
  file_name1=`echo $1 | sed 's/timestep1_0/timestep1/'`
   echo "*** file $file_name1 ***"
   $UTIL_DIR/combinefiles $file_name1 &&  mv "$file_name1"_* uncombined_files
}
export -f combine_and_move

This function receives a filename, then calls the program that will combine all the files together and (it is supposed to) moves the already combined files to another directory. This last part is supposedly indicated by the &&.

Now I call the function using find | xargs like this:

find . -maxdepth 1 -name 'timestep*_0' -print0 | xargs --replace=@ -0 -P 64  bash -c 'combine_and_move $"@"' _ &

On the understanding that the process may be killed randomly by the system administrator.

The function sort of works, but sometimes if the process is killed, I find myself with uncombined files that were moved to the uncombined_files folder. Apparently meaning that the second command ran even though the first one did not succeed.

What am I doing wrong? What can I change so that the function combine_and_move only move the files if the combinefiles program was succesful?

aramirezreyes
  • 101
  • I'm pretty sure mv would never run if combinefiles didn't complete successfully. An issue with your globs is more likely: for instance, if you have two files named timestep1_0 and timestep1_0_0, combine_and_move timestep1_0_0 will execute mv ./timestep1_0_0 uncombined_files while combine_and_move timestep1_0 will execute mv ./timestep1_0 ./timestep1_0_0 uncombined_files. Note that timestep1_0_0 is moved in both cases. – fra-san Sep 10 '20 at 19:03
  • (Also, since you are not quoting your variables, a file named timestep*_0 will likely wreak havoc). – fra-san Sep 10 '20 at 19:23

1 Answers1

0

When combinefiles completes - either successfully, or if it's killed, it should produce a return code. Test for this return code before executing the mv command.

For example, from the shell, you could run "echo $?" to see the return code of the previous command. For a successful run, it would show 0, but for a command that was stopped via CTRL-C (SIGINT), it would show 130. There are lots of exit codes you could test for, and given their value, then decide to run mv or not. You can even specify your own exit codes.

Perhaps these links will help:

https://tldp.org/LDP/abs/html/exit-status.html

https://www.cyberciti.biz/faq/bash-get-exit-code-of-command/

mikem
  • 845