I have the following script which gets the absolute paths of some directories and pipes them into GNU parallel for zipping. I keep getting a signal 13 error and I'm not sure why.
find $directory -maxdepth 1 | \
grep -v "^${directory}$" | \
xargs realpath | \
parallel -j 4 zip -r ${new_directory}/{/.}.zip {}
The error is terminated with signal 13; aborting. Sometimes the error is not raised but no zip file is present in the output, new_directory.
Any help would be appreciated.
Before getting to the actual failure that you're having, there are few things you need to fix in your command to handle some corner cases
find "$directory" -maxdepth 1 -mindepth 1 -print0 | \
xargs -0 realpath | \
parallel -j 4 zip -r "$new_directory"/{/.}.zip {}
The changes in find are:
-mindepth 1 - this will exclude your top directory, so you won't need your grep -v command.-print0 - This will solve a possible problem case any of you files include spaces or any escape characters. All the files will be delimited by the the null character instead of a new line.That's why you also need to add the -0 to your xargs command, so it would read the input delimited by the null characters.
In case it doesn't solve your issue, you'll need to debug it further.
First, remove the parallel command altogether and check if the input for this command looks as expected.
If it seems ok, add verbosity to the parallel command to capture exactly what it's doing:
parallel -t -j 4 zip -r ${new_directory}/{/.}.zip {} 2> parallel.log
parallel.log file. Check if the zip commands seem to be generated correctly.zip commands in the log file, try the commands in the file:bash -x parallel.log
At some during the process you'll have to see at what stage you receive the error.
xargs by replacing the -print0 with -exec realpath -z {} +. realpath's -z option tells it to produce NUL-separated output. Also, both $directory and $new_directory need to be double-quoted, and there's no need for the curly braces around $new_directory (see $VAR vs ${VAR} and to quote or not to quote). Finally, GNU parallel needs a -0 or --null option for NUL separated input.
– cas
Jul 06 '23 at 14:41
find "$directory" -maxdepth 1 -mindepth 1 -exec realpath -z {} + | parallel -0 -t -j 4 zip -r "$new_directory"/{/.}.zip {} 2> parallel.log
– cas
Jul 06 '23 at 14:42
xargs, because then realpath will run once for all the files at once instead of for each file. If there are many files and the parallel command will need to wait until all the files are found, it might be better to add -n (max-args) to xargs with some reasonable number.
– aviro
Jul 12 '23 at 12:56
+ instead of \; to terminate the -exec causes find to fit as many filename args into the command line as it can - just like xargs. xargs is useful when you're not using find (i.e. when you have something else generating your list of arguments, e.g. when the args aren't filenames), but when you're using find, it's redundant...-exec is better, find's own built-in xargs. BTW, parallel has its own -N option for max args, similar to xargs' -n option.
– cas
Jul 12 '23 at 14:14
-P / --max-procs option for parallel execution. It's not as flexible as parallel, but it is much simpler to use and should be good enough for zipping some directories. IMO & IME, a task like that isn't well suited to parallel execution, anyway, because it's a mostly I/O-bound job (especially on any modernish CPU, say <= 20 years old)...and running zip multiple times in parallel is just going to make that worse, with each of the jobs contending with each other for read and write disk I/O.
– cas
Jul 12 '23 at 14:21
find "$(realpath "$directory")" -maxdepth 1 -mindepth 1. That wayrealpathis also only called once. – FelixJN Jul 05 '23 at 14:19