Executing Command in Pipes

Question

I need to understand why I am not able to execute the following.

ls -d /data/VT60000* | xargs -IDIRECTORY find DIRECTORY -type f -iname RJ\*.xml | xargs -IFILE sh -c ' XML=$( xmlstarlet fo -D "FILE" );
EXTRACT=$( xmlstarlet sel -t -c "concat('JD',/root/item1,/root/item2)" <<< "$XML" );
echo '$EXTRACT'; '

I have a bunch of folders which contains thousands XML files. I would like to extract information from these files and then manipulate them for later processing. The reason why I am piping them is because I would like to execute multiple processes (-P)

The second xmlstarlet fails to execute and I fail to understand why.

The question is how can i achieve this.? To get a value in $EXTRACT.

Thanks for the assistance in advance.

What happens if that xargs sh -c 'XML=$(xmlstarlet)' runs multiple times? Should the later calls write over the values of the earlier calls (like would happen if you assigned them to a variable one by one), or should the values be concatenated, or ...? — ilkkachu, Jan 20 '18 at 18:52
just some clarification.. what do you mean multiple times.? could that command process the file twice.? — Noel Alex Makumuli, Jan 20 '18 at 20:20

score 1 · Answer 1 · answered Jan 20 '18 at 18:50

The main issue here seems to be that you're assigning the variable XML inside a shell called from xargs, but you try to use the variable from the main script. That won't work, as there's no way for the subprocess to pass the variable directly to the parent shell.

Instead, you'll need to either print out the value, or use it in the same shell.

From the script, it looks to me that what you want to do is this:

xmlstarlet fo -D "FILE" | xmlstarlet sel -t -c "concat('JD',/root/item1,/root/item2)"

or with the sh -c

sh -c 'xmlstarlet fo -D "FILE" | xmlstarlet sel -t -c "concat('\''JD'\'',/root/item1,/root/item2)"'

Though the quoting gets awful, so it might be worth it to put that inside another script, or in a variable (either just the string, or the whole thing in an array).

Also, I'm not sure what you want to do with the output of the xmlstarlet on the first line, if it gets called multiple times.

Instead of ls /data/VT60000* | xargs, you could use printf "%s\0" * | xargs -0 to avoid the issues with problematic file names.

score 0 · Answer 2 · answered Jan 21 '18 at 04:59

echo '$EXTRACT' wont work because of incorrect use of single quotes (single quotes do not allow substitutions) and incorrect nested single quotes (single quotes inside single quotes). It should be double quotes inside single quotes. I cannot test your codes, so I wrote this and tested:

ls -d $HOME | \
   xargs -IDIR find "DIR" -type f -name '*\.conf' | \
   xargs -IFILE sh -c 'l=$(wc -l "FILE"); echo "$l"'

Executing Command in Pipes

2 Answers2