grep a pattern across huge fileset without GNU parallel

Question

I want to grep a pattern across 10M files really fast in a 36 core machine i tried this

find . -name '*.xml' -type f | xargs  -P 20 grep "username" >> output

But i am getting some other results in between.

Is there any better way to do this ?

Thanks in advance.

What are the "other" results and what are you looking for? The line content? The filename? What is your expected output? — bu5hman, Nov 08 '19 at 12:04
You're working on XML files? What is it that you are trying to extract from these? Can you give examples of actual data? XML documents are best parsed using XML parsers and queried using e.g. XPath queries, not by line-oriented tools like grep. — Kusalananda, Nov 08 '19 at 12:10
CPU is likely not the bottleneck if you're grepping for fixed strings. So parallelising is more likely to harm than help. See also Grep in couple thousands files and Recursive grep vs find / -type f -exec grep {} ; Which is more efficient/faster? — Stéphane Chazelas, Nov 08 '19 at 13:09
How are files stored? Redundant RAID arrays? HDDs or SSDs? Can you arrange for the output to be stored on different HDDs? — Stéphane Chazelas, Nov 08 '19 at 13:17
@user3398900 So using 20 separate processes that all read off the one disk would possibly saturate the disk I/O. It's disk-bound rather than CPU bound. — Kusalananda, Nov 08 '19 at 13:51

Stéphane Chazelas · Accepted Answer · 2019-11-08T13:55:05.470

Given that your data is on non-RAIDed HDDs, I doubt you'll get better performance from parallelizing, the bottle neck is most likely to be I/O, not CPU.

LC_ALL=C grep -rwF --include='*.xml' username . > /on/some/other/disk/output

May be close to the best you can achieve.

To parallelize, you'd want to do it as:

LC_ALL=C find . -name '*.xml' -type f -print0 |
  LC_ALL=C xargs -r0P20 -n 1000 grep -HFw --line-buffered username > output

assuming there's no output line (input line + file pathname) longer than 4KiB, and note the lines of all 20 concurrent greps will end up interleaved.

See:

for details.

score 0 · Answer 2 · answered Nov 08 '19 at 12:12

0

I think it's because of the pattern being too general. Usually the command line utilities print errors to the stderr or the terminal. They shouldn't get in the output file.

answered Nov 08 '19 at 12:12

saga

1,401

Ok updated the one i have tried is there any better way to do it ??? – user3398900 Nov 08 '19 at 12:18

bu5hman · Answer 3 · 2019-11-08T13:12:15.477

If you are grepping xml files in this way then your search will return the entire line containing the search string and, if the xml file has no newlines, the entire file contents. Quite a lot of "other" across 10M files.

As per @Kusalananda comment it isn't good practice to brute force an xml with grep and an xml parser e.g. xmllint is a better tool, however, if you insist ......

Check man for grep and read up on the -o option to restrict the returned value and use regex that defines the whole length of the match you are looking to find.

If username is an attribute

grep -o 'username="[^"]*"'

Or better

xmllint --xpath "//@username"

If username is a node then something like

grep -o "username>[^<][^<]*"

Or better

xmllint --xpath "//username"

With either of the xmllint queries, just wrap the query in string() to extract the attribute or node text.

xmllint --xpath "string(//username)"
xmllint --xpath "string(//@username)"

grep a pattern across huge fileset without GNU parallel

3 Answers3