grep: memory exhausted

Question

I was doing a very simple search:

grep -R Milledgeville ~/Documents

And after some time this error appeared:

grep: memory exhausted

How can I avoid this?

I have 10GB of RAM on my system and few applications running, so I am really surprised a simple grep runs out of memory. ~/Documents is about 100GB and contains all kinds of files.

grep -RI might not have this problem, but I want to search in binary files too.

Answer 1

Two potential problems:

• grep -R (except for the modified GNU grep found on OS/X 10.8 and above) follows symlinks, so even if there's only 100GB of files in ~/Documents, there might still be a symlink to / for instance and you'll end up scanning the whole file system including files like /dev/zero. Use grep -r with newer GNU grep, or use the standard syntax:

  find ~/Documents -type f -exec grep Milledgeville /dev/null {} +
  

  (however note that the exit status won't reflect the fact that the pattern is matched or not).

• grep finds the lines that match the pattern. For that, it has to load one line at a time in memory. GNU grep as opposed to many other grep implementations doesn't have a limit on the size of the lines it reads and supports search in binary files. So, if you've got a file with a very big line (that is, with two newline characters very far appart), bigger than the available memory, it will fail.

  That would typically happen with a sparse file. You can reproduce it with:

  truncate -s200G some-file
  grep foo some-file
  

  That one is difficult to work around. You could do it as (still with GNU grep):

  find ~/Documents -type f -exec sh -c 'for i do
    tr -s "\0" "\n" < "$i" | grep --label="$i" -He "$0"
    done' Milledgeville {} +
  

  That converts sequences of NUL characters into one newline character prior to feeding the input to grep. That would cover for cases where the problem is due to sparse files.

  You could optimise it by doing it only for large files:

  find ~/Documents -type f \( -size -100M -exec \
    grep -He Milledgeville {} + -o -exec sh -c 'for i do
    tr -s "\0" "\n" < "$i" | grep --label="$i" -He "$0"
    done' Milledgeville {} + \)
  

  If the files are not sparse and you have a version of GNU grep prior to 2.6, you can use the --mmap option. The lines will be mmapped in memory as opposed to copied there, which means the system can always reclaim the memory by paging out the pages to the file. That option was removed in GNU grep 2.6

Answer 2

I usually do

find ~/Documents | xargs grep -ne 'expression'

I tried a bunch of methods, and found this to be the fastest. Note that this doesn't handle files with spaces the file name very well. If you know this is the case and have a GNU version of grep, you can use:

find ~/Documents -print0 | xargs -0 grep -ne 'expression'

If not you can use:

 find ~/Documents -exec grep -ne 'expression' "{}" \;

Which will exec a grep for every file.

Answer 3

I can think of a few ways to get around this:

• Instead of grepping all files at once, do one file at a time. Example:

    find /Documents -type f -exec grep -H Milledgeville "{}" \;
  

• If you only need to know which files contain the words, do grep -l instead. Since grep will there stop searching after the first hit, it won't have to keep reading any huge files

• If you do want the actual text as well, you could string two separate greps along:

    for file in $( grep -Rl Milledgeville /Documents ); do \
        grep -H Milledgeville "$file"; done
  

Answer 4

I'm grepping a 6TB disk to search for lost data, and got the memory exhausted -error. This should work for other files too.

The solution we came up with was to read the disk in chunks by using dd, and grepping the chunks. This is the code (big-grep.sh):

#problem: grep gives "memory exhausted" error on 6TB disks
#solution: read it on parts
if [ -z $2 ] || ! [ -e $1 ]; then echo "$0 file string|less -S # greps in chunks"; exit; fi

FILE="$1"
MATCH="$2"

SIZE=`ls -l $1|cut -d\  -f5`
CHUNKSIZE=$(( 1024 * 1024 * 1 )) 
CHUNKS=100 # greps in (100 + 1) x 1MB = 101MB chunks
COUNT=$(( $SIZE / $CHUNKSIZE * CHUNKS ))

for I in `seq 0 $COUNT`; do
  dd bs=$CHUNKSIZE skip=$(($I*$CHUNKS)) count=$(( $CHUNKS+1)) if=$FILE status=none|grep -UF -a --context 6 "$MATCH"
done