0

What is the maximum character range of grep in bash? How long can a string be as input to search using the grep command?

KHAN irfan
  • 113

1 Answers1

2

The maximum length of the arguments to the exec() function (used by the shell to start a process) is defined by the value of ARG_MAX in the limits.h header. This value is also available through the getconf command. On my system:

$ getconf ARG_MAX
262144

This means that the complete command line (after all variable substitutions etc. has been processed), including any environment variables and their values, must not exceed 256 KiB for the shell to be guaranteed to accept it.

POSIX guarantees that ARG_MAX is at least 4096.

For a Linux-specific discussion about ARG_MAX, see Stéphane Chazelas' answer to a related question.

For an in-depth discussion about this issue see https://www.in-ulm.de/~mascheck/various/argmax/ (external link).

With grep -f patterns file you may bypass the maximum length of the command line by putting the patterns that you'd like to grep for in the file patterns. The limit to the number of patterns will then be your available RAM rather than a system-imposed limit on the command line length.

Community
  • 1
Kusalananda
  • 333,661
  • 1
    Not exactly. From Linux 3.11, the getconf ARG_MAX reports a quarter of the limit set on the stack size, or 128 KiB if that's less than 512 KiB) – cuonglm Apr 13 '17 at 08:17
  • @cuonglm Is that not the real limit on such a system then? – Kusalananda Apr 13 '17 at 08:18
  • Yes, it's not the real limit. See http://unix.stackexchange.com/a/110301/38906 – cuonglm Apr 13 '17 at 08:25
  • @Kusalananda -f should be the last argument just before the file name? while read line; do grep -i -w -f patterns "$line" 13April_SMILEandFINGERPRINTS.txt; done < 4804Smiles.txt >>4804matchedSMILESandFINGERPRINTS.txt because this is not working for me – KHAN irfan Apr 13 '17 at 11:17
  • If your regex is 256MB there may be additional trouble ahead, unless it's completely trivial. – tripleee Apr 13 '17 at 11:17
  • Option order makes no difference. – tripleee Apr 13 '17 at 11:18
  • @KHANirfan You use a file for the patterns or give them on the command line. From your comment, it's unclear whether you want to match with $line as the pattern, or if that's the filename, and I don't know what the last file on the command line is doing there. You really have to explain what you want to be happening. – Kusalananda Apr 13 '17 at 11:21
  • @Kusalananda $line is the pattern. It only contains tokens/records in the files. when even there is a match in the second file it is stored in output file as complete line. the last file is the output file. – KHAN irfan Apr 13 '17 at 11:47
  • @KHANirfan You should replace the loop with grep -iw -f 4804Smiles.txt 13April_SMILEandFINGERPRINTS.txt >>4804matchedSMILESandFINGERPRINTS.txt. This uses 4804Smiles.txt as the source of the patterns. – Kusalananda Apr 13 '17 at 11:49
  • 1
    Begging for upvotes is strongly discouraged here. Please don't. – tripleee Apr 13 '17 at 13:35