Wanderlust -- how to grep the contents of thousands of emails

Question

Q: How can I grep thousands of emails within Wanderlust while avoiding the dreaded error of "Too many arguments" or finding zero matches (when there should be one or more)?

Wanderlust (and related elmo libraries) support searching the contents of emails by using grep: http://wanderlust.github.io/wl-docs/wl.html#grep . This works well for a small number of emails in the directory to be searched, but does not work when there are thousands of files in the directory -- i.e., zero results are returned even though there should have been one or more hits.

score 1 · Accepted Answer · edited May 23 '17 at 12:38

The limitation of the number of files is due to the number of arguments grep can accept before throwing an error of "Too many arguments". Wanderlust and elmo related libraries are filled with condition-case statements that mask errors and make troubleshooting extremely time consuming. The default configuration for grep uses the function elmo-search-grep-target, which creates a listing of files -- sometimes too many. In researching this answer, I found it helpful to inspect the functions elmo-search-engine-do-search and elmo-map-folder-list-message-locations, the latter of which when supplied with a handy (message ...) told me that the problem was due to "Too many arguments". This error message led me to an answer by Barmar suggesting to use the recursive grep feature and supply a directory, rather than a zillion files. [ https://stackoverflow.com/a/29727258/2112489 ] The upshot is that I had also been wanting to enable recursive grepping, which the default configuration did not provide. So, we create a new function called elmo-search-rgrep-target and add the r argument to the grep search. The method of searching is essentially the same, from with the Summary or Folder buffer, type the letter "g" and then enter into the minibuffer something like [hello-world]/path/to/be/recursively/searched!grep

(defun elmo-search-rgrep-target (engine pattern)
  (let ((dirname (expand-file-name (elmo-search-engine-param-internal engine))))
    dirname))

;;; Setup `elmo-search-engine-alist'
(unless noninteractive
  (or (assq 'namazu elmo-search-engine-alist)
      (elmo-search-register-engine
        'namazu 'local-file
        :prog "namazu"
        :args '("--all" "--list" "--early" pattern elmo-search-namazu-index)
        :charset 'iso-2022-jp))
  (or (assq 'mu elmo-search-engine-alist)
      (elmo-search-register-engine
       'mu 'local-file
       :prog "/path/to/executable/mu"
       :args '("find" pattern "--fields" "l" "--muhome=/path/to/muhome/.mu")
       :charset 'utf-8))
  (or (assq 'grep elmo-search-engine-alist)
      (elmo-search-register-engine
       'grep 'local-file
       :prog "grep"
       ;; :args '("-l" "-e" pattern elmo-search-grep-target)
       :args '("-rle" pattern elmo-search-rgrep-target))))

I've opened up an issue on Github for Wanderlust, but am unsure whether anyone would be interested in this solution given that it could be viewed as relating more to the elmo related libraries. https://github.com/wanderlust/wanderlust/issues/140

Wanderlust -- how to grep the contents of thousands of emails

1 Answers1