Print process output line by line

Question

Read process output line by line gives some partial ideas about how to deal with sub-process output in an intelligent manner, but no example and no working solution. I have a couple of examples, one of them works (badly) and the other doesn't.

To invoke the code: save the Emacs Lisp snippets as code1 and code2 then make them executable with chmod +x code* and run, e.g. ./code1 test.tex.

test.tex

\documentclass{article}
\begin{document}
Emacs compilation test.
\end{document}

this prints output from the inferior process all at once

#!/usr/bin/emacs --script

(call-process "pdflatex" nil '("*scratch*" t) nil (car argv))
(set-buffer (get-buffer "*scratch*"))

(prin1 (buffer-string))

(setq argv nil)

;;; Local Variables: 
;;; mode:emacs-lisp 
;;; End: 

This does not print output from the inferior process at all

#!/usr/bin/emacs --script

(defun my-process-filter (proc string)
  (print "%s\n" string))

;; my understanding from comments on the question linked above
;; is that this would cause the output of the inferior process
;; to print out line by line.  It doesn't seem to work though.
(set-process-filter
 (start-process "tex" "*Messages*" "pdflatex" (car argv))
 #'my-process-filter)

(setq argv nil)

;;; Local Variables: 
;;; mode:emacs-lisp 
;;; End: 

---

tl;dr

Another "solution" that serves as an interactive "proof of concept" is to run M-x shell followed by pdflatex test.tex. Output from pdflatex is printed line by line. How to achieve the same result non-interactively?

Answer 1

Emacs can invoke processes either synchronously or asynchronously. In the former case, the calling program has to wait for the process to terminate before processing the entire output in one go (whether character-wise, line-wise, or other-wise is up to the program). In the latter case, the program can still wait for the process to terminate before processing its output, but the program additionally has the ability to process intermediate output in chunks of any size it likes.

Here's an example of processing entire synchronous output in line-wise fashion:

(require 'cl-lib)                       ; For `cl-oddp'

(defun my-process-line (line)
  "Print LINE if it comprises an odd number."
  (when (ignore-errors (cl-oddp (string-to-number line)))
    (message ">>> %s" line)))

(mapc #'my-process-line (process-lines "seq" "10"))

This generates the following output:

>>> 1
>>> 3
>>> 5
>>> 7
>>> 9

If process-lines is too slow (less likely) or inflexible (more likely), you can achieve the same effect with call-process:

(mapc #'my-process-line
      (split-string (with-output-to-string
                      (call-process "seq" nil standard-output nil "10"))
                    "\n" t))

Equivalently, with slightly better performance due to less GC:

(with-temp-buffer
  (call-process "seq" nil t nil "10")
  (goto-char (point-min))
  (while (not (eobp))
    (my-process-line
     (buffer-substring-no-properties (point) (line-end-position)))
    (forward-line)))

Here's the same example, but invoked asynchronously:

(defun my-sentinel (proc _msg)
  "Process entire output of PROC line-wise."
  (when (and (eq (process-status proc) 'exit)
             (zerop (process-exit-status proc))
             (buffer-live-p (process-buffer proc)))
    (with-current-buffer (process-buffer proc)
      (mapc #'my-process-line (split-string (buffer-string) "\n" t))
      (kill-buffer))))

(let ((process-connection-type nil))
  (set-process-sentinel (start-process "my-proc" " *my-proc*" "seq" "10")
                        #'my-sentinel))

Equivalently, using the Emacs 25 function make-process:

(make-process :name "my-proc"
              :buffer " *my-proc*"
              :command '("seq" "10")
              :connection-type 'pipe
              :sentinel #'my-sentinel)

Here's the same example, but with the output processed in chunks as it comes in, rather than after it's complete:

(defun my-filter (proc str)
  "Process each line produced by PROC in STR."
  (when (buffer-live-p (process-buffer proc))
    (with-current-buffer (process-buffer proc)
      (insert str)
      (goto-char (point-min))
      (while (progn (skip-chars-forward "^\n")
                    (not (eobp)))
        (my-process-line (delete-and-extract-region (point-min) (point)))
        (delete-char 1)))))

(let* ((process-connection-type nil)
       (proc (start-process "my-proc" " *my-proc*" "seq" "10")))
  (set-process-filter   proc #'my-filter)
  (set-process-sentinel proc #'my-sentinel))

Equivalently, using make-process:

(make-process :name "my-proc"
              :buffer " *my-proc*"
              :command '("seq" "10")
              :connection-type 'pipe
              :filter #'my-filter
              :sentinel #'my-sentinel)

As lunaryorn mentioned in a comment, the reason your sample asynchronous script fails to print anything is because it exits before being given the chance to accept output from its subprocess; you need to block somehow until the process has terminated. One way to do this is to use call-process et al. synchronously, but then you forgo the ability to process output lines as they come in. Another way is to block until the asynchronous subprocess is done:

(let* ((process-connection-type nil)
       (proc (start-process "my-proc" " *my-proc*" "seq" "10")))
  (set-process-filter   proc #'my-filter)
  (set-process-sentinel proc #'my-sentinel)
  (while (accept-process-output proc)))

Equivalently, using make-process:

(let ((proc (make-process :name "my-proc"
                          :buffer " *my-proc*"
                          :command '("seq" "10")
                          :connection-type 'pipe
                          :filter #'my-filter
                          :sentinel #'my-sentinel)))
  (while (accept-process-output proc)))

The gory details for all of this and more are documented under (elisp) Processes.

Answer 2

*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (mapc 'print
      (split-string
       (with-temp-buffer
         (call-process "printf" nil t nil "%s\n" "one" "two" "three")
         (buffer-substring-no-properties (point-min)
                                         (point-max)))
       "\n"))

"one"

"two"

"three"

""

("one" "two" "three" "")

ELISP> 

This is all done in a M-x ielm buffer for demonstration. You can use this code anywhere Emacs Lisp code works in Emacs.

with-temp-buffer creates a temporary buffer. The expressions in its body are evaluated in sequence. The temporary buffer will hold the output of the process temporarily and be automatically killed when our Lisp code is done.

I chose to demonstrate with the external printf command but you can use any command you like to produce output lines.

call-process invokes the external printf command to print three words each followed by a newline. The output of the printf command is inserted into the temporary buffer.

buffer-substring-no-properties yields the full contents of the the temporary buffer. This is the return value of the with-temp-buffer form.

split-string breaks the string returned from with-temp-buffer on newline characters. It returns a list of strings with the newlines removed.

At this point we have a list of lines from the process output. You may process this list in any way Emacs Lisp can process lists. I chose to quote and print each output line separated by empty lines.

mapc applies the function from its first argument to the list in its second argument. Here I use the print function to be applied to the list. Here the list is being returned from the split-string function. mapc returns the original list unmodified.

print inserts its output into the IELM buffer as a side effect. The output can be redirected to other buffers or processed in other ways, but that would make for an unnecessarily complex demonstration code.