Asynchrony when collecting output in a process buffer?

Question

I want to capture output from a process buffer. This is not a duplicate of How do I reliably get output from a process? because that questioner is presupposing comint and an architecture for a function that returns the output. My question is at a lower level.

My theory was to get the value of point before sending a command string to the process buffer, call that value p1; send the command; get the new value of point, call it p2, then call (buffer-substring p1 p2) to get the output.

My theory works when sequentially executed in the REPL in the *scratch* buffer. The commands don't work when sequentially executed in a progn or in a let*.

I started off doing my theory in a let*. I'm using shell just as an example, the real target is another interpreter, so async-shell-command won't solve my real problem.

(with-current-buffer "*shell*"
  (let* (( p1   (point))
         ( _    (process-send-string "*shell*" "date\n"))
         ( p2   (point))
         ( text (buffer-substring-no-properties p1 p2)))
    `((p1 ,p1) (p2 ,p2) (text ,text))
    ))

((p1 326) (p2 326) (text ""))

and it didn't work. I am evaling everything in the *scratch* buffer with C-uC-xC-e, using *scratch* like a traditional lisp REPL.

I unrolled everything into separate calls setting global variables to investigate why it didn't work, and discovered that it DOES work if I run the commands sequentially in the REPL:

Here, I execute each line independently and show the output:

(setq p1 (with-current-buffer "*shell*" (point)))

275

(process-send-string "*shell*" "date\n")

nil

(setq p2 (with-current-buffer "*shell*" (point)))

326

(with-current-buffer "*shell*"
  (buffer-substring-no-properties p1 p2))

"date Sat Sep 15 10:47:38 PDT 2018

~ ⌚ 10:47:38 $ "

That's great. My theory is not completely crazy. Now, package that all back up in a let*, just in baby steps. First, just put it all in a progn, so that instead of four commands, I will just have one:

(progn (setq p1 (with-current-buffer "*shell*" (point)))
       (process-send-string "*shell*" "date\n")
       (setq p2 (with-current-buffer "*shell*" (point)))
       (setq text (with-current-buffer "*shell*"
                    (buffer-substring-no-properties p1 p2)))
       `((p1 ,p1) (p2 ,p2) (text ,text))
       )

((p1 473) (p2 473) (text ""))

Now it doesn't work.

To summarize: the commands work when sequentially executed in the REPL in the *scratch* buffer. The commands don't work when sequentially executed in a progn or in a let*.

There is some asynchrony going on that I don't understand and don't know how to manage. The first call to point, the call that sets the value of p1, somehow happens after the call to process-send-string, even though I explicitly call point before calling process-send-string. p1 has the same value before and after the call of process-send-string, and that value is at the end of the buffer. There is no way that the first call of point can clairvoyantly know what the value of point will be after processing the command, so the call of point must happen after the call of process-send-string, or else some magical updating is going on, or some kind of lazy evaluation.

My questions are:

1. (this is what I really want) What is the right way to collect output from a command buffer?

2. Out of curiosity, what about this hidden asynchrony? How and where is it documented? How could I work my way around it?

Answer 1

Emacs reads output from processes only when it is idle, or certain functions are called.

The first condition occurs, when you evaluate the expressions one after the other and that's why you notice the process's output in this case.

One of the mentioned functions is accept-process-output, which, as its name suggest, makes Emacs read some output from processes. And if you insert a call to that function in-between sending to the process and reading the value of point the second time, you should see it having been changed.

But in order to reliably capture a process's output you additionally need some form of synchronization, i.e. some way of knowing, that the process's output has been fully received, i.e. a protocol.