The forbidden question about defmacro: when does the expansion occur?

Question

I have recently learned (in shock!) that it is impossible to predict the moment a macro defined via defmacro is expanded (i.e., its code is run in order to generate the expanded form which will in turn run when the macro is invoked). For instance, when you run the macro

      (defmacro report-case-fold-search ()
        `(message "case-fold-search is currently %s" ,case-fold-search))

there is no way to predict the output since it depends on the value of case-fold-search at the moment the macro was expanded and you cannot predict that!

After swallowing that information (a bit reluctantly), out of sheer curiosity I started to experiment with it trying to understend how it all works. The first question that comes to mind is this: OK, if a macro is expanded ahead of time, how can it possibly know in advance what the values of its parameters are? I explain: suppose the expansion of a macro depends both on case-fold-search and on a parameter p such as

      (defmacro report-case-fold-search-and-parameter (p)
        `(message "case-fold-search is currently %s and the parameter given is %s" ,case-fold-search ,p))

(notice that the last occurence of p is evaluated, so the macro needs to know the value of p in order to be expanded).

When I run it with various values of case-fold-search and p, I notice that any change in p is immediately reflected in the output, but changes in case-fold-search are not.

To see for yourself try repeatedly running the following in random order:

(progn
  (setq case-fold-search t)
  (report-case-fold-search-and-parameter 1))

(progn
  (setq case-fold-search t)
  (report-case-fold-search-and-parameter 2))

(progn
  (setq case-fold-search nil)
  (report-case-fold-search-and-parameter 1))

(progn
  (setq case-fold-search nil)
  (report-case-fold-search-and-parameter 2))

This shows that at least part of the work leading to the expansion of this macro is taking place after the parameter p is provided, hence at run-time. However, this late expansion is not updating the value of case-fold-search or at least not doing it immediately.

Even though I was previously frowned upon for asking about the time a macro expansion takes place, I'd like to know what is going on? In the last part of the macro definition above, namely around ,case-fold-search ,p)), can it possibly be that the evaluation of case-fold-search takes place before the macro is invoked, while the evaluation of p is done afterwards? To put it another way, it looks like the evaluation of the form

    `(message "case-fold-search is currently %s and the parameter given is %s" ,case-fold-search ,p))

is partially performed at some earlier time and then concluded after the value of p is known. Is that right?

Is your example phony, for some purpose of exposition, or is this the kind of macro you really want to create? (If the latter, why a macro? Just use `defun` instead.) — Drew, Jan 19 '23 at 18:08
@Drew, please se my comment below my question in https://emacs.stackexchange.com/questions/75331/how-to-distinguish-from-within-a-macro-definition-whether-a-parameter-is-a-const?noredirect=1#comment123424_75331 — Ruy, Jan 19 '23 at 21:04
Linking to the preceding question for context: https://emacs.stackexchange.com/questions/75338/bug-in-defmacro — phils, Jul 06 '23 at 00:53

score 4 · Accepted Answer · answered Jan 19 '23 at 19:37

This is easily explainable with the example given, particularly the third one:

(progn
  (setq case-fold-search nil)
  (report-case-fold-search-and-parameter 1))

Suppose this code is in your *scratch* buffer and the cursor is immediately to the right of the final close–paren when you hit C-j to run eval-print-last-sexp. This will evaluate the code and then print the following result:

"case-fold-search is currently t and the parameter given is 1"

This is where people wig out and complain that macros don’t make any sense. Clearly they set case-fold-search to nil, but here it is reported as t!

Well, if we follow the correct order of operations, all becomes clear.

First, we must perform macro expansion. Macro expansion walks the code looking for macros, and then replaces each macro by the result of calling that macro’s function definition. progn isn’t a macro, and neither is setq, so those stay as they are. But report-case-fold-search-and-parameter is a macro. We must now call it with one argument (which happens to be the very simple expression 1, but it is instructive to see what happens if it is instead something more complex like (+ 2 2)), expecting to get back some new code. Eval then inserts this new code in place of the macro call, like this:

(progn
  (setq case-fold-search nil)
  (message "case-fold-search is currently %s and the parameter given is %s" t 1))

You might already begin to see what is going on.

Eval now evaluates the code, running it to produce some value. The progn is a special form that runs each of it’s arguments in strict sequence, so the call to setq will be evaluated before the call to message. setq changes the value of case-fold-search to nil. Calling message causes all % expressions in the message to be substituted with values from the rest of the arguments, which in this case are the values t and 1. It then returns the resulting string. The string correctly has a t and a 1 substituted into it.

Your confusion comes from running the code in a single pass in your head. You imagined that setq set the value before Emacs noticed that report-case-fold-search-and-parameter in the next expression was a macro. The reality is that all macros are resolved before any evaluation takes place.

You can debug situations like this by passing code to the macroexpand line of functions. In particular, try running this:

(macroexpand-all '(progn
                    (setq case-fold-search nil)
                    (report-case-fold-search-and-parameter (+ 2 2))))

Hopefully it explains phils answer to your prior question as well. As he says, the code may be loaded (and macros expanded) long before it is run. In fact, while Emacs ships with over a million lines of Elisp code spread across many files, it doesn’t actually read those files directly. Instead, they are loaded during the build process, then the resulting memory image is saved to disk. Then when you run Emacs it loads the memory image rather than opening and reading from many thousands of smaller files. Thus the macros were expanded long ago, on a completely different computer. — db48x, Jan 20 '23 at 02:02

The forbidden question about defmacro: when does the expansion occur?

1 Answers1