byte-compile and lexical-binding

Question

Could someone in the know please help me understand what's going on here:

;; -*- lexical-binding: t; -*-
(let* ((a 0))
  (let* ((a 1))
    (funcall (byte-compile (lambda () a)))))
;;=> 0

wat? Not only that, but with lexical-binding off this actually returns 1. I even Edebug stepped the byte-compile and it definitely sees closure with a correctly shadowed and holding 1. Do I misunderstand scoping rules here or how byte-compile works?

PS: or is that an artifact of eval-last-sexp somehow? Because I just tried this and it worked as I expected:

(lexical-let* ((a 0))
  (lexical-let* ((a 1))
    (funcall (byte-compile (lambda () a)))))

---

EDIT Yet another unexpected quirk that maybe related.

Let's define a function that checks if lexical-binding is enabled. Then define a macro that inserts that check before doing whatever.

;; -*- lexical-binding: t; -*-

(defun test-for-lexical-binding ()
  (unless lexical-binding
    (error "lexical binding disabled!")))

(defmacro foo-macro ()
  `(progn
     (test-for-lexical-binding)
     42))

Let's now use that macro in foo.el.

;; -*- lexical-binding: t; -*-

(foo-macro)

You can byte-compile-file fine, but the moment you load the byte-code the check fails:

(byte-compile-file "foo.el" 'load)
;;=> lexical binding disabled!

You can easily verify with (eval-when-compile lexical-binding) that lexical-binding is indeed set when you compile, but nil when the code actually runs.

Answer 1

The lexical-binding is a setting that by-and-large operates at compile-time and not at run-time (more specifically, it applies to code that's being manipulated rather than code that's being executed). So at run-time, better not pay too much attention to it (its value will depend on the buffer that happens to be current when the code is executed) unless you're manipulating code (in which case you will want to consult lexical-binding to determine how that code should be handled).

You can easily fix your test by moving it to macro-expansion-time: macros can consult lexical-binding in order to know whether the code they emit will be interpreted in the lexbind version of Elisp or in the dynbind version of Elisp.

Answer 2

GNU Emacs 26.1 (build 2, i686-pc-linux-gnu, GTK+ Version 3.22.30) of 2018-05-29:

The problem is in byte-compile--reify-function within byte-compile.

In an edebug session byte-compile--reify-function transforms
'(closure ((a . 1) (a . 0) t) nil a)
into
(lambda nil (let ((a (quote 1)) (a (quote 0))) a))
which is wrong.

(funcall '(closure ((a . 1) (a . 0) t) nil a)) gives 1 which shows that the lexical environment is looked up in the sense of assoc, i.e., the first match wins.

Opposed to that, the last binding for a is relevant in the let-form.

---

Since it is let and not let* a simple fix would be to reverse the order of the bindings in the let-form. Note that such a simple fix works for your case with constant expressions but not for the general case where the expressions are evaluated themselves. The evaluation order of the expressions is changed by the fix which can be disastrous.

The original version of byte-compile--reify-function is as follows.

(defun byte-compile--reify-function (fun)
  "Return an expression which will evaluate to a function value FUN.
FUN should be either a `lambda' value or a `closure' value."
  (pcase-let* (((or (and `(lambda ,args . ,body) (let env nil))
                    `(closure ,env ,args . ,body))
                fun)
               (preamble nil)
               (renv ()))
    ;; Split docstring and `interactive' form from body.
    (when (stringp (car body))
      (push (pop body) preamble))
    (when (eq (car-safe (car body)) 'interactive)
      (push (pop body) preamble))
    ;; Turn the function's closed vars (if any) into local let bindings.
    (dolist (binding env)
      (cond
       ((consp binding)
        ;; We check shadowing by the args, so that the `let' can be moved
        ;; within the lambda, which can then be unfolded.  FIXME: Some of those
        ;; bindings might be unused in `body'.
        (unless (memq (car binding) args) ;Shadowed.
          (push `(,(car binding) ',(cdr binding)) renv)))
       ((eq binding t))
       (t (push `(defvar ,binding) body))))
    (if (null renv)
        `(lambda ,args ,@preamble ,@body)
      `(lambda ,args ,@preamble (let ,(nreverse renv) ,@body)))))

We fix it by removing the last nreverse:

(defun byte-compile--reify-function (fun)
  "Return an expression which will evaluate to a function value FUN.
FUN should be either a `lambda' value or a `closure' value."
  (pcase-let* (((or (and `(lambda ,args . ,body) (let env nil))
                    `(closure ,env ,args . ,body))
                fun)
               (preamble nil)
               (renv ()))
    ;; Split docstring and `interactive' form from body.
    (when (stringp (car body))
      (push (pop body) preamble))
    (when (eq (car-safe (car body)) 'interactive)
      (push (pop body) preamble))
    ;; Turn the function's closed vars (if any) into local let bindings.
    (dolist (binding env)
      (cond
       ((consp binding)
        ;; We check shadowing by the args, so that the `let' can be moved
        ;; within the lambda, which can then be unfolded.  FIXME: Some of those
        ;; bindings might be unused in `body'.
        (unless (memq (car binding) args) ;Shadowed.
          (push `(,(car binding) ',(cdr binding)) renv)))
       ((eq binding t))
       (t (push `(defvar ,binding) body))))
    (if (null renv)
        `(lambda ,args ,@preamble ,@body)
      `(lambda ,args ,@preamble (let ,renv ,@body)))))

You can evaluate the last defun in the *scratch* buffer. Afterwards eval-last-sexp of your first elisp snippet gives 1, as expected.