Why does defvar scoping work differently without an initvalue?

Question

Suppose I have a file named elisp-defvar-test.el containing:

;;; elisp-defvar-test.el ---  -*- lexical-binding: t -*- 

(defvar my-dynamic-var)

(defun f1 (x)
  "Should return X."
  (let ((my-dynamic-var x))
    (f2)))

(defun f2 ()
  "Returns the current value of `my-dynamic-var'."
  my-dynamic-var)

(provide 'elisp-dynamic-test)

;;; elisp-defvar-test.el ends here

I load this file and then go into the scratch buffer and run:

(setq lexical-binding t)
(f1 5)
(let ((my-dynamic-var 5))
  (f2))

(f1 5) returns 5 as expected, indicating that the body of f1 is treating my-dynamic-var as a dynamically scoped variable, as expected. However, the last form gives a void-variable error for my-dynamic-var, indicating that it is using lexical scoping for this variable. This seems at odds with the documentation for defvar, which says:

The defvar form also declares the variable as "special", so that it is always dynamically bound even if lexical-binding is t.

If I change the defvar form in the test file to supply an initial value, then the variable is always treated as dynamic, like the documentation says. Can anyone explain why the scoping of a variable is determined by whether or not defvar was supplied with an initial value when declaring that variable?

Here is the error backtrace, in case it matters:

Debugger entered--Lisp error: (void-variable my-dynamic-var)
  f2()
  (let ((my-dynamic-var 5)) (f2))
  (progn (let ((my-dynamic-var 5)) (f2)))
  eval((progn (let ((my-dynamic-var 5)) (f2))) t)
  elisp--eval-last-sexp(t)
  eval-last-sexp(t)
  eval-print-last-sexp(nil)
  funcall-interactively(eval-print-last-sexp nil)
  call-interactively(eval-print-last-sexp nil nil)
  command-execute(eval-print-last-sexp)

Answer 1

Why the two are treated differently is mostly "because that's what we needed". More specifically, the single-argument form of defvar appeared a long time ago, but later than the other and was basically a "hack" to silence compiler warnings: at execution time it had no effect at all, so as an "accident" it meant that the silencing behavior of (defvar FOO) only applied to the current file (since the compiler had no way to know that such a defvar had been executed in some other file).

When lexical-binding was introduced in Emacs-24, we decided to re-use this (defvar FOO) form, but that implies that it now does have an effect.

Partly to preserve the previous "only affects the current file" behavior, but more importantly to allow a library to use toto as a dynamically-scoped var without preventing other libraries from using toto as a lexically scoped var (usually the package-prefix naming convention avoids those conflicts, but it's not used everywhere sadly), the new behavior of (defvar FOO) was defined to apply only to the current file, and was even refined so it only applies to the current scope (e.g. if it appears within a function, it only affects uses of that var within that function).

Fundamentally, (defvar FOO VAL) and (defvar FOO) are just two "completely different" things. They just happen to use the same keyword for historical reasons.

Answer 2

Based on experimentation, I believe the issue is that (defvar VAR) with no init value only has an effect on the library(s) it appears in.

When I added (defvar my-dynamic-var) to the *scratch* buffer, the error no longer occurred.

I originally thought this was on account of evaluating that form, but I then noticed firstly that simply visiting the file with that form present was sufficient; and furthermore that merely adding (or removing) that form in the buffer, without evaluating it was enough to change what happened when evaluating (let ((my-dynamic-var 5)) (f2)) inside that same buffer with eval-last-sexp.

(I have no real understanding of what is happening here. I find the behaviour surprising, but I'm unacquainted with the details of how this functionality is implemented.)

I will add that this form of defvar (with no init value) prevents the byte compiler from complaining about uses of an externally-defined dynamic variable in the elisp file being compiled, but on its own it doesn't cause that variable to be boundp; so it's not strictly defining the variable. (Note that if the variable was boundp then this issue would not occur at all.)

In practice I suppose this is going to work out ok provided that you do include (defvar my-dynamic-var) in any lexical-binding library which uses your my-dynamic-var variable (which presumably would have a real definition elsewhere).

---

Edit:

Thanks to the pointer from @npostavs in the comments:

Both eval-last-sexp and eval-defun use eval-sexp-add-defvars in order to:

Prepend EXP with all the defvars that precede it in the buffer.

Specifically it locates all defvar, defconst, and defcustom instances. (Even when commented out, I notice.)

As this is searching the buffer at call-time, it explains how these forms can have an effect in the buffer even without being evaluated, and confirms that the form must appear in the same elisp file (and also earlier than the code being evaluated).

Answer 3

I cannot reproduce this at all, evaluating the latter snippet works just fine here and returns 5 as expected. Are you sure that you're not evaluating my-dynamic-var on its own? That will throw an error because the variable is void, it hasn't been set to a value and it will only have one if you dynamically bound it to one.