1

When running ert tests, the seem to be run by default in string< order. For instance running tests

(require 'ert)
(ert-deftest t1111 () (print 'AAAA))
(ert-deftest t3333 () (print 'BBBB))
(ert-deftest t2222 () (print 'CCCC))
(ert-run-tests-batch :new)

will print

;; Actual output              ;; Wanted output

AAAA                          AAAA

   passed  1/3  t1111            passed 1/3  t1111

CCCC                          BBBB

   passed  2/3  t2222            passed 2/3  t3333

BBBB                          CCCC

   passed  3/3  t3333            passed 3/3  t2222

For interpretation it would be helpful to guarantee the order of the tests. How can this be achieved?

NickD
  • 27,023
  • 3
  • 23
  • 42
kdb
  • 1,561
  • 12
  • 21

1 Answers1

1

As far as I could tell, it is not possible to cause the selectors t (universe) or :new to run tests in the order of definition. The (member TEST1 TEST2 ...) selector however executes tests in the order specified.

For my tests I am therefore using one of two workarounds (both written ad-hoc and not tested for balanced parentheses):

1. Wrapper, that records the tests in a list.

(defconst NAMESPACE-test-list nil)
(defmacro NAMESPACE-deftest (name &rest args)
  `(progn
     (ert-deftest ,name ,@args)
     (setq NAMESPACE-test-list (nconc NAMESPACE-test-list (list ',name)))))

;; TEST DEFINITIONS

(ert `(member ,@NAMESPACE-test-list))

2. Scan file for ert-deftest with regular expression search.

;; TEST DEFINITIONS

(ert (cons 'member
       (with-temp-buffer
         (insert-file load-file-name)
         (cl-loop
           while (search-forward-regexp (rx bol "(ert-deftest" (+ blank)
                                            (group-n 1 symbol-start (*? nonl) symbol-end)))
                                        nil t)
           collect (intern (match-string 1))))))

3. Numbered test names

Alternatively, the alphabetic order of the tests can be (ab?)used, e.g.

(ert-deftest NAMESPACE-001:some-function () ...)
(ert-deftest NAMESPACE-002:other-function () ...)
...
(ert :new)

or

(defvar NAMESPACE-test-counter 0)
(defmacro NAMESPACE-deftest (name &rest args)
  `(ert-deftest ,(intern (format "NAMESPACE-%03d:%s" 
                           (cl-incf NAMESPACE-test-counter)
                           name))
      ,@args))
(NAMESPACE-deftest some-function () ...)
(NAMESPACE-deftest other-function () ...)

Both variants have serious downsides however:

  • The first variant causes maintenance issues. Want to reorder the test definitions? Now we also need to renumber all tests, which in turn confuses the version history.
  • The second variant breaks interactive lookup of the test definition from the M-x ert buffer.

If I have just missed a setting, that enables "execute in definition order", I'll be happy to hear of it however.

kdb
  • 1,561
  • 12
  • 21