run a single ERT test programmatically

Question

I'm trying to write a piece of code that runs a single ERT test programmatically. In the ERT manual, they show an example that creates a test body and then runs the body, like so:

(ert-deftest ert-test-record-backtrace ()
   (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
    (let ((result (ert-run-test test)))

But... (ert-run-test won't work with a normal ert-deftest name such as the one shown above, ert-test-record-backtrace. If I try doing that I see an error

(ert-run-test 'ert-test-record-backtrace)
Debugger entered--Lisp error: (wrong-type-argument arrayp my-deftest)

The code (ert-run-tests-interactively) runs the usual ERT function that prompts for tests in the minibuffer.

How can I run one or more ERT tests programmatically? Is it even possible? I'm working on this kind of syntax right now, but it is not working. I think I need to extract the function body value of my-deftest, but I don't know how to do that yet.

(ert-run-test (make-ert-test :body my-deftest))

Update: I tried to put my whole test function inside the make-ert-test format shown in the example above, and was able to run the test. Not pretty code, because of the indenting. And the result was in square brackets, which I don't understand yet. But I can see the value: t in there that I would like to extract. So I'm getting closer.

(setq result
  (ert-run-test
   (make-ert-test :body
                  (lambda ()
                    ;; my ert test code here
                    (should (equal tcoll output))
                    ))))

[cl-struct-ert-test-passed "" (((should ...) :form (equal ... ...) :value t :explanation nil))]

After more experimentation to dig the answer out of the result returned (it was an array), here is the code that gets me to the t that I could see.

(setq data (aref result 2))
(((should (equal tcoll output)) :form (equal ...)) :value t :explanation nil))
(nth 4 (car data))
t

It works, but it seems like a brute force approach because I have to copy my ert-deftest code into a separate lambda function as shown above.

Stefan · Answer 1 · 2017-09-18T22:31:33.307

5

You can try ert-run-tests-batch, e.g.

(ert-run-tests-batch 'ert-test-record-backtrace)

but the results are not returned as a value, but rather via messages in *Messages* (or on stdout). Otherwise, you can use ert-run-tests which is a bit more complex to use because you need to provide a listener function:

(ert-run-tests 'ert-test-record-backtrace
               (lambda (event &rest args)
                 (message "Test event %S: %S" event args)))

edited Sep 18 '17 at 22:31

answered Jun 22 '16 at 16:10

Stefan

26,154
3
46
84

Thank you for your test event example. Wow, what a huge amount of output from the little listener that dumps the event and args (and my little deftest function was tiny!). – Kevin Jun 22 '16 at 16:38

score 4 · Accepted Answer · answered Jun 22 '16 at 16:18

4

Let's use the example from the ERT manual:

(ert-deftest ert-test-record-backtrace ()
  (let ((test (make-ert-test :body (lambda () (ert-pass "foo")))))
    (let ((result (ert-run-test test)))
      (should (ert-test-failed-p result)))))

The test should pass obviously. A list of symbols naming an ert-test object will be returned with

(apropos-internal "" #'ert-test-boundp) ;⇒ (ert-test-record-backtrace)

And finally show the number of available tests that passed as expected:

(ert--stats-passed-expected (ert-run-tests 't (lambda (&rest args)))) ;⇒ 1

The listener in ert-run-tests may be an arbitrary function called for side-effects.

answered Jun 22 '16 at 16:18

mutbuerger

3,434
14
22

1

Thank you for your help. With `apropos-internal` I can see my deftest names, but they are not test objects, since `ert-run-test` rejects them with `wrong type argument` errors. I will go an experiment with listeners now, since I've never used them either. – Kevin Jun 22 '16 at 16:29
1

`(ert-test-passed-p (ert-run-test (ert-get-test 'ert-test-record-backtrace)))` is what you're looking for. – mutbuerger Jun 22 '16 at 16:35
The `(ert--stats-passed-expected (ert-run-tests 'my-deftest ..listener))` syntax was closest to answering my question, I think. I can work with it. But I find it quite odd that in order to do something "so simple" as running a single test programmatically, I must call one of the "don't call functions with double hyphens in the name" functions. Thank you again. – Kevin Jun 22 '16 at 16:37
1

Sorry, our comments were out of sync. You are exactly right, `(ert-test-passed-p ...)` was exactly what I was looking for. Thank you again. I wonder why I couldn't find any of this in the manuals or online doc... Did I miss something obvious, or did you find out this info by working with the source code or something? – Kevin Jun 22 '16 at 17:54
You're welcome. IIRC the [ert-modeline](https://github.com/chrisbarrett/ert-modeline) package got me inspired to look into ERT. In addition, source code is always enlightening. – mutbuerger Jun 22 '16 at 18:03

run a single ERT test programmatically

2 Answers2