How can I skip a test if another test failed?

Question

Say I have two tests:

(defun func-one () nil)
(defun func-two () (and (func-one) t))

(ert-deftest test-one ()
  "Test basic functionality"
  (should (func-one)))

(ert-deftest test-two ()
  "Test advanced functionality"
  (should (func-two)))

How can I run test-two only if test-one succeeds?

Or, perhaps,

(defun func-one () nil)
(defun func-two () (and (func-one) t))

(ert-deftest test-all ()
  "Test basic functionality"
  (should (func-one))
  (should (func-two)))

Testing func-two only if func-one passes, for example.

Or even

(defun func-one () nil)
(defun func-two () (and (func-one) t))
(defun func-thr () (or  (func-one) t))

(ert-deftest test-one ()
  "Test basic functionality"
  (should (func-one)))

(ert-deftest test-two ()
  "Test advanced functionality"
  (should (func-two)))

(ert-deftest test-thr ()
  "Test advanced functionality"
  (should (func-thr)))

Testing

func-two only if func-one passes
func-thr only if func-one passes

but func-two failing should not impede func-thr from running.

@abo-abo This will work for this simple case, but more complete testing suites will likely have a full-fledged DAG of tests. — Sean Allred, Jan 01 '15 at 19:34
@SeanAllred I'd argue that such a test suite is flawed. Test cases should be independent. What if the test you depend on is never run? If it's for resource management, create macros that handle this for you. — , Jan 02 '15 at 12:32
@lunaryorn I agree that test cases should be independent, but in reality they aren't. I don't like having a 'failed' test when this failure is really the result of another function behaving improperly. The real use case is a function (A) that parses a string and a function (B) that uses the parsed information. Should (B) be marked 'failed' because (A) has failed? ('Yes' could very well be the right answer here -- I've never worked with CI in a professional setting.) — Sean Allred, Jan 02 '15 at 16:49
@SeanAllred In (B), don't call the function from (A) again. Instead create a literal value of whatever result (A) returns and use that in the test case for (B). Or keep things they are: It's perfectly ok if a single issue causes multiple tests to fail. — , Jan 03 '15 at 11:26
@lunaryorn I'm just going to keep things as they are – we have a lot of interrelated functions and nullifying them with `cl-letf` or similar would be far more trouble than it's worth :) — Sean Allred, Jan 03 '15 at 15:30

score 2 · Answer 1 · answered Jan 01 '15 at 22:53

2

I have used something like this in vc-tests.el:

(ert-deftest vc-test-git00-create-repo ()
  "Check `vc-create-repo' for the git backend."
  ...)

(ert-deftest vc-test-git01-register ()
  "Check `vc-register' and `vc-registered' for the git backend."
  (skip-unless
   (ert-test-passed-p
    (ert-test-most-recent-result
     (ert-get-test 'vc-test-git00-create-repo))))

  ...)

answered Jan 01 '15 at 22:53

Michael Albinus

6,647
14
20

If I understand this correctly, this is almost *exactly* what I'm looking for! I'll try this out and get back to you `:)` – Sean Allred Jan 02 '15 at 00:20
ERT doesn't guarantee any order of test execution, and the alphabetical ordering in current ERT solely depends on `apropros-internal` returning matching symbols in alphabetical order. Neither does it guarantee that all tests are run. This is pretty flaky, imho. – Jan 02 '15 at 12:30
@lunaryorn That's strange, then -- the form itself would suggest a dependency (`ert-test-most-recent-result`), but I suppose that's a farce :) ERT could certainly be patched to recognize this, though. It would actually be pretty easy if I'm not over-simplifying things in my head. – Sean Allred Jan 02 '15 at 16:46
`skip-unless` is robust against errors. If the first test didn't run, the second one simply skips. – Michael Albinus Jan 02 '15 at 17:32
@lunaryorn maybe you file a "wishlist" bug towards Emacs asking for order of tests and alike? If time permits, I could check what to do. – Michael Albinus Jan 02 '15 at 17:34

score 1 · Answer 2 · answered Jan 01 '15 at 20:17

Maybe something like this:

(defun func-one ()
  (message "running: func-one")
  nil)
(defun func-two ()
  (message "running: func-two")
  (and (func-one) t))
(defun func-thr ()
  (message "running: func three")
  (or  (func-one) t))

(require 'ert)
(ert-delete-all-tests)
(setq func-tests nil)
(ert-deftest test-one ()
  "Test basic functionality"
  (when (should (func-one))
    (push 'test-one func-tests)))

(ert-deftest test-two ()
  "Test advanced functionality"
  (should
   (and (memq 'test-one func-tests)
        (func-two))))

(ert-deftest test-thr ()
  "Test advanced functionality"
  (should
   (func-thr)))

(ert t)

I just put everything into one buffer and call eval-buffer to test this.

How can I skip a test if another test failed?

2 Answers2