[racket-dev] behavior of check-expect on errors during evaluation of expected values

I think the current behavior of check-expect is unfortunate w.r.t. errors that occur during the evaluation of the "expected" values.

Consider this program:

(check-expect (+ 3 4) 7)
(check-expect (* 4 5) 20)
(check-expect (+ 4 9) (/ 1 0))
(check-expect (+ 2 3) 5)

It produces this output in intermediate:

/: division by zero
Both tests passed!

In the example given, it's pretty clear that something major went wrong.  If you have thirty or forty test cases, though, this can look a lot like success.

I propose one of the following two solutions; I think the first is less work than the second.

1) keep a flag around during evaluation that records whether an exception has occurred during the evaluation of an 'expected' value, and update the report accordingly, e.g.:

2 tests passed. Some tests failed to run!

(could be better, I know.)

2) Record this as a test failure, and continue running the tests. This would mean adding a new kind of test failure message. Actually, this might not be too much work, either.

Thoughts?

John



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4370 bytes
Desc: not available
URL: <http://lists.racket-lang.org/dev/archive/attachments/20121024/ba9bd7dd/attachment.p7s>