[racket-dev] [racket] tests/eli-tester feedback (Was: Racket unit testing)

On 02/15/2011 07:28 AM, Eli Barzilay wrote:
> And finaly, there's the litmus test for existing code:
>
> * Ryan: is something like this enough to implement the GUI layer?

Not well, I think. The Test-Result type in Noel's racktest code is too
simple and inflexible. It represents the minimal essence of testing,
but it would be awkward to extend to richer testing sytems. Here's my
counter-proposal for representing the results of tests:


TestResult = (test-result header execution status)
   where header is a TestHeader, execution is a TestExecution,
   and status is a TestStatus

A TestResult combines a description of the test, a record of its
behavior during execution, and its final status.

(This fixes the awkward distribution of information in racktest's
Test-Result, with every status variant including a name field---except
for exn:fail, which is anonymous or perhaps stashes it in the
continuation marks.)


TestHeader = (test-header name suite info)
   where name is (U string #f), suite is (Listof string)
   and info is a dictionary.

A TestHeader identifies the test that has been run via its name and
its enclosing test suite. (Suites can be nested.) It also accommodates
additional information that may be provided by some frameworks but not
others. Examples include the location of the test expression and the
mode of the test (expected to fail, test not yet implemented, etc).


TestExecution = dictionary

TestExecution contains information about the execution of the test
that is unrelated to its success or failure. For example, the rackunit
gui reports the execution time and any custodian-managed resources that
were left around when the test finished. If output is captured during
execution, it would also belong here.


TestStatus is one of
   - (test-success)
   - (test-failure info)
       where info is a dictionary
   - (test-error raised-value)

TestStatus represents the status of the test. For the error case, note
that any value can be raised, not just exn structures.

Common info-dict keys include 'actual and 'expected, and perhaps
'comparison, but those keys don't apply to all failures. If the
testing framework views failure as something that occurs during
execution of code (as in rackunit, as opposed to check-expect), then
the info-dict would also contain the continuation marks.

Perhaps there should be another variant for skipped/pending tests, or
perhaps that should be represented by a successful TestStatus and an
annotation on the TestHeader.

----

And that's not quite the end of it. The rackunit gui creates an entry 
for a test case as soon as it starts running, so the user can see what 
test case is hanging and interrupt it if they choose. That requires 
additional communication between test execution and test display.

Ryan