[racket-dev] [racket] tests/eli-tester feedback (Was: Racket unit testing)

Ryan, thank you very much for refocusing the discussion
about testing on the essence and away from the superficial
syntactic issues. Syntax, as we all know, is just a macro
[which some people may get backwards and conclude that 
testing is just a macro. Nothing could be further from 
the truth, however]. 

Find someone to implement this idea. -- Matthias




On Feb 17, 2011, at 8:21 PM, Ryan Culpepper wrote:

> On 02/15/2011 07:28 AM, Eli Barzilay wrote:
>> And finaly, there's the litmus test for existing code:
>> 
>> * Ryan: is something like this enough to implement the GUI layer?
> 
> Not well, I think. The Test-Result type in Noel's racktest code is too
> simple and inflexible. It represents the minimal essence of testing,
> but it would be awkward to extend to richer testing sytems. Here's my
> counter-proposal for representing the results of tests:
> 
> 
> TestResult = (test-result header execution status)
>  where header is a TestHeader, execution is a TestExecution,
>  and status is a TestStatus
> 
> A TestResult combines a description of the test, a record of its
> behavior during execution, and its final status.
> 
> (This fixes the awkward distribution of information in racktest's
> Test-Result, with every status variant including a name field---except
> for exn:fail, which is anonymous or perhaps stashes it in the
> continuation marks.)
> 
> 
> TestHeader = (test-header name suite info)
>  where name is (U string #f), suite is (Listof string)
>  and info is a dictionary.
> 
> A TestHeader identifies the test that has been run via its name and
> its enclosing test suite. (Suites can be nested.) It also accommodates
> additional information that may be provided by some frameworks but not
> others. Examples include the location of the test expression and the
> mode of the test (expected to fail, test not yet implemented, etc).
> 
> 
> TestExecution = dictionary
> 
> TestExecution contains information about the execution of the test
> that is unrelated to its success or failure. For example, the rackunit
> gui reports the execution time and any custodian-managed resources that
> were left around when the test finished. If output is captured during
> execution, it would also belong here.
> 
> 
> TestStatus is one of
>  - (test-success)
>  - (test-failure info)
>      where info is a dictionary
>  - (test-error raised-value)
> 
> TestStatus represents the status of the test. For the error case, note
> that any value can be raised, not just exn structures.
> 
> Common info-dict keys include 'actual and 'expected, and perhaps
> 'comparison, but those keys don't apply to all failures. If the
> testing framework views failure as something that occurs during
> execution of code (as in rackunit, as opposed to check-expect), then
> the info-dict would also contain the continuation marks.
> 
> Perhaps there should be another variant for skipped/pending tests, or
> perhaps that should be represented by a successful TestStatus and an
> annotation on the TestHeader.
> 
> ----
> 
> And that's not quite the end of it. The rackunit gui creates an entry for a test case as soon as it starts running, so the user can see what test case is hanging and interrupt it if they choose. That requires additional communication between test execution and test display.
> 
> Ryan
> _________________________________________________
> For list-related administrative tasks:
> http://lists.racket-lang.org/listinfo/dev