[racket-dev] [racket] tests/eli-tester feedback (Was: Racket unit testing)

50 minutes ago, Ryan Culpepper wrote:
> On 02/15/2011 07:28 AM, Eli Barzilay wrote:
> > And finaly, there's the litmus test for existing code:
> >
> > * Ryan: is something like this enough to implement the GUI layer?
> 
> Not well, I think. The Test-Result type in Noel's racktest code is
> too simple and inflexible. It represents the minimal essence of
> testing, but it would be awkward to extend to richer testing
> sytems. Here's my counter-proposal for representing the results of
> tests:
> [...]

I can't make sense of it, besides a vague "waaaay to heavy" feeling
for something that should be core-ishly minimalistic.

In an attempt to follow it, I did this:

  TestResult = header
               execution
               status

but your TestHeader is used only there, so it could be folded in:

  TestResult = name      (U String #f)
               suite     (Listof String)
               info      Dictionary
               execution
               status

TestExecution is also used only once so it can also be folded in --
but since it's just a generic dictionary, it can be dropped.

  TestResult = name      (U String #f)
               suite     (Listof String)
               info      Dictionary
               status

Now, status is one of three options, the failure one has a dictionary
so it can be removed (folded into the above).

So overall, it looks like a simple struct with a name, a "suite" (kind
of, defined indirectly by a hierarchy of string lists), a generic
dictionary for "stuff", and a status.  This is all modulo some
questions/issues that are unclear to me:

* It looks like it tries to break away too many pieces into a formal
  description.  For example, it looks like an overkill to have fields
  with the actual value and expected value and worse -- the
  comparison.  For example, What happens when the comparison is
  parameterized (eg, "close within dx to some number")?

* What happens when there's no specific expected value to compare?
  For example, run some two pieces of code 10 times each and check
  that the average runtime of the first is below the runtime of the
  second.  This could be phrased in terms of an expected value, but in
  a superficial way, and will prevent useful information from being
  expressed (since the information would have to be reduced to two
  numbers).

* What if you don't want to hold on to the value?  (For example, free
  some related resource.)

* This solidifies the list-of-strings as a representation of the test
  hierarchy.  But perhaps there is no way to avoid this -- if it's
  made into a proper hierarchy of objects it will probably complicate
  things in a way that requires the listener to get "update" events
  that tells it how the structure changed.

* I'm not sure about the error result.  It seems to me that this is a
  meta issue that you're dealing with when you develop the test suite,
  and as such it should be something that you'd deal with in the usual
  ways => throw an exception.  It's the tools that should be in charge
  of catching such an exception and deal with it -- which means that
  - in my tester's case, it'll defer to racket as usual, meaning that
    you'd just get an error.
  - in rackunit's case you'd probably get some report listing the
    erroneous tests, instead of propagating the error.
  - and in your gui case you'd catch exceptions and show them as error
    results.

Also:

> And that's not quite the end of it. The rackunit gui creates an
> entry for a test case as soon as it starts running, so the user can
> see what test case is hanging and interrupt it if they choose. That
> requires additional communication between test execution and test
> display.

Yes, that would e part of the protocol for the listener -- and it
makes sense to allow tests to invoke it to let it know that a test has
started.


I'll try to synthesize all of the above, with Noel's suggestion, into
something more concrete later today.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!