[racket-dev] [racket] tests/eli-tester feedback (Was: Racket unit testing)

20 minutes ago, Noel Welsh wrote:
> 
> On Tue, Feb 15, 2011 at 12:32 PM, Eli Barzilay <eli at barzilay.org> wrote:
> 
> > * It uses TR -- but it has to be very minimal.  (*Extremely* minimal,
> >  since it's a candidate for inclusion in the `racket' language, which
> >  is why I was thinking of not much more than agreed structs and
> >  parameters.)
> 
> Not a problem. TR just keeps the types straight.

OK, good idea then.


> > * I don't see why some results are structs and some are
> >   exceptions.
> 
> Test failures should include continuation marks etc. Exceptions feel
> like the right types for these.

Good point re the continuation marks, but^1 now that we went over the
name + source information + some parameter that composes a string that
indicates the test name, I think that a plain struct is more
appropriate.  This way the generic default listener will just take in
the test struct with that information, compose an error message using
the string that is built with that parameter, and throw an exception
with the whole thing.

But^2, there's also a place for a uniform exception to be thrown from
failed tests -- assuming that most testing facilities will want to
eventually throw an error (I think that test-engine is an exception
here), so such an exception should also be standardized in an
exception on top of that test failure struct.  (Probably by making it
contain it.)


> > * I don't see the point of a `pending' struct -- if it's supposed to
> >  represent a pending test, then it seems to me that the test
> >  itself should be modified when it has been executed.
> 
> I don't understand this. Some Rackunit users want to write tests
> that are have not yet been implemented, and want the UI to report
> how many yet-to-be implemented tests they have. I think you
> interpreted pending in the sense of concurrency. I mean it here in
> the sense of yet to be written. It is useful to annotate tests you
> know you have to write but haven't yet done so.

Indeed I misunderstood this, and it makes sense now.  It's a good
example for a difference between rackunit (write such pending test
cases in the code), vs my tester (defer to racket tools => write some
FIXME a comment).


> > * I'm not sure about things around "test-name" -- it should be free
> >  form information that the testing framework should render later on.
> >  Perhaps with a minimal spec of source and name, where a name can be
> >  #f.  (I'm thinking of these "anonymous" tests here.)  (This might be
> >  some unification of your name and info things, which should really
> >  always come together since you'd never modify one without the
> >  other.)
> 
> Agreed info and name should be unified. In the current design a name
> can always be null, indicating no name. It is a list to represent a
> hierarchy of names.

That makes sense.  But it also reminds me of another issue -- test
results should be put in a hierarchy somehow, right?  Is it better to
have some container struct or some sexpr of test results?  (Something
like (Rec Test-Results (U Test-Result (Listof Test-Results))).)


And finaly, there's the litmus test for existing code:

* Ryan: is something like this enough to implement the GUI layer?

* Kathy: can something like this be used to rebase test-engine on?

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!