[racket] tests/eli-tester feedback (Was: Racket unit testing)

Yesterday, Neil Van Dyke wrote:
> Eli Barzilay wrote at 02/13/2011 09:41 PM:
> > It currently shoots for (and will continue in the future) a very
> > low readability overhead -- that's the whole reason for the
> > infixish `=>' syntax. [...] To put this differently, I view tests
> > as an important thing that lives in the api neighborhood.  So
> > anything that requires looking at the documentation for casual
> > readers is as bad as writing the manual in hebrew and and handing
> > out dictionaries.
> 
> I'm not so sure about the requirement "readability by casual readers
> of the source without requiring looking at the documentation".

I'm tempted to ask what would you consider unclear about

  (test (+ 1 2) => 3)

but that's getting into subjectiveland.  In any case, I figured that
a much better solution to avoid some new `/=>' is to have instead a
new `true' so that (test E => true) works for any non-#f value.


> But if we do have that requirement, to me, it implies:
> 
> * Racket-idiomatic syntax (which usually means grouping parens, and
>   no infix keywords); and
> 
> * fairly self-explanatory human-readable identifiers (like using the
>   words like "equals", "eq", "result", "returns", "yields",
>   "expect", and "value", and not using more ambiguous or cryptic
>   symbols like "=>").

Obviously, there's additional directions that are important (IMO,
possibly less for others).  Two of these are:

1. Needs to be very convenient to write.  IOW, I want to just add some
   tests quickly, without typing a whole pile of boilerplate for
   defining a test suite, etc.

2. Errors should be helpful in finding out what went wrong.

Without these, you could reduce your tests to:

  (unless (and (equal? E1 V1) ...) (error "failure"))


> Going back to that root requirement... I think that unit testing
> should be so central to contemporary programming that we should just
> pick some syntax that makes sense for practical development both
> large and small, use it everywhere, and simply expect people who are
> looking at the source to know what Racket unit tests look like.  If
> we can do this canonical test syntax sensibly, and the syntax is set
> up so that we can plug in our own user interfaces for running the
> tests, I will convert all my existing and new open source code to
> use this canonical syntax.

My own conclusion here is that there are a lot of variables in a test
system -- even with something as minimal and small as my thing there
are already a number of non-trivial choices to make, and possibly some
alternatives that will be provided (like a `test*' thing that would do
the same was what its `test' is doing now).

So instead of trying to get a canonical syntax, it's better to get to
a canonical support that all test frameworks will build on.  For
example, they'd all run in a similar way, return some common structs
for failures etc.  This way a tool like the rackunit gui becomes
independent of what you choose to use.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!