[plt-scheme] ANN: Beta SchemeUnit available

From: Noel Welsh (noelwelsh at yahoo.com)
Date: Fri Feb 17 15:38:37 EST 2006

Hi all,

Another beta of the next version of SchemeUnit is available


Install it thus:

  planet -f schemeunit.plt schematics 2 0

Feedback on the many changes is sought.  The documentation
is included below, so you don't need to install it to see
what is new.

Major changes:

 - make-test-case becomes test-case
 - make-teest-suite becomes test-suite
 - shortcuts for common cases (see documentation below)  
 - graphical UI is not (yet) available
 - before/after/around forms for test cases
 - test-suite takes before/after keyword arguments

Note that some changes that break the API from the previous


_SchemeUnit: Unit Testing in Scheme_

By  Noel Welsh (noelwelsh at yahoo dot com)
and Ryan Culpepper (ryan_sml at yahoo dot com)

This manual documents SchemeUnit version 3.0
Time-stamp: <06/02/17 15:41:04 noel>

Keywords: _schemeunit_, _test_, _testing_, _unit testing_,
          _unit test_


Unit testing is the testing in isolation of individual
elements of a program.  SchemeUnit is a framework for
defining, organizing, and executing unit tests written in
the PLT Scheme (http://www.plt-scheme.org/).  SchemeUnit
draws inspiration from two strands of work: existing
practice in interactive environments and the development of
unit testing frameworks following the growth of Extreme
Programming (http://www.extremeprogramming.org/).

In an interactive environment it is natural to write in a
``code a little, test a little'' cycle: evaluating
definitions and then immediately testing them in the
read-eval-print loop (REPL).  We take the simplicity and
immediacy of this cycle as our model.  By codifying these
practices we preserve the test cases beyond the running
of the interpreter allowing the tests to be run again when
code changes.

Unit testing is one of the core practices of the Extreme
Programming software development methodology.  Unit testing
is not new to Extreme Programming but Extreme Programming's
emphasis on unit testing has spurred the development of
software frameworks for unit tests.  SchemeUnit draws
inspiration from, and significantly extends the
expressiveness of, these frameworks.

Quick Start

Suppose you want to test the code contained in "file.scm".
Create a file called "file-test.scm".  This file will
contain the test cases.

At the top of file-test.scm import file.scm and the
SchemeUnit library:

  (require "file.scm"
  	   (planet "test.ss" ("schematics" "schemeunit.plt" 2)))

Now create a test suite to hold all the tests:

  (define file-tests
     "Tests for file.scm"

Now we define test cases within the test suite.  Let's
assume that file.scm implements versions of + and - called
my-+ and my--.  We are going to test my-+ and my-- for
integer arthimetic.

  (define file-tests
     "Tests for file.scm"
     (test-equal? "Simple addition" (my-+ 1 1) 2)
     (test-equal? "Simple subtraction" (my-- 1 1) 0))))

Above we implemented two simple test cases.  They have the

  (test-equal? name actual expected)

For example

  (test-equal? "Simple addition" (my-+ 1 1) 2)

creates a test case called "Simple addition" that tests

  (equal? (my-+ 1 1) 2)

is true.  There are test case constructors that test for
properties other than equality.  For example, we could

  (test-true "Simple addition" (= (my-+ 1 1) 2))

However this will give a less informative error message if
the test fails.

Now lets run our tests, using SchemeUnit's simple textual
user interface (there are fancier interfaces available but
this will do for our example).  In file-test.scm we import
the textual user interface, and then run our tests using

  (require (planet "text-ui.ss" ("schematics"
"schemeunit.plt" 2)))

  (test/text-ui file-tests)

Now we can execute this file in DrScheme and the textual
user interface will report the success or failure of tests.

Let's say you want to test that a number of properties
How do you do this in a single test case?  So far we've
seen test cases that hold a single expression.  It turns
that the test case constructors we've seen so far are
shortcuts for common patterns that involve a test case and
an assertion.  In general a test case may consist of any
number of expressions, so of which may be assertions.  For
example, here's a test case that checks a list has both
length 4 and all its members are even:

   "List has length 4 and all elements even"
   (let ((lst (list 2 4 6 8)))
     (assert = (length lst) 4)
       (lambda (elt)
         (assert-pred even? elt))

These are the basics of SchemeUnit.  Refer to the
documentation below for more advanced topics, such as
defining your own assertions.  Have fun!

SchemeUnit API

PLaneT is the main distribution mechanism for SchemeUnit.
To use SchemeUnit insert the require statement below,
the documentation for a specific section states otherwise.

  (require (planet "test.ss" ("schematics" "schemeunit.plt"

Core SchemeUnit Types

The _test_ is the core type in SchemeUnit.  A test is
a _test case_, which is a single action to test, or a _test
suite_, which is a collection of tests.  Test suites may
have optional setup and teardown actions.

> test       :=  test-case | test-suite
> test-case  := name action
> test-suite := name [before] [after] (list-of test)

An _assertion_ is a predicate that checks a condition.  The
assertion fails if it's condition is false.  Otherwise it
succeeds.  In SchemeUnit assertions fail by raising an
exception of type _exn:test:assertion_.  Otherwise
they return #t.

A test-case _action_ is a function of no arguments.  A
test-case succeeds if the action evaluates without raising
an exception; otherwise it fails.  Test-case failures are
divided into two cases: those thrown as the result of an
assertion failure, which we call _failures_ and those
due to other reasons, which we call _errors_.  We define a
type _test-result_ to encapsulate these concepts:

> test-result   := test-failure | test-error | test-success
> test-failure  := test-case failure-exn
> test-error    := test-case error-exn
> test-success  := test-case result

Constructing Test Cases and Test Suites

Test cases are constructed using the _test-case_ macro.

> (test-case name expr ...)

The name is a string that is reported as the name of the
test case.

The exprs are one or more expressions that are run as the
test case action.


   (assert = (+ 1 1) 2)
   (assert string=? (symbol->string 'foo) "foo"))

Often a test case will include actions that must be
before or after the tested expressions.  For example, a
might require a database connection is opened before it can
run, or require that a temporary file is deleted regardless
of test success or failure.  The _before_, _after_, and
_around_ macros can be used to specify these actions.

> (before (before-expr ...) expr ...)

Whenever control enters the scope execute the before-expr
before executing expr

> (after expr ... (after-expr ...))

Whenever control exits the scope execute the after-expr
after executing expr.  After-expr are executed even if expr
raise an exception.

> (around (before-expr ...) expr ... (after-expr ...))

Whenever control entres the scope execute the before-expr
before expr, and the after-expr after expr.  After-expr are
executed even if before-expr or expr raise an exception.


The test below checks that the file "test.dat" contains the
string "foo".  The before action writes to this file.  The
after action deletes it.

   "The name"
     ((with-output-to-file "test.dat"
        (lambda ()
          (write "foo"))))
     (with-input-from-file "test.dat"
       (lambda ()
         (assert-equal? "foo" (read))))
     ((delete-file "test.dat"))))

Test suites are constructed using the _test-suite_

> (test-suite name [#:before thunk] [#:after thunk] test
... )

The name is a string that is reported as the name of the
test suite.

The tests are one or more tests (test cases or test

The optional before thunk is an function of no arguments
that is executed before the tests are run.  The before
is signalled by placing the keyword #:before before the

The optional after thunk is a function of no arguments that
is executed after the tests are run.  The after thunk is
signalled by placing the keyword #:after before the thunk.

Before and after thunks, if specified, must be given before
any tests.


   "Suite name"
   #:before (lambda () (display "before"))
   #:after (lambda () (display "after"))
     (assert = 1 1)))

Predefined Assertions

SchemeUnit provides a rich library of predefined
Every assertion takes an optional _message_, which is a
string that is displayed if the assertion fails.  Each
assertion comes in two variants: a macro with the given
name, and a function with a * appended to the name.  The
macro version grabs source location information, which aids
debugging, and so should be used wherever possible.

Binary assertions expect to receive the actual computed
value first, and the expected value second (think
alphabetical order).

The predefined assertions are:

> (assert binary-predicate actual expected [message])
> (assert-equal? actual expected [message])
> (assert-eqv? actual expected [message])
> (assert-eq? actual expected [message])
> (assert-true actual [message])
> (assert-false actual [message])
> (assert-not-false actual [message])
> (assert-pred unary-predicate actual [message])
> (assert-exn exn-predicate thunk [message])
> (assert-not-exn thunk [message])
> (fail [message])


For the common case where a test case consists of a single
assertion there are shortcuts that create simultaneously a
test case and assertion.  The predefined shortcuts are:

> (test-assert name operator expr1 expr2)
> (test-pred name predicate expr)
> (test-equal? name actual expected)
> (test-eq? name actual expected)
> (test-eqv? name actual expected)
> (test-true name expr)
> (test-false name expr)
> (test-not-false name expr)
> (test-exn name pred thunk)
> (test-not-exn name thunk)


  (test-equal? "Example" 1 1)

is equivalent to

   (assert-equal? 1 1))

  (test-assert "Example 2" < 1 2)

is equivalent to

   "Example 2"
   (assert < 1 2))

Providing Additional Information With Assertions

When an assertion fails it stores information including the
name of the assertion, the location and message (if
available), the expression the assertion is called with,
the parameters to the assertion.  Additional information
be stored by using the _with-assertion-info*_ function, and
the _with-assertion-info_ macro.

> (make-assertion-info name value)

An assertion-info structure stores information associated
with the context of execution of an assertion.  Name is a
symbol.  Value is any value.

The are several predefined functions that create assertion
information structures with predefined names.  This avoids
misspelling errors:

> (make-assertion-name name)
> (make-assertion-params params)
> (make-assertion-location loc)
> (make-assertion-expression msg)
> (make-assertion-message msg)

> (with-assertion-info* info thunk)

The info is a list of assertion-info structures.

The thunk is a function of no arguments

The with-assertion-info* function stores the given info in
the assertion information stack for the duration of the
execution of thunk.

     (list (make-assertion-info 'time (current-seconds)))
     (lambda () (assert = 1 2))))

When this assertion fails the message

  time: <current-seconds-at-time-of-running-assertion>

will be printed along with the usual information on an
assertion failure.

> (with-assertion-info ((name val) ...) body ...)

The with-assertion-info macro stores the given information
in the assertion information stack for the duration of the
execution of the body expressions.  Name is a quoted symbol
and val is any value.


    (lambda (elt)
       (('current-element elt))
       (assert-pred odd? elt)))
    (list 1 3 5 7 8)))

When this test fails the message

  current-element: 8

will be displayed along with the usual information on an
assertion failure.

User Defined Assertions

SchemeUnit provides a way for user's to extend its builtin
collection of assertions using the
_define-simple-assertion_, _define-binary-assertion_, and
_define-assertion_ macros.  To understand these macros it
useful to understand a few details about an assertions
evaluation model.

Firstly, an assertion should be considered a function, even
though most uses are actually macros.  In particular,
assertions always evaluate their arguments exactly once
before executing any expressions in the body of the
assertions.  Hence if you wish to write assertions that
evalute user defined code that code must be wrapped in a
thunk (a function of no arguments) by the user.  The
predefined assert-exn is an example of this type of

It is also useful to understand how the assertion
information stack operates.  The stack is stored in a
parameter and the with-assertion-info forms evaluate to
calls to parameterize.  Hence assertion information has
lexical scope.  For this reason simple assertions (see
below) cannot usefully contain calls to with-assertion-info
to report additional information.  All assertions created
using define-simple-assertion or define-assertion grab some
information by default: the name of the assertions and the
values of the parameters.  Additionally the macro forms of
assertions grab location information and the expressions
passed as parameters.

> (define-simple-assertion (name param ...) expr ...)

The define-simple-assertion macro constructs an assertion
called name that takes the params as arguments and an
optional message and evaluates the exprs.  The assertion
fails if the result of the exprs is #f.  Otherwise the
assertion succeeds.  Note that simple assertions cannot
report extra information using with-assertion-info.

The define-simple-assertion macro actually constructs two
assertions: a macro with the given name that collects
location information, and a function with name name* that
does not collection source location information but can be
used in a higher-order fashion.  In my experience
higher-order assertions are useful for testing SchemeUnit
but not much else.


To define assertions assert-odd? and assert-odd?* we could
do this:

  (define-simple-assertion (assert-odd? number)
    (odd? number))

We could use these assertions in the usual way:

  (assert-odd? 3)  ;; Success
  (assert-odd? 2) ;; Failure

> (define-binary-assertion (name pred actual expected))
> (define-binary-assertion (name actual expected) expr ...)

The define-binary-assertion macro constructs a macro that
tests a binary predicate.  This allows actual and expected
values to be reported.  The first form of the macro accepts
a binary predicate and tests if the predicate holds for the
given values.  The second form tests if expr ... return


Here's the first form, where we use a predefined predicate
to construct a binary assertion:

  (define-binary-assertion (assert-char=? char=? actual

In use:

  (assert-char=? (read-char a-port) #\a)

If the expression is more complicated the second form
be used.  For example, below we define a binary assertion
that tests a number if within 0.01 of the expected value:

  (define-binary-assertion (assert-in-tolerance actual
    (< (abs (- actual expected)) 0.01))

> (define-assertion (name param ...) expr ...)

The define-assertion acts in exactly the same way as the
define-simple-assertion macro, except the assertion only
fails if the macro _fail-assertion_ is called in the body
the assertion.  This allows more flexible assertions, and
particular more flexible reporting options.

> (fail-assertion)

The fail-assertion macro raises an exn:test:assertion with
the contents of the assertion information stack.


Below is an assertion that tests if two lists are the same,
and if they aren't it reports the first elements that

  (define-assertion (assert-list= lst1 lst2)
     (lambda (elt1 elt2)
        (('element1 elt1)
         ('element2 elt2))
        (if (not (equal? elt1 elt2))

We could even get fancy, and allow the equality predicate
specified, and reuse assert-pred to test for equality:

  (define-assertion (assert-list= pred lst1 lst2)
     (lambda (elt1 elt2)
        (('element1 elt1)
         ('element2 elt2))
        (assert-pred pred elt1 elt2)))

You are encouraged to submit libraries of assertions to

User Defined Shortcuts

To define your own shortcuts use the _define-shortcut_

> (define-shortcut (name param ...) expr)

Name is the name of the shortcut.

Param ... are the parameters the shortcut takes, not
including the test case name.

Expr is a single expression that performs the test

The define-shortcut macro will create a macro of the form

  (name test-case-name param ...)


The predefined test-equal? shortcut is defined as follows:

  (define-shortcut (test-equal? expr1 expr2)
    (assert-equal? expr1 expr2))

User Interfaces

To use:

  (require (planet "text-ui.ss" ("schematics"
"schemeunit.plt" 2)))

SchemeUnit provides a text user interface.  Previous
versions also provided a graphical user interface.  This
will (eventually) be provided for the 3.x series, but is
currently available.

The text UI is run via the function

> (test/text-ui test [verbosity])
   test/text-ui : test (U 'quiet 'normal 'verbose) ->

The given test is run and the result of running it output
the current-output-port.  The output is compatable with the
(X)Emacs next-error command (as used, for example, by
(X)Emac's compile function)

The optional verbosity is one of 'quiet, 'normal, or
'verbose.  Quiet output displays only the number of
successes, failures, and errors.  Normal reporting
suppresses some extraneous assertion information (such as
the expression).  Verbose reports all information.

test/text-ui returns the number of unsuccessful tests.


To use the following definitions:

  (require (planet "util.ss" ("schematics" "schemeunit.plt"

Sometimes it is useful to test definitions that are not
exported by a module.  SchemeUnit supports this via the
_require/expose_ macro.

> (require/expose module (id ...))

The require/expose macro is like a normal require form,
except it can require definitions that are defined but not
provided (i.e. exported) by a module.  The ids are the
identifiers that are to be exposed in the current module or
namespace.  Module is a module specifier (as given to
require) where the ids are found.

The _test-suite*_ macro provided a shortcut for the common
case of a test suite that simply defines a number of test

> (test-suite* name (test-case-name test-case-expr ...)

Makes a test suite with the given name that contains test
cases with the given names and actions.

> (assert-regexp-match regex-or-string string-or-port

An assertion that tests if a regular expression (string or
regexp) matches a string or port.

Running Tests and Inspecting Results

SchemeUnit provides an API for running tests, from which
custom UIs can be created.


> (exn:test:assertion stack)
  struct exn:test:assertion : (list-of assertion-info)

An exn:test:assertion is raised when an assertion fails,
contains the contents of the assertion-info stack at the
time of failure.

> (test-result test-case-name)
  struct test-result : string

A test-result is the result of running the test case with
the given name.

> (test-failure result)
  struct (test-failure test-result) : exn:test:assertion

Subtype of test-result representing a test failure.

> (test-error result)
  struct (test-error test-result) : exn

Subtype of test-result representing a test error

> (test-success result)
  struct (test-success test-result) : any

Subtype of test-result representing a test success


> (run-test-case name action)
   run-test-case : string thunk -> test-result

> (run-test test)
   run-test : test -> (list-of test-result)

Runs the given test (test case or test suite) returning a
tree (list of lists) of results


  > (run-test
      (test-case "Dummy" (assert-equal? 1 2))))

> (fold-test-results suite-fn case-fn seed test)
   fold-test-results : (string 'a -> 'a) (test-results 'a
-> 'a) 'a test -> 'a

Fold the given functions over the results of running test.
A pre-order left-to-right depth-first fold.

Suite-fn is a function from test suite name (string) and
seed to a new seed

Case-fn is a function from test-result and the seed to a

Seed is any value

Test is a test-case or test-suite


The following code counts the number of successes

  (define (count-successes test)
     (lambda (suite seed) seed)
     (lambda (result seed)
       (if (test-success? result)
           (add1 seed)

When run over the SchemeUnit test suite:

  > (count-successes all-schemeunit-tests)

> (foldts fdown fup fhere seed test)
   foldts : (string thunk thunk 'a -> 'a) 
            (string thunk thunk 'a 'a -> 'a)
            (string thunk 'a -> 'a)
         -> 'a

Foldts is a nifty tree fold (created by Oleg Kiselyov) that
folds over a test in a useful way (fold-test-results isn't
that useful as you can't specify actions around test cases)

Fdown is a function of test suite name, before action,
action, and the seed.  It is run when a test suite is
encountered on the way down the tree (pre-order).

Fup is a function of test suite name, before action, after
action, the seed at the current level, and the seed
by the children.  It is run on the way up the tree

Fhere is a function of the test case name, the test case
action, and the seed. (Note that this might change in the
near future to just the test case.  This change would be to
allow fhere to discriminate subtypes of test-case, which in
turn would allow test cases that are, for example,


Here's the implementation of fold-test-results in terms of

  (define (fold-test-results suite-fn case-fn seed test)
     (lambda (name before after seed)
       (suite-fn name seed))
     (lambda (name before after seed kid-seed)
     (lambda (name action seed)
        (run-test-case name action)

If you've made it this far you truly are a master
hacker.  As a bonus prize we'll just mention that the code
in hash-monad.ss and monad.ss might be of interest for
constructing user interfaces.  The API is still in flux, so
isn't documented here.  However, do look at the
implementation of test/text-ui for examples of use.

Tips For Using SchemeUnit

I create one module of tests for each module of code.  If
the module is called "foo" the test module is called
"foo-test" and exports a single test suite, called
"foo-tests".  For each project (or collection) called, say,
"bar" I have a single module "all-bar-tests" that exports a
single test suite (called "all-bar-tests") which collects
together all the test suites for that project.  I often
create another file, called "run-tests.ss" which runs
"all-bar-tests' using the text user interface.  To run all
the project's tests I can then simply execute "run-test.ss"
in Dr/MzScheme or (X)Emacs.  

To run tests from (X)Emacs the following command is can be
given as the "Compile command" to compile:

   mzscheme -qe '(begin (load "run-test.ss") (exit))'

This command causes mzscheme to execute the code in
"run-test.ss" and then exit.  If some preventing loading of
"run-test.ss", for example a syntax error, mzscheme will
pause waiting for input and you will have to kill it next
time you run compile ((X)Emacs asks if you want to do

Defining your own assertions is one of the most powerful
features of SchemeUnit.  Whenever you find yourself writing
out the same series of assertions define your own assertion
and use that instead.  Your assertions will act just like
the pre-defined assertions; they take optional message
strings and will display locations and parameters.  They
really easy to create as well.  For instance, to define an
assertion called "assert-is-2" that asserts a value is
to 2, you'd just evaluate

  (define-simple-assertion (assert-is-2 x)
    (= x 2))

Assertions compose as well, so you could write:

  (define-assertion (assert-is-2 x)
    (assert = x 2))

If you find you're creating a library of assertions please
submit them to us so we can share them with all SchemeUnit

Extended Example

This example test suite is included in the SchemeUnit
distribution as "example.ss":

  ;; The tests below are intended as examples of how to use
  ;; the test package.  They test PLT Scheme's arithmetic
  ;; operations and some simple file reading

  (require (file "test.ss"))
  (require (file "text-ui.ss"))
    "Example tests"
     "Arithmetic tests"
     (test-assert "Multiply by zero" = (* 1234 0) 0)
     (test-assert "Add zero" = (+ 1234 0) 1234)
     (test-assert "Multiply by one" = (* 123.0 1) 123)
     (test-assert "Add one" = (+ 123.0 1) 124)
     (test-assert "Expt 0 0" = 1 (expt 0 0))
     (test-assert "Expt 0 1" = 0 (expt 0 1))
     (test-assert "Expt 0.0 0.0" = 1.0 (expt 0.0 0.0))
     (test-assert "Expt 0.0 1.0" = 0.0 (expt 0.0 1.0))
     "File tests"
     ;; An example with an after action
      "String port read"
      (let ((port (open-input-string "this is a test
         (assert-equal? "this is a test string" (read-line
         ((close-input-port port)))))
     ;; An example with an around action
      "File port read"
       ((with-output-to-file "test.dat"
          (lambda ()
            (write "foo"))))
       (with-input-from-file "test.dat"
         (lambda ()
           (assert-equal? "foo" (read))))
       ((delete-file "test.dat"))))

Further Reading

The paper "Two Little Languages: SchemeUnit and SchemeQL"
describes the design rationale and implementation of the
series of SchemeUnit.

There are comments in the source code, and tests of the
are good examples of usage.  The file "example.ss",
as part of SchemeUnit, is a good example of a reasonably
complex test suite.

Good references for the Extreme Programming methodology

  - http://c2.com/cgi-bin/wiki?ExtremeProgrammingRoadmap

  - http://www.extremeprogramming.org

  - http://www.xprogramming.com

  - http://www.junit.org; JUnit}, the Java unit test

Most of the code at Schematics
(http://schematics.sourceforge.net/) has extensive test
suites written using SchemeUnit.


The following people have contributed to SchemeUnit:

  - Eric Hanchow suggested test/text-ui return a useful

  - Richard Cobbe provided require/expose

  - John Clements suggested several new assertions

  - Jose A. Ortega Ruiz alerted me a problem in the
    packaging system and helped fix it.

  - Sebastian H. Seidel provided help packaging SchemeUnit
    into a .plt

  - Don Blaheta provided the method for grabbing line
    and file name in assertions

  - Patrick Logan ported example.ss to version 1.3

  - The PLT team made PLT Scheme

  - The Extreme Programming community started the whole
    testing framework thing

Email: noelwelsh <at> yahoo <dot> com   noel <at> untyped <dot> com
AIM: noelhwelsh
Blogs: http://monospaced.blogspot.com/  http://www.untyped.com/untyping/

Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 

Posted on the users mailing list.