[racket-dev] Test results are slightly incomprehensible...

The `with-limit' test is not completely deterministic. It's looking for
an out-of-memory exception, and you got an out-of-time exception,
instead. That sometimes happens if the machine is loaded or otherwise
distracted.

The `find-seconds' test may be wrong or it may be a bug. I see it for
64-bit builds, but I haven't had time to investigate, yet.

The number of tests run can vary for many reasons. Sometimes the
platform makes a difference, and probably 32-bit versus 64-bit makes a
difference. Maybe there are some random numbers that determine how many
times some tests are run. I don't think you should worry about it.

One of the reasons the tests don't print more nicely is that they're
really old. To a lesser extent, they're using minimal infrastructure to
exercise the low-level functionality on which better test tools are
implemented.

At Wed, 28 Sep 2011 21:02:14 +0100, Paulo J. Matos wrote:
> ...if you're not used to them, that is.
> 
> Hi,
> 
> I am running the racket tests to know if something went wrong with my
> fork of racket. The fork was done at tag 2ef9f5ae.
> 
> I run the tests with from the install directory with:
> ,----
> | racket$ bin/racket -r ./collects/tests/racket/quiet.rktl
> `----
> 
> Attached, you can find the output I obtain for both my repository and
> the main repository. The diff looks like:
> Section(basic)
> Section(unicode)
> Section(rx)
> Section(reading)
> Section(readtable)
> Section(printing)
> Section(macro)
> Section(syntax)
> Section(procs)
> Section(stx)
> Section(module)
> Section(numbers)
> Section(unsafe)
> Section(object)
> Section(struct)
> Section(unit)
> Section(unit/sig)
> Section(threads)
> Section(logger)
> Section(synchronization)
> Section(deep)
> Section(continuation-marks)
> Section(prompt)
> Section(wills)
> Section(namespaces)
> Section(modprot)
> Section(chaperones)
> Section(parameters)
> Section(port)
> Section(file)
> Section(udp)
> Section(file-after-udp)
> Section(path)
> Section(optimization)
> Section(names)
> Section(setup)
> Section(for)
> Section(list)
> Section(math)
> Section(vector)
> Section(function)
> Section(dict)
> Section(contract)
> Section(fixnum)
> Section(flonum)
> Section(mpair)
> Section(etc)
> Section(structlib)
> Section(async-channel)
> Section(restart)
> Section(mzlib-string)
> Section(path)
> Section(filelib)
> Section(portlib)
> Section(threadlib)
> Section(sets)
> Section(date)
> 
> Performed 35 expression tests (31 value expressions, 4 exn expressions)
> and 6 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> (Other messages report successful tests of error-handling behavior.)
> Section(compat)
> Section(command-line)
> Section(stream)
> Section(sequence)
> Section(generator)
> Section(pconvert)
> Section(pretty)
> Section(control)
> Section(serialization)
> Section(packages)
> Section(mzlib/contract)
> Section(sandbox)
> 
> Performed 281 expression tests (281 value expressions, 0 exn expressions)
> and 0 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(shared)
> Section(kw)
> Section(macrolib)
> Section(resource)
> Section(syntax/....)
> 
> Performed 152701 expression tests (139300 value expressions, 13401 exn 
> expressions)
> and 26923 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(moddep)
> Section(boundmap)
> Section(id-table)
> Section(cm)
> Section(module-reader)
> 
> Performed 153275 expression tests (139865 value expressions, 13410 exn 
> expressions)
> and 26941 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(version)
> Section(foreign)
> Section(uni-norm)
> 
> Performed 505944 expression tests (492534 value expressions, 13410 exn 
> expressions)
> and 26941 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> Section(basic)
> Section(unicode)
> Section(rx)
> Section(reading)
> Section(readtable)
> Section(printing)
> Section(macro)
> Section(syntax)
> Section(procs)
> Section(stx)
> Section(module)
> Section(numbers)
> Section(unsafe)
> Section(object)
> Section(struct)
> Section(unit)
> Section(unit/sig)
> Section(threads)
> Section(logger)
> Section(synchronization)
> Section(deep)
> Section(continuation-marks)
> Section(prompt)
> Section(wills)
> Section(namespaces)
> Section(modprot)
> Section(chaperones)
> Section(parameters)
> Section(port)
> Section(file)
> Section(udp)
> Section(file-after-udp)
> Section(path)
> Section(optimization)
> Section(names)
> Section(setup)
> Section(for)
> Section(list)
> Section(math)
> Section(vector)
> Section(function)
> Section(dict)
> Section(contract)
> Section(fixnum)
> Section(flonum)
> Section(mpair)
> Section(etc)
> Section(structlib)
> Section(async-channel)
> Section(restart)
> Section(mzlib-string)
> Section(path)
> Section(filelib)
> Section(portlib)
> Section(threadlib)
> Section(sets)
> Section(date)
> 
> Performed 35 expression tests (31 value expressions, 4 exn expressions)
> and 6 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> (Other messages report successful tests of error-handling behavior.)
> Section(compat)
> Section(command-line)
> Section(stream)
> Section(sequence)
> Section(generator)
> Section(pconvert)
> Section(pretty)
> Section(control)
> Section(serialization)
> Section(packages)
> Section(mzlib/contract)
> Section(sandbox)
> 
> Performed 281 expression tests (281 value expressions, 0 exn expressions)
> and 0 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(shared)
> Section(kw)
> Section(macrolib)
> Section(resource)
> Section(syntax/....)
> 
> Performed 152694 expression tests (139295 value expressions, 13399 exn 
> expressions)
> and 26919 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(moddep)
> Section(boundmap)
> Section(id-table)
> Section(cm)
> Section(module-reader)
> 
> Performed 153268 expression tests (139860 value expressions, 13408 exn 
> expressions)
> and 26937 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> (Other messages report successful tests of error-handling behavior.)
> Section(version)
> Section(foreign)
> Section(uni-norm)
> 
> Performed 505937 expression tests (492529 value expressions, 13408 exn 
> expressions)
> and 26937 exception field tests.
> 
> Errors were:
> (Section (got expected (call)))
> ((date) ((638931660) Error (find-seconds 0 1 2 1 4 1990)))
> ((sandbox) ((bad-exception-message: "with-limit: out of time") #t 
> (#<procedure:e-match?> "out of memor(?:y)" #<procedure:run> 
> #<procedure:...ket/sandbox.rktl:71:55>)))
> ((sandbox) (#f #t (#<procedure:run*> #<procedure:...ket/sandbox.rktl:67:55>)))
> ,----
> | --- /home/pmatos/repositories/racket/test_results.txt   2011-09-28 
> 20:44:41.000000000 +0100
> | +++ test_results.txt    2011-09-28 20:51:12.000000000 +0100
> | @@ -90,8 +90,8 @@
> |  Section(resource)
> |  Section(syntax/....)
> |  
> | -Performed 152701 expression tests (139300 value expressions, 13401 exn 
> expressions)
> | -and 26923 exception field tests.
> | +Performed 152694 expression tests (139295 value expressions, 13399 exn 
> expressions)
> | +and 26919 exception field tests.
> |  
> |  Errors were:
> |  (Section (got expected (call)))
> | @@ -105,8 +105,8 @@
> |  Section(cm)
> |  Section(module-reader)
> |  
> | -Performed 153275 expression tests (139865 value expressions, 13410 exn 
> expressions)
> | -and 26941 exception field tests.
> | +Performed 153268 expression tests (139860 value expressions, 13408 exn 
> expressions)
> | +and 26937 exception field tests.
> |  
> |  Errors were:
> |  (Section (got expected (call)))
> | @@ -118,8 +118,8 @@
> |  Section(foreign)
> |  Section(uni-norm)
> |  
> | -Performed 505944 expression tests (492534 value expressions, 13410 exn 
> expressions)
> | -and 26941 exception field tests.
> | +Performed 505937 expression tests (492529 value expressions, 13408 exn 
> expressions)
> | +and 26937 exception field tests.
> |  
> |  Errors were:
> |  (Section (got expected (call)))
> `----
> 
> Some of the interesting things are:
> - My fork seems to, after `Section(syntax/....)', perform less
> expression tests and less exception field tests. Why is this?
> - Same applies for all the other performed tests that were reported.
> 
> Now, what I don't understand is that I was expecting something like:
> ,----
> | 10 tests PASS, 5 tests FAIL, 2 tests CHECK, ...
> | <list of failed tests>
> `----
> 
> How do I go about checking what failed or not and where is the code that
> failed?
> 
> Cheers,
> 
> -- 
> PMatos
> _________________________________________________
>   For list-related administrative tasks:
>   http://lists.racket-lang.org/listinfo/dev