[racket-dev] Contract Errors in the Test Suite

Hello developers,

I've been compiling Racket from git a lot lately (working on some iteration forms that now appear in the repository---thanks Sam!), and each time I do, I run the test suite.  For the past few weeks (month?) I've been getting a lot of errors related to the contract system.  I think these should be addressed, but I'm not sure if I'm the one to do it; this email is an attempt to figure out whether there is anything I can do.  A sample of the test output appears after this message.  

Apparently, the failures are related to the assignment of blame in define/contract contracts (i.e. contracts on an individual procedure that don't necessarily appear in a provide form).  A somewhat cryptic comment at the relevant section of the test suite states (contract-test.rktl):

  ;; For some of the following tests, it may not be clear
  ;; why the blame is what it is.  The contract(s) entered
  ;; into via with-contract are between the contracting
  ;; region and its context.  If the context allows the
  ;; value to flow into other regions without contracts
  ;; that protect it from misuses in those regions, it's
  ;; the context's fault.

This sounds to me like it's saying that passing an invalid value to a define/contract procedure should be blamed on its definition context not the source of the improper value; in other words, it's the fault of the definition context for letting this private value out into the big, bad world where improper values could be provided to it by entities that have no knowledge of its contract.  

When the tests error out by passing an invalid value to a define/contract procedure, the blame is currently being assigned to the source of the invalid value; based on the above, the tests want it to be assigned to the definition context.  The conflict is what is causing the test failure.  I can see three possibilities for the code change that resulted in these failures:

1) The guts of the contract library have changed, and it's now assigning blame incorrectly.
2) The contract library was changed deliberately to assign blame to the context that provides the invalid value, but the tests were not changed to keep up with this.
3) Something about the module system has changed, and it now considers the defining context to be "top level" where it used to be the name of the module.  The contract library is correctly assigning blame, but the *name* of the blame target has changed, and so the tests are failing to find what they are expecting in the blame message.

If the situation is 2 or 3, I would be happy to change the tests to expect the new assignment of blame so that things no longer fail all the time.  If the situation is 1, I doubt I can be of much help, but would be happy to be in touch with whomever is responsible for that code as a tester of potential fixes/wall to bounce ideas off of.  Either way, I think somebody should know about this.

Thanks,
Will

Test output follows:
---------------------------------

eAmu:racket farr$ ./test.sh 

>>> racket/quiet.rktl: running...
Section(basic)
Section(unicode)
Section(rx)
Section(reading)
Section(readtable)
Section(printing)
Section(macro)
Section(syntax)
Section(procs)
Section(stx)
Section(module)
Section(numbers)
Section(unsafe)
Section(object)
Section(struct)
Section(unit)
Section(unit/sig)
Section(threads)
Section(logger)
Section(synchronization)
Section(deep)
Section(continuation-marks)
Section(prompt)
Section(wills)
Section(namespaces)
Section(modprot)
Section(chaperones)
Section(parameters)
Section(port)
Section(file)
Section(udp)
Section(file-after-udp)
Section(path)
Section(optimization)
Section(names)
Section(for)
Section(list)
Section(math)
Section(vector)
Section(function)
Section(dict)
Section(contract)

Performed 3072 expression tests (1881 value expressions, 1191 exn expressions)
and 2382 exception field tests.

Errors were:
(Section (got expected (call)))
((contract) (#(struct:exn:fail:contract:blame "foo-dc13: top-level broke the contract (-> number? number?) on foo-dc13; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc13 scheme/base (require scheme/contract) (define/contract (foo-dc13 n) (-> number? number?) (+ n 1)) (foo-dc13 #t)))) (eval (quote (require (quote foo-dc13)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc14: top-level broke the contract (-> number? number?) on foo-dc14; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc14 scheme/base (require scheme/contract) (provide foo-dc14) (define/contract (foo-dc14 n) (-> number? number?) (+ n 1))))) (eval (quote (module bar-dc14 scheme/base (require (quote foo-dc14)) (foo-dc14 #t)))) (eval (quote (require (quote bar-dc14)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc15: top-level broke the contract (-> number? number?) on foo-dc15; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc15 scheme/base (require scheme/contract) (provide foo-dc15) (define/contract (foo-dc15 n) (-> number? number?) (+ n 1))))) (eval (quote (require (quote foo-dc15)))) (eval (quote (foo-dc15 #t))))))
((contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 scheme/base (require scheme/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
(Other messages report successful tests of error-handling behavior.)
Section(fixnum)
Section(flonum)
Section(mpair)
Section(etc)
Section(structlib)
Section(async-channel)
Section(restart)
Section(mzlib-string)
Section(path)
Section(filelib)
Section(port)
Section(threadlib)
Section(sets)
Section(date)
Section(compat)
Section(command-line)
Section(pconvert)
Section(pretty)
Section(control)
Section(serialization)
Section(packages)
Section(mzlib/contract)

Performed 1790 expression tests (1132 value expressions, 658 exn expressions)
and 1316 exception field tests.

Errors were:
(Section (got expected (call)))
((mzlib/contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 mzscheme (require mzlib/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
(Other messages report successful tests of error-handling behavior.)
Section(sandbox)
Section(shared)
Section(kw)
Section(macrolib)

Performed 141172 expression tests (127981 value expressions, 13191 exn expressions)
and 27912 exception field tests.

Errors were:
(Section (got expected (call)))
((contract) (#(struct:exn:fail:contract:blame "foo-dc13: top-level broke the contract (-> number? number?) on foo-dc13; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc13 scheme/base (require scheme/contract) (define/contract (foo-dc13 n) (-> number? number?) (+ n 1)) (foo-dc13 #t)))) (eval (quote (require (quote foo-dc13)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc14: top-level broke the contract (-> number? number?) on foo-dc14; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc14 scheme/base (require scheme/contract) (provide foo-dc14) (define/contract (foo-dc14 n) (-> number? number?) (+ n 1))))) (eval (quote (module bar-dc14 scheme/base (require (quote foo-dc14)) (foo-dc14 #t)))) (eval (quote (require (quote bar-dc14)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc15: top-level broke the contract (-> number? number?) on foo-dc15; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc15 scheme/base (require scheme/contract) (provide foo-dc15) (define/contract (foo-dc15 n) (-> number? number?) (+ n 1))))) (eval (quote (require (quote foo-dc15)))) (eval (quote (foo-dc15 #t))))))
((contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 scheme/base (require scheme/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 mzscheme (require mzlib/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
(Other messages report successful tests of error-handling behavior.)
Section(moddep)
Section(boundmap)
Section(id-table)
Section(cm)
Section(module-reader)

Performed 141744 expression tests (128544 value expressions, 13200 exn expressions)
and 27930 exception field tests.

Errors were:
(Section (got expected (call)))
((contract) (#(struct:exn:fail:contract:blame "foo-dc13: top-level broke the contract (-> number? number?) on foo-dc13; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc13 scheme/base (require scheme/contract) (define/contract (foo-dc13 n) (-> number? number?) (+ n 1)) (foo-dc13 #t)))) (eval (quote (require (quote foo-dc13)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc14: top-level broke the contract (-> number? number?) on foo-dc14; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc14 scheme/base (require scheme/contract) (provide foo-dc14) (define/contract (foo-dc14 n) (-> number? number?) (+ n 1))))) (eval (quote (module bar-dc14 scheme/base (require (quote foo-dc14)) (foo-dc14 #t)))) (eval (quote (require (quote bar-dc14)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc15: top-level broke the contract (-> number? number?) on foo-dc15; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc15 scheme/base (require scheme/contract) (provide foo-dc15) (define/contract (foo-dc15 n) (-> number? number?) (+ n 1))))) (eval (quote (require (quote foo-dc15)))) (eval (quote (foo-dc15 #t))))))
((contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 scheme/base (require scheme/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 mzscheme (require mzlib/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
(Other messages report successful tests of error-handling behavior.)
Section(version)
Section(foreign)
Section(uni-norm)

Performed 494106 expression tests (480906 value expressions, 13200 exn expressions)
and 27930 exception field tests.

Errors were:
(Section (got expected (call)))
((contract) (#(struct:exn:fail:contract:blame "foo-dc13: top-level broke the contract (-> number? number?) on foo-dc13; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc13 scheme/base (require scheme/contract) (define/contract (foo-dc13 n) (-> number? number?) (+ n 1)) (foo-dc13 #t)))) (eval (quote (require (quote foo-dc13)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc14: top-level broke the contract (-> number? number?) on foo-dc14; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc14 scheme/base (require scheme/contract) (provide foo-dc14) (define/contract (foo-dc14 n) (-> number? number?) (+ n 1))))) (eval (quote (module bar-dc14 scheme/base (require (quote foo-dc14)) (foo-dc14 #t)))) (eval (quote (require (quote bar-dc14)))))))
((contract) (#(struct:exn:fail:contract:blame "foo-dc15: top-level broke the contract (-> number? number?) on foo-dc15; expected <number?>, given: #t" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module foo-dc15 scheme/base (require scheme/contract) (provide foo-dc15) (define/contract (foo-dc15 n) (-> number? number?) (+ n 1))))) (eval (quote (require (quote foo-dc15)))) (eval (quote (foo-dc15 #t))))))
((contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 scheme/base (require scheme/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg scheme/base (require scheme/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "contract-test-suite3: top-level broke the contract integer? on x; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (let () (eval (quote (module contract-test-suite3 mzscheme (require mzlib/contract) (define x #f) (provide/contract (x integer?))))) (eval (quote (require (quote contract-test-suite3)))) (eval (quote x)))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "pos: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module pos mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote pos)))))))
((mzlib/contract) (#(struct:exn:fail:contract:blame "neg: top-level broke the contract integer? on i; expected <integer?>, given: #f" #<continuation-mark-set> #<blame>) exn-type (begin (eval (quote (module neg mzscheme (require mzlib/contract) (define i #f) (provide/contract (i integer?))))) (eval (quote (require (quote neg)))))))
>>> racket/quiet.rktl: exit with error code 1
>>> racket/quiet.rktl: BOOM!

>>> typed-scheme/nightly-run.rkt: running...
>>> typed-scheme/nightly-run.rkt: no failures.

>>> match/plt-match-tests.rkt: running...
342 success(es) 0 failure(s) 0 error(s) 342 test(s) run
>>> match/plt-match-tests.rkt: no failures.

>>> lazy/main.rkt: running...
282 tests passed
>>> lazy/main.rkt: no failures.

>>> scribble/main.rkt: running...
324 tests passed
>>> scribble/main.rkt: no failures.

>>> net/main.rkt: running...
109 tests passed
39385 tests passed
>>> net/main.rkt: no failures.

>>> file/main.rkt: running...
112 tests passed
>>> file/main.rkt: no failures.

>>> profile/main.rkt: running...
21 tests passed
>>> profile/main.rkt: no failures.