[racket-dev] how to test rackunit

In general, you should look at what DrDr does to decide how to run
some tests. For instance, with Rackunit, you can look here:

http://drdr.racket-lang.org/27688/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/

and here in particular:

http://drdr.racket-lang.org/27688/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt

One thing that you will see is that as part of testing there are some
failing tests to see what happens, but the tests deliberately switch
current-error-port with current-input-port so that it does not result
in STDERR output that DrDr would count as a failure.

Jay


On Sat, Nov 2, 2013 at 10:21 PM, David T. Pierson <dtp at mindstory.com> wrote:
> What is the correct way to run the tests which test rackunit?
>
> I am running the following after building with a clean tree:
>
>  $ racket/bin/raco test pkgs/rackunit-pkgs/rackunit-test/tests/rackunit
>
> This seems to show several "failures", though since it is testing rackunit
> I'm pretty sure most if not all are intentional (to verify that things
> which should fail do fail.)  The full output is below [*].
>
> What is not clear is how do people run this and determine whether they
> have any unintentional failures?  Is there a separate command I should
> be using?  Perhaps people just diff the output from a clean tree vs
> the output with their changes?
>
> BTW, I have read:
>
> http://blog.racket-lang.org/2012/11/tutorial-contributing-to-racket.html
>
> which has been very helpful since Joe's example was a change to
> rackunit, but I didn't see anything addressing this problem.
>
> For additional context: my motivation for running these tests is to improve
> check-exn so it no longer passes when given a non-thunk.  I think it
> should raise an error instead.  I already have a patch for this but
> running the tests has me thinking I'm not using the correct command to
> test rackunit.
>
> David
>
> [*] Here is the output from the above 'raco test ...' command, with stdout
> and stderr redirected to a file.  Note the redirection changes the
> ordering of some lines compared to the terminal output.
>
>
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/base-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/check-info-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/check-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/counter-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/format-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/hash-monad-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/location-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/log.rkt"
> --------------------
> FAILURE
> name:       check-true
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/log.rkt> 38 2 1185 15)
> expression: (check-true #f)
> params:     (#f)
>
> Check failure
> --------------------
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/monad-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/pr10950.rkt"
> 1 test--------------------
> ERROR
> Outta here!
>   context...:
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:133:29
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:58:0: top-level-check-around
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:119:21: check-pred492750
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt: [running body]
>    f88
>    loop
>    loop
>    loop
>    f88
>    /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body]
>
> --------------------
> --------------------
> FAILURE
> name:       check
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt> 44 0 1344 17)
> expression: (check = 1 2)
> params:     (#<procedure:=> 1 2)
> message:    0.0
>
> Check failure
> --------------------
> --------------------
> ERROR
> First Outta here!
>   context...:
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt: [running body]
>    f88
>    loop
>    loop
>    loop
>    f88
>    /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body]
>
> --------------------
> --------------------
> error
> ERROR
> Second Outta here!
>   context...:
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt: [running body]
>    f88
>    loop
>    loop
>    loop
>    f88
>    /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body]
>
> --------------------
> --------------------
> FAILURE
> actual:     1
> expected:   2
> name:       check-eq?
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt> 19 12 520 15)
> expression: (check-eq? 1 2)
>
> Check failure
> --------------------
> --------------------
> failure
> FAILURE
> actual:     1
> expected:   2
> name:       check-eq?
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt> 20 21 558 15)
> expression: (check-eq? 1 2)
>
> Check failure
> --------------------
> --------------------
> FAILURE
> actual:     #""
> expected:   #"Oh HAI!\nI didn't run\n"
> name:       check-equal?
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> 29 4 956 23)
> expression: (check-equal? cout #"Oh HAI!\nI didn't run\n")
>
> Check failure
> --------------------
> --------------------
> FAILURE
> actual:     #""
> expected:   #"--------------------\nERROR\nOutta here!\n\n--------------------\n--------------------\nFAILURE\nname:       check\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-check-test.rkt> 44 0 1344 17)\nexpression: (check = 1 2)\nparams:     (#<procedure:=> 1 2)\nmessage:    0.0\n\nCheck failure\n--------------------\n"
> name:       check-equal?
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> 30 4 984 23)
> expression: (check-equal? cerr #"--------------------\nERROR\nOutta here!\n\n--------------------\n--------------------\nFAILURE\nname:       check\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-check-test.rkt> 44 0 1344 17)\nexpression: (check = 1 2)\nparams:     (#<procedure:=> 1 2)\nmessage:    0.0\n\nCheck failure\n--------------------\n")
>
> Check failure
> --------------------
> --------------------
> FAILURE
> actual:     #""
> expected:   #"--------------------\nERROR\nFirst Outta here!\n\n--------------------\n--------------------\nerror\nERROR\nSecond Outta here!\n\n--------------------\n--------------------\nFAILURE\nactual:     1\nexpected:   2\nname:       check-eq?\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 19 12 520 15)\nexpression: (check-eq? 1 2)\n\nCheck failure\n--------------------\n--------------------\nfailure\nFAILURE\nactual:     1\nexpected:   2\nname:       check-eq?\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 20 21 558 15)\nexpression: (check-eq? 1 2)\n\nCheck failure\n--------------------\n"
> name:       check-equal?
> location:   (#<path:/home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt> 30 4 984 23)
> expression: (check-equal? cerr #"--------------------\nERROR\nFirst Outta here!\n\n--------------------\n--------------------\nerror\nERROR\nSecond Outta here!\n\n--------------------\n--------------------\nFAILURE\nactual:     1\nexpected:   2\nname:       check-eq?\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 19 12 520 15)\nexpression: (check-eq? 1 2)\n\nCheck failure\n--------------------\n--------------------\nfailure\nFAILURE\nactual:     1\nexpected:   2\nname:       check-eq?\nlocation:   (#<path:PLTHOME/collects/tests/rackunit/standalone-test-case-test.rkt> 20 21 558 15)\nexpression: (check-eq? 1 2)\n\nCheck failure\n--------------------\n")
>
> Check failure
> --------------------
>  passed
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/result-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt"
> 127 success(es) 0 failure(s) 0 error(s) 127 test(s) run
> 0
> --------------------
> Failures > Intended to fail
> Intended to fail
> FAILURE
> name:       fail
> location:   pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:47:33
> params:     Check failure
> --------------------
> --------------------
> Failures > Also intended to fail
> Also intended to fail
> FAILURE
> name:       check-eq?
> location:   pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:48:38
> actual:     'apples
> expected:   'orange
> Check failure
> --------------------
> --------------------
> Failures > Yet again intended to fail
> Yet again intended to fail
> FAILURE
> name:       check-equal?
> location:   pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:49:3
> actual:     "apples"
> expected:   "oranges"
> Check failure
> --------------------
> --------------------
> Failures > Intended to throw error
> Intended to throw error
> ERROR
> testing: <<This is an error message>>
>   context...:
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/result.rkt:99:3
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:28:2
>    the-tests
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:60:0: apply-test-suite
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/text-ui.rkt:238:0: run-tests14
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt: [running body]
>    f88
>    loop
>    loop
>    loop
>    f88
>    /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body]
>
> --------------------
> --------------------
> Failures > Error within a check
> Error within a check
> ERROR
> name:       check
> location:   pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/all-rackunit-tests.rkt:51:37
> params:     #<procedure:error>
> 'foo
> 'bar
> error: contract violation
>   expected: string?
>   given: 'bar
>   argument position: 2nd
>   other arguments...:
>    'foo
>   context...:
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:133:29
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/check.rkt:119:21: check491437
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/result.rkt:99:3
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:28:2
>    the-tests
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/private/test-suite.rkt:60:0: apply-test-suite
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-lib/rackunit/text-ui.rkt:238:0: run-tests14
>    /home/dtp/src/dtp-racket/pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/run-tests.rkt: [running body]
>    f88
>    loop
>    loop
>    loop
>    f88
>    /home/dtp/src/dtp-racket/pkgs/compiler-pkgs/compiler-lib/compiler/commands/test.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/raco.rkt: [running body]
>    /home/dtp/src/dtp-racket/racket/collects/raco/main.rkt: [running body]
>
> --------------------
> 0 success(es) 3 failure(s) 2 error(s) 5 test(s) run
> 5
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-check-test.rkt"
> Oh HAI!
> I didn't run
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone-test-case-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/standalone.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-case-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-suite-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/test-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/text-ui-test.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/text-ui-util-test.rkt"
> 55/340 test failures
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/tl.rkt"
> raco test: "pkgs/rackunit-pkgs/rackunit-test/tests/rackunit/util-test.rkt"
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev



-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93