[racket] Writing checkers for handin-server

At Tue, 6 Jan 2015 16:33:04 -0500, Suzanne Menzel wrote:
> I’m trying to set up the handin-server to use in my class this semester and 
> I’ve run into a few problems that I hope someone can help with.
> 
> I’ve got the server up and running and the Handin button appears in DrRacket. I 
> currently have only one assignment, called a1. Everything works fine when 
> submitting this sample homework file (on the client machine):
> 
> ;; contents of student submission a1.rkt
> (check-expect (double-inc 4) 9)
> (check-expect (double-inc 0) 1)
> 
> (define (double-inc x)
>   (add1 (* 2 x)))
> 
> with this checker (on the server machine):
> 
> ;; contents of a1/checker.rkt
> (module checker handin-server/checker
>   (require handin-server/grading-utils)
>   (pre:
>    (check-deadline))
> 
>   (check: :language  '(special beginner)
>    (update-submission-timestamp!)
>    (add-header-line! (get-submission-timestamp))
>    (add-report-line! (get-submission-timestamp))
> 
>    (!test (double-inc 2) 5)
>    (!test (double-inc 3) 7)
>    (!test (double-inc 0) 1)
>    (!test (double-inc -5) -9))
> 
>   (post:                                                           
>    #f)
>   )
> 
> So far, so good. Now, I want to add a !procedure check before the tests for 
> double-inc, like this:
> 
> (!procedure double-inc 1)
> 
> This produces the following error:
> 
> submit error: while evaluating double-inc:
>   eval:1:0: double-inc: expected a function call, but there is no open 
> parenthesis before this function
>   in: double-inc
> 
> So, I change the language to (special advanced) and the error goes away. This 
> brings me to my first question:
> 
> (1) How can I set it up so that the checker runs in racket and the evaluator 
> for the tests run in BSL (for example)?

I think you can't use `!procedure` with the Beginner language. You
should be able to use `!defined`, though.

More generally, I think you could use `!eval` and use the results in
arbitrary Racket expressions that use the result (and potentially raise
an exception for errors), but I think you always have to start with
something in `eval!` that's legal in the Beginner program.


> Next, I tried putting in the coverage check. 
> 
> coverage? #t
> 
> This produces the error (even though all lines are covered by check-expects):
> 
> submit error: your code is not completely covered by tests: uncovered 
> expression at 1:0 ()
> 
> (2) How can I get the coverage check to work?

I think coverage checking is broken, unfortunately; I see that I had it
commented out the last time I used a checker with Beginner programs. If
no one else gets to it, I'll try to investigate sometime soon.


> Finally, I tried putting write-report in the post:
> 
> (write-report)
> 
> And here’s the error:
> 
> commit error: start-timer: arity mismatch;
>  the expected number of arguments does not match the given number
>   expected: 3
>   given: 2
>   arguments...:
>    0
>    #<procedure>
> 
> (3) What’s up with the error from start-timer?!

I've never tried to use `write-report` myself, but I can take a look at
that when investigating coverage (again, if no one else gets to it
first).


> Any advice would be greatly appreciated. Also, if anyone has any
> sample checkers that they wouldn’t mind sharing with me, that would
> be terrific. I’d love to see what others have already developed.

I'll send you my checkers.