[racket-dev] Performance Lab

On 01/21/2013 10:25 PM, Curtis Dutton wrote:
> I've been using racket now for about 4 years now. I use it for
> everything that I can and I love it. It is really an awesome system, and
> I just can't say "THANKS" enough to all of you for racket.
>
> That being said, I'd like to become more active with the development
> process. In a past life, I worked for Microsoft as a development tools
> engineer. Most of what I did was develop and operate large scale
> automated testing systems for very large teams. I once had a room full
> of 500 or so machines at my disposal, which I tried hard to keep busy.
> I've maintained rolling build systems, that included acceptance tests, a
> performance testing system, a stress testing system and a security
> fuzzing system.
>
> I'm not sure how people feel about automated systems like this, part of
> this email is just to see what people think. But used in the right way
> they can be used to shape and control the directions that a project
> evolves in.

We feel very good about them, and we can always use a lot more.

Every push, a machine named "DrDr" (see drdr.racket-lang.org) rebuilds 
all of Racket, including GUI stuff and tools, from the updated codebase, 
and runs almost every file. Many of them are in a directory called 
"tests" and subdirectories of top-level collections like "plot/tests". 
DrDr emails the pusher and any responsible parties if tests fail or a 
file's output changes.

Most of these tests are standard, deterministic tests. Some are 
randomized tests (Typed Racket and the math library do this, and 
probably Redex). A small number are stress tests and performance tests.

Here's where I can't speak for everybody, just myself. It seems to me 
that most of the testing is ad-hoc, depending on the whims of the 
collection owners. It could be very nice to standardize most of it.

> Now I'm willing to build it and I'm willing to host it with a number of
> machines. I have pieces and parts of code lying around and I already
> have a decent harness implementation that collects statistics about a
> racket process as it runs.

Jay McCarthy owns DrDr, and runs it from our research lab. You'll want 
to talk to him about hardware.

> I'd like to collect a list of desired "tests" that this system would
> monitor for us. If you already have code that you run on your own, even
> better! Detailed examples would be welcome, as I need to gather some
> ideas about what people would want to do with this thing.

I've wanted more automated testing for `math/special-functions' and 
`math/distributions' exports, and performance testing for `math/array' 
and `math/matrix', but I haven't gotten around to doing it. I'm not sure 
what I'd use as a baseline for the latter two. For `math/distributions', 
I've got no idea how to reliably test a sampler, since there's always a 
nonzero probability that any computed statistic is wrong. Do you have 
experience with that?

Neil ⊥