[racket-dev] Performance Lab

[Disclaimer: I'm not part of the core team, nor do I play one on TV.]

A few miscellaneous thoughts:

1. I love the idea of performance regression testing, with history
charts. (I read a great example of this a few months ago, but can't
dredge it up right now.)

2. I've wanted to do this on some of my own projects. It would be
awesome if the same tools used for Racket itself could be used by
individuals for their collections. (Maybe they have to provide their
own hardware or EC2 instances).

2.5 Racket will be breaking up its distribution into packages. I think
this sorta relates to 2?

3. Recently I was impressed to see how Travis integrates with GitHub.
For example if you make a pull request to a Linguist project, Travis
tests automatically run and the pass/fail result posted in the pull
request thread. (AFAIK Travis isn't _performance_ regression testing,
but still.)

4. Similarly, it could be neat if Planet (1 and/or 2) could show a
Travis-like test result badge for a package. And even a performance
test history chart. We could see at a glance that a package has been
trending faster or slower; more information to evaluate it.


On Tue, Jan 22, 2013 at 12:25 AM, Curtis Dutton <curtdutt at gmail.com> wrote:
> I've been using racket now for about 4 years now. I use it for everything
> that I can and I love it. It is really an awesome system, and I just can't
> say "THANKS" enough to all of you for racket.
>
> That being said, I'd like to become more active with the development
> process. In a past life, I worked for Microsoft as a development tools
> engineer. Most of what I did was develop and operate large scale automated
> testing systems for very large teams. I once had a room full of 500 or so
> machines at my disposal, which I tried hard to keep busy. I've maintained
> rolling build systems, that included acceptance tests, a performance testing
> system, a stress testing system and a security fuzzing system.
>
> I'm not sure how people feel about automated systems like this, part of this
> email is just to see what people think. But used in the right way they can
> be used to shape and control the directions that a project evolves in.
>
>
> An example of the type of system that I'd like to see for racket, would be a
> performance measuring system that would work in principle like so...
>
> I have an exampled I'll use. I'm concerned about the racket/openssl transfer
> speeds.
>
> The test:
>
>  Create 2 places. 1 with a client. 1 with a server.
>  Establish an ssl session.
>  Output a "start time event".
>  transfer 1MB of random data.
>  output an "end time event"
>
>
> Now once I write that test, and commit it, the performance system picks it
> up from the repository. And it runs that test for every commit that is made
> there after. That establishes a "baseline" for the performance of that test.
> If a commit is made, and suddenly that test takes longer, it generates an
> alert. At which point, we either investigate to find out why the test slowed
> down and fix it, or due to circumstances we can't control (which does
> happen) we tell the system that its acceptable and to accept it as a new
> baseline. Now of course if there is a marked improvement, we sound out a pat
> on the back too!
>
> Now as a user of this system, I can monitor the performance characteristics
> of racket that I care about. People can write "tests" just to track racket's
> performance over time, and catch unexpected regressions. They can also add
> these tests before they begin on a campaign of improving their pet
> measurements.
>
>
> That is the gist of the type of system I wish I had with racket.
>
> I can go more into how a stress test works, and perhaps fuzzing tests,
> etc...
>
>
> Now I'm willing to build it and I'm willing to host it with a number of
> machines. I have pieces and parts of code lying around and I already have a
> decent harness implementation that collects statistics about a racket
> process as it runs.
>
>
> What do you think? If could have something like this, would you want it?
> (Does something like this exist already?) What would it look like? How would
> it work, etc....
>
>
> I'd like to collect a list of desired "tests" that this system would monitor
> for us. If you already have code that you run on your own, even better!
> Detailed examples would be welcome, as I need to gather some ideas about
> what people would want to do with this thing.
>
> Racket is so awesome! I'd like to help improve it, and I think this is
> something that I can offer to help get us there.
>
> Thanks,
>     Curt
>
>
>
>
>
>
>
>
>
>
>
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>