[racket-dev] Performance Lab

It would be fantastic to have this kind of help. Thank you for offering! As
Neil said, you'll want to consult with Jay some to avoid duplication of
work, but also Eli can help with sending out notifications when pushes to
the repo happen.

Performance testing is a place where I think our automated testing support
is especially weak, too. Thanks again!

Robby


On Mon, Jan 21, 2013 at 11:25 PM, Curtis Dutton <curtdutt at gmail.com> wrote:

> I've been using racket now for about 4 years now. I use it for everything
> that I can and I love it. It is really an awesome system, and I just can't
> say "THANKS" enough to all of you for racket.
>
> That being said, I'd like to become more active with the development
> process. In a past life, I worked for Microsoft as a development tools
> engineer. Most of what I did was develop and operate large scale automated
> testing systems for very large teams. I once had a room full of 500 or so
> machines at my disposal, which I tried hard to keep busy. I've maintained
> rolling build systems, that included acceptance tests, a performance
> testing system, a stress testing system and a security fuzzing system.
>
> I'm not sure how people feel about automated systems like this, part of
> this email is just to see what people think. But used in the right way they
> can be used to shape and control the directions that a project evolves in.
>
>
> An example of the type of system that I'd like to see for racket, would be
> a performance measuring system that would work in principle like so...
>
> I have an exampled I'll use. I'm concerned about the racket/openssl
> transfer speeds.
>
> The test:
>
>    1.  Create 2 places. 1 with a client. 1 with a server.
>    2.  Establish an ssl session.
>    3.  Output a "start time event".
>    4.  transfer 1MB of random data.
>    5.  output an "end time event"
>
>
> Now once I write that test, and commit it, the performance system picks it
> up from the repository. And it runs that test for every commit that is made
> there after. That establishes a "baseline" for the performance of that
> test. If a commit is made, and suddenly that test takes longer, it
> generates an alert. At which point, we either investigate to find out why
> the test slowed down and fix it, or due to circumstances we can't control
> (which does happen) we tell the system that its acceptable and to accept it
> as a new baseline. Now of course if there is a marked improvement, we sound
> out a pat on the back too!
>
> Now as a user of this system, I can monitor the performance
> characteristics of racket that I care about. People can write "tests" just
> to track racket's performance over time, and catch unexpected regressions.
> They can also add these tests before they begin on a campaign of improving
> their pet measurements.
>
>
> That is the gist of the type of system I wish I had with racket.
>
> I can go more into how a stress test works, and perhaps fuzzing tests,
> etc...
>
>
> Now I'm willing to build it and I'm willing to host it with a number of
> machines. I have pieces and parts of code lying around and I already have a
> decent harness implementation that collects statistics about a racket
> process as it runs.
>
>
> What do you think? If could have something like this, would you want it?
> (Does something like this exist already?) What would it look like? How
> would it work, etc....
>
>
> I'd like to collect a list of desired "tests" that this system would
> monitor for us. If you already have code that you run on your own, even
> better! Detailed examples would be welcome, as I need to gather some ideas
> about what people would want to do with this thing.
>
> Racket is so awesome! I'd like to help improve it, and I think this is
> something that I can offer to help get us there.
>
> Thanks,
>     Curt
>
>
>
>
>
>
>
>
>
>
>
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/dev/archive/attachments/20130122/162abc45/attachment-0001.html>