[plt-dev] lessons learned

On May 22, Jacob Matthews wrote:
> I don't mean to argue with you about this, but I don't really find
> these points convincing. After all, we manage to make it work even
> though I'm pretty sure we've got way more code than PLT Scheme. (To
> quote Kevin Bourrillion, Google's Java library curator: "If code
> were ice cream, we would have ... a whole lot of ice cream.")

re (a) I'm sure that you also are running things a little faster, and
utilizing a few more cpus on the way.  Last time (only time) I tried
to compile all of plt in errortrace mode, it took around 2 hours;
extrapolating from the current build times, this means that the
solaris run will take more than 8 hours not counting running the code.

re (b) -- yes, you have much more code, probably enough that keeping
it all in memory is impractical anyway.  This is another possible
improvement to errortrace: keep the coverage points on disk.  It won't
help with the current system though, and in any case we don't have a
server farm.  Not even a server garden.  More like a window box in my
office and sneaking into other people's parties at night.  This is
also related to my (c) point.


> I think what you're really saying is that it's impractical to test
> all of PLT Scheme with one central testing service that tries to do
> the whole thing at once. I can definitely believe that. I think the
> key might be in making the process more modular --- a continuous
> build with "all of PLT Scheme" as its target is orders of magnitude
> too big to be very useful IMO. One way to chop things up would be to
> have separate builds for different components: a mzscheme build, a
> mred build, a drscheme build (or maybe further break down DrS into
> smaller pieces and potentially several other components for
> different collects. Each sub-build runs the test suites appropriate
> to it, and just measures code coverage in the files it cares about.
> (In practice, even though a test suite written to cover, say, a mred
> component might incidentally cover some mzlib stuff, that coverage
> isn't very high-quality and probably shouldn't count as mzlib tests
> anyway.)  Sub-builds can run independently on different machines,
> set up by individual groups. When I was working on planet, for
> instance, I could've set up my own sub-project just for it, and had
> the system run on my local machine. (I do effectively the same thing
> on my current project.)

Yes, all of that makes sense.  Care to donate a small farm?  IOW, it
will require work to make this possible -- not much work, but
currently there is no visible gains from it.


> One way to think about it is: suppose you're monitoring the the
> output of tests, and you get a message saying some tests have
> failed. Do you care? If you don't, you need to think about making
> better targets, and only monitoring the targets for which you can
> unhesitatingly say yes.  This will incidentally make it a lot easier
> for smaller projects to get up and running.

That's a good point too.  Had there been more tests, I'd arrange for
failure alerts to be sent to the responsible parties.  Currently, I
get an email when tests fail, and I nag the responsible party.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                  http://www.barzilay.org/                 Maze is Life!