[plt-dev] Continuous Integration Server

Thanks for the comments, Eli. I will act on most of them, I have a few
questions below though.

On Wed, May 27, 2009 at 11:27 PM, Eli Barzilay <eli at barzilay.org> wrote:
> On May 27, Jay McCarthy wrote:
>> I'd like to get some comments on what I have so far and stuff I plan
>> on doing.
>
> In complete as-I-see-it random order:
>
> * Nice interface, but the links are kind of easy to miss (since some
>  are the usual blue, and some are black).
>
> * I like the "alert only on change" policy -- that solves a bunch of
>  issues.  But it might be problematic if there are files that produce
>  random output -- so plan on having the meta-data should hold
>  information like directions to ignore the file, the output, stderr
>  output, exit code, etc.
>
> * If you're already computing the durations, then this would be
>  another good point: if the time is very different than the previous
>  measurement, then it's worth an alert.  Also, it might be good to
>  keep some historical data (perhaps on some selected files) to see
>  how mzscheme is doing in general.  (The plan is that I'll eventually
>  add that to the nightly build too.)
>
> * Also, allowing to sort the result list on the time could be a useful
>  indicator that something is wrong.
>
> * You can probably tackle timeouts in a similar way.  Another thing to
>  do with things like drscheme and plt-games is have the meta-data
>  specify that a new window is expected to pop up with some specific
>  title, and just declare success when you see it (see
>  "iplt/build/test-drscheme.ss" -- it does that for drscheme).
>
> * It might be better to read a file and run it only if it's a module
>  by the expected name, to avoid running bogus files.  (Looks like
>  this is the bulk of the loading errors.)  Since this depends only on
>  the contents of the file, you could just cache it.  (Perhaps by just
>  catching the right load error and ignoring it.)
>
> * Links from a file's page to the svn location would be nice.  Maybe
>  even dump the file's contents on the same page.  (Makes it much
>  easier to see that some problem is expected.)
>
> * The font used for the file rows is different from the Total row.
>  (Makes it weird when there are two 1s that look different...)
>
> * Also, the meta-data should contain instructions that apply to a
>  whole collection -- this will be usful in the case of
>  tests/mzscheme, where it doesn't make sense to load each file, since
>  there's a central file that loads them all anyway.  (So it should
>  probably look like some inclusion exclusion thing, where I can say
>  "run only all.ss like that, and ignore all other files".)
>
> * If you're running a separate process for each file, you can just as
>  well clean the directory and avoid those problems.

Which directory do you mean here?

> * I'll repeat a point I made earlier -- putting the meta data on svn
>  properties means that some people will never use that.  I'd prefer
>  some file (maybe in iplt/build, maybe even using the same language
>  that is used for the specs).

I don't understand this point. If it is some file in iplt then there's
an even bigger disincentive for people to update it themselves. If it
is a property it is not terribly hard to do a:

svn pset plt:tester:command-line "mzscheme -qu ~a" the-file.ss

or

svn pset plt:tester:timeout "60" the-file.ss

Maybe you are saying that in your experience people won't ever do
anything, so I might as well make it easier for me.

>
>> Combining this with the saving of terminal output is, IMHO, a fairly
>> robust way to locate errors and be testing suite agnostic. Basically,
>> there will be another column that says "Changed?" and whether the
>> output has changed. This is will be the basis of the "nag" emails with
>> some heuristics to avoid unnecessary naggery.
>
> But when would "change" go away?  My best guess for this involves
> people logging in and clicking somewhere to acknowledge that a change
> is expected.

These changes would be computed on a per-revision basis. The only data
is whether it is changed relative to the last revision. Therefore,
there's no reason to acknowledge anything on the part of users. Am I
misunderstanding your question?

>
>> For example, if the file you just edited in the commit displays
>> something different, it won't consider it any error if it always
>> displayed something in that way. For example, if X.ss never printed
>> anything on stderr than it will nag if it starts to even though you
>> just edited it. However, if X.ss always printed on stderr, it won't
>> nag if it prints something different. Obviously these heuristics will
>> be very fluid.
>
> I think that nagging also on a different output would be good, with a
> meta-data way to avoid it.
>
>> * Using Subversion properties to set the timeout on a per-file basis
>>
>> This will help the build not wait for ever for DrScheme, as it won't
>> complete. By including it in Subversion, it will be versioned with
>> the file, so the metadata is not in a magical place on the server.
>
> The expected window title works better for this, IMO.  Adds a test
> instead of a fragile timeout.  (And BTW, timeouts can be very fragile,
> since the OS might do something like a backup which will throw
> everything off.)

This machine will do nothing else (including backups, because I could
just rerun the software... everything it does is soft state.)

>
>> * Emitting the status of a build on twitter
>
> (Is that *really* necessary...?)

Well, I use twitter, so it would be nice for me to know what the
server is doing without getting lots of emails or having to check a
site. (My main point in including the bullet is that you won't
shouldn't expect informative emails unless something bad happened.)

>
>> * Eventually I'd like to do two different kinds of builds: A "fast"
>> build that uses the previous slow build, but updates to the next
>> version and perhaps does some sort of dependency heuristic to not
>> run everything. A "clean" build corresponding to this. The goal
>> would be that the fast build would be done within 30 minutes, but
>> the clean would be available in a few hours.
>
> The time to run each of the files is probably far more dominating than
> the time to do a full build.

We're using the word "build" differently. I mean: everything my
software does, i.e., running all the files. My "fast" version would
not run _everything_ but would have some heuristic about what files
should be run given commits. (Such as, everything in the same
collection.)

>
>> It took 7.34 hours to do the build on my Macbook in power save mode.
>> At the average of 8 commits per day, this is too slow to keep up
>> with the edge, but perhaps a better machine will do better. I
>> currently don't even run test two files at once. (I could because I
>> have dual CPUs on the laptop.) Plus with better timeouts I could
>> shave off almost 5 hours.
>
> IME, getting a linux installation on a good recent machine will be
> much faster.
>
>> It took 500MBs of space for the source & compiled source.
>
> Actually I have that at 600MB now.  (Which can lead to some issues
> when dealing with the nightly builds -- since that's multiplied by the
> number of builds, then there are several packages for each OS...)
>
> --
>          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
>                  http://www.barzilay.org/                 Maze is Life!
>



-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://teammccarthy.org/jay

"The glory of God is Intelligence" - D&C 93