[plt-dev] Continuous Integration Server

On May 27, Jay McCarthy wrote:
> I'd like to get some comments on what I have so far and stuff I plan
> on doing.

In complete as-I-see-it random order:

* Nice interface, but the links are kind of easy to miss (since some
  are the usual blue, and some are black).

* I like the "alert only on change" policy -- that solves a bunch of
  issues.  But it might be problematic if there are files that produce
  random output -- so plan on having the meta-data should hold
  information like directions to ignore the file, the output, stderr
  output, exit code, etc.

* If you're already computing the durations, then this would be
  another good point: if the time is very different than the previous
  measurement, then it's worth an alert.  Also, it might be good to
  keep some historical data (perhaps on some selected files) to see
  how mzscheme is doing in general.  (The plan is that I'll eventually
  add that to the nightly build too.)

* Also, allowing to sort the result list on the time could be a useful
  indicator that something is wrong.

* You can probably tackle timeouts in a similar way.  Another thing to
  do with things like drscheme and plt-games is have the meta-data
  specify that a new window is expected to pop up with some specific
  title, and just declare success when you see it (see
  "iplt/build/test-drscheme.ss" -- it does that for drscheme).

* It might be better to read a file and run it only if it's a module
  by the expected name, to avoid running bogus files.  (Looks like
  this is the bulk of the loading errors.)  Since this depends only on
  the contents of the file, you could just cache it.  (Perhaps by just
  catching the right load error and ignoring it.)

* Links from a file's page to the svn location would be nice.  Maybe
  even dump the file's contents on the same page.  (Makes it much
  easier to see that some problem is expected.)

* The font used for the file rows is different from the Total row.
  (Makes it weird when there are two 1s that look different...)

* Also, the meta-data should contain instructions that apply to a
  whole collection -- this will be usful in the case of
  tests/mzscheme, where it doesn't make sense to load each file, since
  there's a central file that loads them all anyway.  (So it should
  probably look like some inclusion exclusion thing, where I can say
  "run only all.ss like that, and ignore all other files".)

* If you're running a separate process for each file, you can just as
  well clean the directory and avoid those problems.

* I'll repeat a point I made earlier -- putting the meta data on svn
  properties means that some people will never use that.  I'd prefer
  some file (maybe in iplt/build, maybe even using the same language
  that is used for the specs).

> Combining this with the saving of terminal output is, IMHO, a fairly
> robust way to locate errors and be testing suite agnostic. Basically,
> there will be another column that says "Changed?" and whether the
> output has changed. This is will be the basis of the "nag" emails with
> some heuristics to avoid unnecessary naggery.

But when would "change" go away?  My best guess for this involves
people logging in and clicking somewhere to acknowledge that a change
is expected.

> For example, if the file you just edited in the commit displays
> something different, it won't consider it any error if it always
> displayed something in that way. For example, if X.ss never printed
> anything on stderr than it will nag if it starts to even though you
> just edited it. However, if X.ss always printed on stderr, it won't
> nag if it prints something different. Obviously these heuristics will
> be very fluid.

I think that nagging also on a different output would be good, with a
meta-data way to avoid it.

> * Using Subversion properties to set the timeout on a per-file basis
> 
> This will help the build not wait for ever for DrScheme, as it won't
> complete. By including it in Subversion, it will be versioned with
> the file, so the metadata is not in a magical place on the server.

The expected window title works better for this, IMO.  Adds a test
instead of a fragile timeout.  (And BTW, timeouts can be very fragile,
since the OS might do something like a backup which will throw
everything off.)

> * Emitting the status of a build on twitter

(Is that *really* necessary...?)

> * Eventually I'd like to do two different kinds of builds: A "fast"
> build that uses the previous slow build, but updates to the next
> version and perhaps does some sort of dependency heuristic to not
> run everything. A "clean" build corresponding to this. The goal
> would be that the fast build would be done within 30 minutes, but
> the clean would be available in a few hours.

The time to run each of the files is probably far more dominating than
the time to do a full build.

> It took 7.34 hours to do the build on my Macbook in power save mode.
> At the average of 8 commits per day, this is too slow to keep up
> with the edge, but perhaps a better machine will do better. I
> currently don't even run test two files at once. (I could because I
> have dual CPUs on the laptop.) Plus with better timeouts I could
> shave off almost 5 hours.

IME, getting a linux installation on a good recent machine will be
much faster.

> It took 500MBs of space for the source & compiled source.

Actually I have that at 600MB now.  (Which can lead to some issues
when dealing with the nightly builds -- since that's multiplied by the
number of builds, then there are several packages for each OS...)

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                  http://www.barzilay.org/                 Maze is Life!