[plt-dev] Continuous Integration Server

I'd like to get some comments on what I have so far and stuff I plan on doing.

--- What I Have ---
If you go to

http://faculty.cs.byu.edu/~jay/tmp/14986/

You'll see the interface for a sample run of my tester on revision
14986. [No particular reason Robby]

The essential details about the revision are listed at the top (notice
the links to inside the testing logs) followed by a table summarizing
the run. This table always shows when there is a directory.

Each entry on the table depends on if the path is a file or a directory.

If it is a file, then you have (1) how long it took to "mred-text -t
<path>" the path; (2) whether the execution timed out (current timeout
for everything is 10 minutes); (3) whether mred-text exited cleanly
(meaning with status code 0); (4) whether there was output to STDERR.

If it is a directory, you see the same information, except it is a sum
of the contents of the directory.

At the bottom is the entry for the entire directory.

The path name on the left is a link to a page for that directory or
file. As you browse, the breadcrumbs at the top of the page
accumulate. Each sub-path is a link to the corresponding page, as
you'd expect.

If you go to the page for a file

http://faculty.cs.byu.edu/~jay/tmp/14986/src/build/make.html

or http://faculty.cs.byu.edu/~jay/tmp/14986/collects/frtime/gui/mod-mrpanel_ss.html

are decent choices for this demo

You'll have more information about the run, including the log. The
stdout output is black; the stderr output is red.

--- What I'd Like From You ---

1) Comments on the interface and information you would want to be displayed

2) A suggestion for a name [I'm thinking pis: the PLT Integration Server =P]

3) Comments on the ideas below

--- What I Plan On Doing ---

Here are some things I know I am planning on doing:

* Determining if a file tests differently than the previous revision.

Combining this with the saving of terminal output is, IMHO, a fairly
robust way to locate errors and be testing suite agnostic. Basically,
there will be another column that says "Changed?" and whether the
output has changed. This is will be the basis of the "nag" emails with
some heuristics to avoid unnecessary naggery.

For example, if the file you just edited in the commit displays
something different, it won't consider it any error if it always
displayed something in that way. For example, if X.ss never printed
anything on stderr than it will nag if it starts to even though you
just edited it. However, if X.ss always printed on stderr, it won't
nag if it prints something different. Obviously these heuristics will
be very fluid.

* Using Subversion properties to set the timeout on a per-file basis

This will help the build not wait for ever for DrScheme, as it won't
complete. By including it in Subversion, it will be versioned with the
file, so the metadata is not in a magical place on the server.

* Using Subversion properties to set the command-line options and
execution program

Most of the files can be run in mzscheme, but about 1000 need to be
run in mred. Also, many files (particular in
collects/tests/mzscheme/benchmarks) need command-line arguments to run
properly. This will include an option to ignore files. The default
will be that if a file ends in .ss, .scm, or .scrbl then mzscheme -t
will do it. Again, if these are in Subversion, then it is more
transparent and trackable what the test server should be doing.

[I will set the initial version of these properties; no need to worry about it]

* Jump to output changes on first page

* Emitting the status of a build on twitter

* Nag emails to committer and plt:responsible

* Client-side sorting of directory listings

* Eventually I'd like to do two different kinds of builds: A "fast"
build that uses the previous slow build, but updates to the next
version and perhaps does some sort of dependency heuristic to not run
everything. A "clean" build corresponding to this. The goal would be
that the fast build would be done within 30 minutes, but the clean
would be available in a few hours.

--- Some Data ---

It took 7.34 hours to do the build on my Macbook in power save mode.
At the average of 8 commits per day, this is too slow to keep up with
the edge, but perhaps a better machine will do better. I currently
don't even run test two files at once. (I could because I have dual
CPUs on the laptop.) Plus with better timeouts I could shave off
almost 5 hours.

It took 500MBs of space for the source & compiled source.

It took 18MBs for the output logs.

The UI takes 20MBs.

There were 20 files that were created in `pwd`. (Most of them with
bizarre names.)

There were 20 PLaneT packages installed.

-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://teammccarthy.org/jay

"The glory of God is Intelligence" - D&C 93