[plt-dev] Continuous Integration Server

On May 28, Jay McCarthy wrote:
> 
> On Wed, May 27, 2009 at 11:27 PM, Eli Barzilay <eli at barzilay.org> wrote:
> >
> > * If you're running a separate process for each file, you can just
> >   as well clean the directory and avoid those problems.
> 
> Which directory do you mean here?

I was referring to the problem of having file leftovers -- so between
running processes you can just remove any leftover contents in the
directory you're running it from (which can be any random directory).


> > * I'll repeat a point I made earlier -- putting the meta data on
> >   svn properties means that some people will never use that.  I'd
> >   prefer some file (maybe in iplt/build, maybe even using the same
> >   language that is used for the specs).
> 
> I don't understand this point. If it is some file in iplt then
> there's an even bigger disincentive for people to update it
> themselves. If it is a property it is not terribly hard to do a:
> 
> svn pset plt:tester:command-line "mzscheme -qu ~a" the-file.ss
> 
> or
> 
> svn pset plt:tester:timeout "60" the-file.ss
> 
> Maybe you are saying that in your experience people won't ever do
> anything, so I might as well make it easier for me.

Yes, that's a better way of putting it...  I think that it's largely
because people don't usually need it (setting properties), so it's
usually just beyond the short-term memory that is required to use it.
Keeping it in one file will be easier for you, since crawling over the
svn tree and making batch changes can be less convenient.


> >> Combining this with the saving of terminal output is, IMHO, a
> >> fairly robust way to locate errors and be testing suite
> >> agnostic. Basically, there will be another column that says
> >> "Changed?" and whether the output has changed. This is will be
> >> the basis of the "nag" emails with some heuristics to avoid
> >> unnecessary naggery.
> >
> > But when would "change" go away?  My best guess for this involves
> > people logging in and clicking somewhere to acknowledge that a
> > change is expected.
> 
> These changes would be computed on a per-revision basis. The only
> data is whether it is changed relative to the last revision.
> Therefore, there's no reason to acknowledge anything on the part of
> users. Am I misunderstanding your question?

Possibly.  I think that a very common situation is that you're
changing something, and you get back an alert -- the alert means that
there's something that needs fixing, but since you don't have time
right now you decide to deal with it later.  So ideally, it would be
nice to have a page of changes that happened in files that are my
responsibility (and/or files that are someone else's but that I
changed or broke), and I can visit that page at any time and mark off
things that are fine, or that I have dealt with somehow (even just
confirmed that the output is expected).


> >> For example, if the file you just edited in the commit displays
> >> something different, it won't consider it any error if it always
> >> displayed something in that way. For example, if X.ss never printed
> >> anything on stderr than it will nag if it starts to even though you
> >> just edited it. However, if X.ss always printed on stderr, it won't
> >> nag if it prints something different. Obviously these heuristics will
> >> be very fluid.
> >
> > I think that nagging also on a different output would be good, with a
> > meta-data way to avoid it.
> >
> >> * Using Subversion properties to set the timeout on a per-file basis
> >>
> >> This will help the build not wait for ever for DrScheme, as it won't
> >> complete. By including it in Subversion, it will be versioned with
> >> the file, so the metadata is not in a magical place on the server.
> >
> > The expected window title works better for this, IMO.  Adds a test
> > instead of a fragile timeout.  (And BTW, timeouts can be very
> > fragile, since the OS might do something like a backup which will
> > throw everything off.)
> 
> This machine will do nothing else (including backups, because I
> could just rerun the software... everything it does is soft state.)

The OS will still do other things.  For example, you might run two
tests, the first uses a lot of memory, which makes the next one load
very slowly because the FS cache is all gone.

Which also reminds me: when you scan the directory for files to run,
sort them.  Without that there might be such differences due to the
particular order the files are in -- it's obviously best to not have
such things, but even worse to run into a situation that is
unreproducible because of it.  (At some point there was a problem like
that with setup-plt, and after a long chase session I made it compile
files and directories alphabetically.)


> >> * Emitting the status of a build on twitter
> >
> > (Is that *really* necessary...?)
> 
> Well, I use twitter, so it would be nice for me to know what the
> server is doing without getting lots of emails or having to check a
> site. (My main point in including the bullet is that you won't
> shouldn't expect informative emails unless something bad happened.)

(Yeah, well, I *prefer* getting everything through emails...  Much
easier when there's a single entry point.  I had a similar discussion
with Jon Rafkind who wanted to do some subversion thing that sends
commit messages on an IRC channel.)


> >> * Eventually I'd like to do two different kinds of builds: A "fast"
> >> build that uses the previous slow build, but updates to the next
> >> version and perhaps does some sort of dependency heuristic to not
> >> run everything. A "clean" build corresponding to this. The goal
> >> would be that the fast build would be done within 30 minutes, but
> >> the clean would be available in a few hours.
> >
> > The time to run each of the files is probably far more dominating
> > than the time to do a full build.
> 
> We're using the word "build" differently. I mean: everything my
> software does, i.e., running all the files. My "fast" version would
> not run _everything_ but would have some heuristic about what files
> should be run given commits. (Such as, everything in the same
> collection.)

OK.  This is probably not too difficult using the dependency files.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                  http://www.barzilay.org/                 Maze is Life!