[plt-dev] lessons learned

Some items to consider (in mostly random order):

* [Disclaimer: there are many items here -- that is not an indication
  that I'm objecting to this, but the exact opposite.  I'm not trying
  to pile up difficulties, instead, I'm trying to list things that you
  need to be aware of -- and maybe it'll make it more practical to
  make this happen.]

* As a general warning, you should plan on spending a significant
  amount of time on maintaining this thing -- by its nature it's close
  to things like the build script or the release procedure -- and I
  find that these things are very unstable in the sense that I'm often
  updating things.  (Not unstable because the code is bad, but because
  the project moves.)  For example, the big commit when Matthew pushed
  the 4.0 world in left a lot of files in bad state and it took a
  while for things to settle -- you'll need to be prepared to tweak
  things at such times.

* While nagging the responsible parties is good in general, you need
  to avoid overdoing it -- for example, if that 4.0 jump leaves a
  large number of files in a bad state which will take you a while to
  fix, you'll eventually start ignoring the incoming nag messages.

* Getting plt:responsible to actually work will be nice, but probably
  will be inconvenient enough that it will still be undermaintained.
  Perhaps it's best to have some heuristic assign ownership to
  existing files, and make the ownership go to whoever adds new
  files.  By "existing files" I mean stuff that existed when the svn
  was made.  (It will need to ignore the first ~5 revisions, where I
  set the repository up.)

* Also, be prepared for disappointments -- like investing time in
  getting those per-file pages up with all the relevant info but
  having noone ever use them.  (That's also very much like the build
  stuff...)

* Some meta-data should be helpful -- since some files are not meant
  to run, and there are several reasons that can lead to this.  I'm
  not sure what's the best way to put this data but svn properties
  might not be best because they're lost when it goes out of svn.

* Assumptions like "any output to stderr is an error" are very bad,
  even an assumption like "an exception is a testing error" is not
  always true.  It'll be challenging to figure it out -- meta data can
  help, but you don't want to overdo it by making it verbose enough
  that nobody will actually do so.  (This is almost a guaranteed way
  of finding yourself doing it for everyone...)

* You will need to sandbox it somehow.  It's probably most convenient
  to make it run from a dedicated username -- and obviously having a
  dedicated machine covers that.  But you'll need to take care of some
  OS restrictions so that the machine doesn't crash when running out
  of memory, while still keeping a hard memory limit (since running
  inside a scheme sandbox is not always practical.)

* The platform should probably be linux, to be able to run mred
  remotely.  Again, if it's a dedicated box it can be Windows or OSX
  with some screen controlling utility, but linux is still more
  flexible, I think.

* That said, there are differences between the different platforms --
  especially in mred, but also in other files, so ideally the setup
  would run on three machines running the three platforms.  It might
  be possible to get this with VM or something, but it can be
  difficult to set it up.

* Some files can have side-effects that conflict other files.  The
  brute force solution is to run each file in a new process -- and
  that might be feasible if the machine is doing just this, all the
  time.  OTOH, some files might require loading other files, so
  solving this will be difficult.

* This also goes for running stuff in mred or mzscheme.  The simple
  solution is to run everything in mred, but that's also more
  expensive (especially with one process per file).  You might be able
  to use the dependencies to know when a file requires mred.

* It's a good idea to combine loading the files and running the tests
  in this process.

* It will be nice to have a big queue of files to try, so whenever a
  file is modified (or a new one is made) it is moved up to the
  beginning.  When the queue is done, reinitizlize from all.  You can
  trigger a modification "event" by the notification emails, or I can
  set up something in the commit hook.

* Assigning blame using plt:responsible might be tricky -- what if A's
  test file fails because it depends on B's files, which depends on
  C's file which D changed?  The ideal of having everything go
  smoothely so there's always a known blame (the recent comitter) is
  never going to happen -- some files will have had failures from
  before, and some blame is actually reversed.  For example, D's
  change might be fully justified because it's a fix to an unrelated
  problem, but that exposed a problem in B's code.  Another example is
  the a major version switch -- where lots of code breaks "by design".

* Running setup-plt after every commit is usually fine, but in some
  cases it might be an issue.  As an approximation, running `make' on
  the existing build directory is good when any C file changes, and
  running setup-plt when other files change -- but in addition, you'll
  want to clear things up when the version changes.  Also, it will be
  good to periodically (daily or weekly, maybe?) remove everything and
  start from a fresh checkout.

* I wasn't referring to errortrace's capability of tracking the stack,
  but the coverage collection tool which is included in it.  I don't
  think that there are much cheaper alternatives to annotating code,
  so if you do want to test coverage, you'll need a high cpu price
  either way you go.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                  http://www.barzilay.org/                 Maze is Life!