[racket-dev] DrDr Feature Request

I love DrDr, but there's a small thing that annoys me about it.

Some tests are prone to intermittent failures. For example, some
benchmarks need to create a file, and several benchmarks share the
same file, which leads to race conditions. Similarly, some DrRacket
tests sometimes fail for focus reasons.

So, whenever someone pushes, they may get failures from these tests,
then have go look at the actual errors, and try to figure out if they
actually broke something or not.

(Or, they ignore these failures, which is bad.)

Here are two potential solutions. Let's assume that I just pushed
something, and a test started failing.

- Have DrDr send me email for every push about the broken test for as
  long as it fails. If I get email more than once, it's likely that I
  actually broke something. If I only get email once, the problem went
  away on its own, and was likely an intermittent failure.

- Have the possiblity to flag some tests as intermittent (something
  like `drdr:random'), and only report failures for these tests if
  they fail twice in a row. This would reduce the amount of noise,
  since I expect most of these tests to pass most of the time. Actual
  breakage would still be detected, since it's unlikely that such
  failures would go away on their own. Detection would happen one push
  late, but that shouldn't be too much of an issue.

  Or, maybe only notify the pusher after two failures in a row, but
  notify the responsible person right away.

Any thoughts?

Vincent