[racket-dev] DrDr Feature Request

Another request: could DrDr process the latest push first? Its a little
annoying to get emails for tests that failed when the latest push fixes
them but DrDr is so far behind. Is there any benefit to testing all the
intermediate pushes?

On 08/08/2011 09:56 AM, Vincent St-Amour wrote:
> I love DrDr, but there's a small thing that annoys me about it.
>
> Some tests are prone to intermittent failures. For example, some
> benchmarks need to create a file, and several benchmarks share the
> same file, which leads to race conditions. Similarly, some DrRacket
> tests sometimes fail for focus reasons.
>
> So, whenever someone pushes, they may get failures from these tests,
> then have go look at the actual errors, and try to figure out if they
> actually broke something or not.
>
> (Or, they ignore these failures, which is bad.)
>
> Here are two potential solutions. Let's assume that I just pushed
> something, and a test started failing.
>
> - Have DrDr send me email for every push about the broken test for as
>   long as it fails. If I get email more than once, it's likely that I
>   actually broke something. If I only get email once, the problem went
>   away on its own, and was likely an intermittent failure.
>
> - Have the possiblity to flag some tests as intermittent (something
>   like `drdr:random'), and only report failures for these tests if
>   they fail twice in a row. This would reduce the amount of noise,
>   since I expect most of these tests to pass most of the time. Actual
>   breakage would still be detected, since it's unlikely that such
>   failures would go away on their own. Detection would happen one push
>   late, but that shouldn't be too much of an issue.
>
>   Or, maybe only notify the pusher after two failures in a row, but
>   notify the responsible person right away.
>
> Any thoughts?
>
> Vincent
> _________________________________________________
>   For list-related administrative tasks:
>   http://lists.racket-lang.org/listinfo/dev