[racket-dev] DrDr Feature Request

This is a rare event (playing catchup like this) so I think it is
probably best if we just let it catch up. Should be just a couple of
more days (maybe a week) by my sketchy guesstimationizing.

Robby

On Mon, Aug 8, 2011 at 12:34 PM, Jon Rafkind <rafkind at cs.utah.edu> wrote:
> Another request: could DrDr process the latest push first? Its a little
> annoying to get emails for tests that failed when the latest push fixes
> them but DrDr is so far behind. Is there any benefit to testing all the
> intermediate pushes?
>
> On 08/08/2011 09:56 AM, Vincent St-Amour wrote:
>> I love DrDr, but there's a small thing that annoys me about it.
>>
>> Some tests are prone to intermittent failures. For example, some
>> benchmarks need to create a file, and several benchmarks share the
>> same file, which leads to race conditions. Similarly, some DrRacket
>> tests sometimes fail for focus reasons.
>>
>> So, whenever someone pushes, they may get failures from these tests,
>> then have go look at the actual errors, and try to figure out if they
>> actually broke something or not.
>>
>> (Or, they ignore these failures, which is bad.)
>>
>> Here are two potential solutions. Let's assume that I just pushed
>> something, and a test started failing.
>>
>> - Have DrDr send me email for every push about the broken test for as
>>   long as it fails. If I get email more than once, it's likely that I
>>   actually broke something. If I only get email once, the problem went
>>   away on its own, and was likely an intermittent failure.
>>
>> - Have the possiblity to flag some tests as intermittent (something
>>   like `drdr:random'), and only report failures for these tests if
>>   they fail twice in a row. This would reduce the amount of noise,
>>   since I expect most of these tests to pass most of the time. Actual
>>   breakage would still be detected, since it's unlikely that such
>>   failures would go away on their own. Detection would happen one push
>>   late, but that shouldn't be too much of an issue.
>>
>>   Or, maybe only notify the pusher after two failures in a row, but
>>   notify the responsible person right away.
>>
>> Any thoughts?
>>
>> Vincent
>> _________________________________________________
>>   For list-related administrative tasks:
>>   http://lists.racket-lang.org/listinfo/dev
>
> _________________________________________________
>  For list-related administrative tasks:
>  http://lists.racket-lang.org/listinfo/dev
>