[racket-dev] DrDr Feature Request

I noticed this functionality just now.. thanks a lot!

On 08/08/2011 12:38 PM, Jay McCarthy wrote:
> Your wish is my command.
>
> On Mon, Aug 8, 2011 at 10:00 AM, Robby Findler
> <robby at eecs.northwestern.edu> wrote:
>> PS: I'm also happy if this class of tests only emails the responsible
>> person, and not the pusher.
>>
>> Robby
>>
>> On Mon, Aug 8, 2011 at 10:59 AM, Robby Findler
>> <robby at eecs.northwestern.edu> wrote:
>>> I like the two-times-in-a-row thought.
>>>
>>> FWIW, please try to avoid race conditions of the second kind.
>>>
>>> I think the drracket test suites are special because they fail
>>> not-so-often and I don't actually know how to fix them.  If either of
>>> those weren't true then I'd say they should just not run in drdr. (So
>>> the race-condition/using the same file thing fails this test.)
>>>
>>> Robby
>>>
>>> On Mon, Aug 8, 2011 at 10:56 AM, Vincent St-Amour <stamourv at ccs.neu.edu> wrote:
>>>> I love DrDr, but there's a small thing that annoys me about it.
>>>>
>>>> Some tests are prone to intermittent failures. For example, some
>>>> benchmarks need to create a file, and several benchmarks share the
>>>> same file, which leads to race conditions. Similarly, some DrRacket
>>>> tests sometimes fail for focus reasons.
>>>>
>>>> So, whenever someone pushes, they may get failures from these tests,
>>>> then have go look at the actual errors, and try to figure out if they
>>>> actually broke something or not.
>>>>
>>>> (Or, they ignore these failures, which is bad.)
>>>>
>>>> Here are two potential solutions. Let's assume that I just pushed
>>>> something, and a test started failing.
>>>>
>>>> - Have DrDr send me email for every push about the broken test for as
>>>>  long as it fails. If I get email more than once, it's likely that I
>>>>  actually broke something. If I only get email once, the problem went
>>>>  away on its own, and was likely an intermittent failure.
>>>>
>>>> - Have the possiblity to flag some tests as intermittent (something
>>>>  like `drdr:random'), and only report failures for these tests if
>>>>  they fail twice in a row. This would reduce the amount of noise,
>>>>  since I expect most of these tests to pass most of the time. Actual
>>>>  breakage would still be detected, since it's unlikely that such
>>>>  failures would go away on their own. Detection would happen one push
>>>>  late, but that shouldn't be too much of an issue.
>>>>
>>>>  Or, maybe only notify the pusher after two failures in a row, but
>>>>  notify the responsible person right away.
>>>>
>>>> Any thoughts?
>>>>
>>>> Vincent
>>>> _________________________________________________
>>>>  For list-related administrative tasks:
>>>>  http://lists.racket-lang.org/listinfo/dev
>>>>
>> _________________________________________________
>>  For list-related administrative tasks:
>>  http://lists.racket-lang.org/listinfo/dev
>
>