[racket-dev] DrDr Feature Request

Your wish is my command.

On Mon, Aug 8, 2011 at 10:00 AM, Robby Findler
<robby at eecs.northwestern.edu> wrote:
> PS: I'm also happy if this class of tests only emails the responsible
> person, and not the pusher.
>
> Robby
>
> On Mon, Aug 8, 2011 at 10:59 AM, Robby Findler
> <robby at eecs.northwestern.edu> wrote:
>> I like the two-times-in-a-row thought.
>>
>> FWIW, please try to avoid race conditions of the second kind.
>>
>> I think the drracket test suites are special because they fail
>> not-so-often and I don't actually know how to fix them.  If either of
>> those weren't true then I'd say they should just not run in drdr. (So
>> the race-condition/using the same file thing fails this test.)
>>
>> Robby
>>
>> On Mon, Aug 8, 2011 at 10:56 AM, Vincent St-Amour <stamourv at ccs.neu.edu> wrote:
>>>
>>> I love DrDr, but there's a small thing that annoys me about it.
>>>
>>> Some tests are prone to intermittent failures. For example, some
>>> benchmarks need to create a file, and several benchmarks share the
>>> same file, which leads to race conditions. Similarly, some DrRacket
>>> tests sometimes fail for focus reasons.
>>>
>>> So, whenever someone pushes, they may get failures from these tests,
>>> then have go look at the actual errors, and try to figure out if they
>>> actually broke something or not.
>>>
>>> (Or, they ignore these failures, which is bad.)
>>>
>>> Here are two potential solutions. Let's assume that I just pushed
>>> something, and a test started failing.
>>>
>>> - Have DrDr send me email for every push about the broken test for as
>>>  long as it fails. If I get email more than once, it's likely that I
>>>  actually broke something. If I only get email once, the problem went
>>>  away on its own, and was likely an intermittent failure.
>>>
>>> - Have the possiblity to flag some tests as intermittent (something
>>>  like `drdr:random'), and only report failures for these tests if
>>>  they fail twice in a row. This would reduce the amount of noise,
>>>  since I expect most of these tests to pass most of the time. Actual
>>>  breakage would still be detected, since it's unlikely that such
>>>  failures would go away on their own. Detection would happen one push
>>>  late, but that shouldn't be too much of an issue.
>>>
>>>  Or, maybe only notify the pusher after two failures in a row, but
>>>  notify the responsible person right away.
>>>
>>> Any thoughts?
>>>
>>> Vincent
>>> _________________________________________________
>>>  For list-related administrative tasks:
>>>  http://lists.racket-lang.org/listinfo/dev
>>>
>>
>
> _________________________________________________
>  For list-related administrative tasks:
>  http://lists.racket-lang.org/listinfo/dev



-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93