[racket-dev] DrDr Feature Request

It is useful to test all of them to find out when errors start. It
doesn't do the newest first, because then the calculation of "new
issue" wouldn't make any sense, because you wouldn't have the previous
push's tests.

Jay

On Mon, Aug 8, 2011 at 11:34 AM, Jon Rafkind <rafkind at cs.utah.edu> wrote:
> Another request: could DrDr process the latest push first? Its a little
> annoying to get emails for tests that failed when the latest push fixes
> them but DrDr is so far behind. Is there any benefit to testing all the
> intermediate pushes?
>
> On 08/08/2011 09:56 AM, Vincent St-Amour wrote:
>> I love DrDr, but there's a small thing that annoys me about it.
>>
>> Some tests are prone to intermittent failures. For example, some
>> benchmarks need to create a file, and several benchmarks share the
>> same file, which leads to race conditions. Similarly, some DrRacket
>> tests sometimes fail for focus reasons.
>>
>> So, whenever someone pushes, they may get failures from these tests,
>> then have go look at the actual errors, and try to figure out if they
>> actually broke something or not.
>>
>> (Or, they ignore these failures, which is bad.)
>>
>> Here are two potential solutions. Let's assume that I just pushed
>> something, and a test started failing.
>>
>> - Have DrDr send me email for every push about the broken test for as
>>   long as it fails. If I get email more than once, it's likely that I
>>   actually broke something. If I only get email once, the problem went
>>   away on its own, and was likely an intermittent failure.
>>
>> - Have the possiblity to flag some tests as intermittent (something
>>   like `drdr:random'), and only report failures for these tests if
>>   they fail twice in a row. This would reduce the amount of noise,
>>   since I expect most of these tests to pass most of the time. Actual
>>   breakage would still be detected, since it's unlikely that such
>>   failures would go away on their own. Detection would happen one push
>>   late, but that shouldn't be too much of an issue.
>>
>>   Or, maybe only notify the pusher after two failures in a row, but
>>   notify the responsible person right away.
>>
>> Any thoughts?
>>
>> Vincent
>> _________________________________________________
>>   For list-related administrative tasks:
>>   http://lists.racket-lang.org/listinfo/dev
>
> _________________________________________________
>  For list-related administrative tasks:
>  http://lists.racket-lang.org/listinfo/dev
>



-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93