[plt-scheme] Log "every change in" Definitions and Interactions...

Let's not be so negative.  Fatih may not be able to "prove" anything,
but he may be able to demonstrate interesting correlations.  For
instance, what if people who wrote tests after the program
consistently got lower grades, and people who wrote tests before got
higher grades, even though the resulting handins were "similar" in
terms of numbers of tests?  It doesn't *prove* that writing tests
first is good, but it does tell students, "You'd be foolish to ignore
this correlation...so we suggest writing tests first".

I do agree with the specific points raised here, which is that it
would help to scope the hypotheses carefully before going off to
compute data, but I also think some data gathering of this form could
be useful and some hypotheses can be usefully falsified.

Shriram

On Mon, Jan 26, 2009 at 7:20 AM, Matthias Felleisen
<matthias at ccs.neu.edu> wrote:
>
> On Jan 26, 2009, at 6:58 AM, M. Fatih Köksal wrote:
>
>> Shriram Krishnamurthi wrote:
>>>
>>> I am going to hazard a guess that Fatih meant was actually the former:
>>> for instance, check whether people are typing the recipe steps first
>>> and working methodically, or only entering it at the last minute
>>> before turning in the solution.  (It would be interesting to hear
>>> whether those who do the former do in fact do better than those who do
>>> the latter!)
>>
>> Exactly...
>> We evaluated the final exams, where the students are assigned two
>> questions, and we graded every step of Design Recipe. Using "R", we
>> conducted a linear model analysis. R says, template and tests are most
>> significant, meaning that students who did the template and the tests
>> properly, also wrote fully functional code.
>
> This is no surprise. I'd go even further and say
>
>  template >> tests > rest
>
> as soon as you hit non-trivial data defs.
>
>
>> Or, vice versa :) Since we don't know whether they did the code first
>> and then the template, test, etc. we want to record every change and
>> replay the session of the student. This will strengthen our case. Using
>> this technique I can tag the students submissions according to the
>> order of the application of design steps, then use this extra
>> information for the analysis.
>
>
> And logging won't necessarily prove it. Everyone who has absorbed
> the DR writes the template in his head and often, though not always
> writes the tests after the fact.
>
> -- Matthias
>
>
>
>