[plt-scheme] Log "every change in" Definitions and Interactions...

It's not negative. It's a public discussion of science.
As far as that is concerned, you need to help your friends
find possible holes in their argument (so that their not-so-friendly
partners in science don't discover them and) so that we know
what they find is valid.

In this spirit, I have been thinking about a different tool
that's widely used in social sciences:

  a self-assessment that's correlated
  with grades on tests and projects.

I conjecture that it would be effective as a preliminary
test AND it may help you find out whether logging keystrokes
is the right tool.

-- Matthias



On Jan 26, 2009, at 8:11 AM, Shriram Krishnamurthi wrote:

> Let's not be so negative.  Fatih may not be able to "prove" anything,
> but he may be able to demonstrate interesting correlations.  For
> instance, what if people who wrote tests after the program
> consistently got lower grades, and people who wrote tests before got
> higher grades, even though the resulting handins were "similar" in
> terms of numbers of tests?  It doesn't *prove* that writing tests
> first is good, but it does tell students, "You'd be foolish to ignore
> this correlation...so we suggest writing tests first".
>
> I do agree with the specific points raised here, which is that it
> would help to scope the hypotheses carefully before going off to
> compute data, but I also think some data gathering of this form could
> be useful and some hypotheses can be usefully falsified.
>
> Shriram
>
> On Mon, Jan 26, 2009 at 7:20 AM, Matthias Felleisen
> <matthias at ccs.neu.edu> wrote:
>>
>> On Jan 26, 2009, at 6:58 AM, M. Fatih Köksal wrote:
>>
>>> Shriram Krishnamurthi wrote:
>>>>
>>>> I am going to hazard a guess that Fatih meant was actually the  
>>>> former:
>>>> for instance, check whether people are typing the recipe steps  
>>>> first
>>>> and working methodically, or only entering it at the last minute
>>>> before turning in the solution.  (It would be interesting to hear
>>>> whether those who do the former do in fact do better than those  
>>>> who do
>>>> the latter!)
>>>
>>> Exactly...
>>> We evaluated the final exams, where the students are assigned two
>>> questions, and we graded every step of Design Recipe. Using "R", we
>>> conducted a linear model analysis. R says, template and tests are  
>>> most
>>> significant, meaning that students who did the template and the  
>>> tests
>>> properly, also wrote fully functional code.
>>
>> This is no surprise. I'd go even further and say
>>
>>  template >> tests > rest
>>
>> as soon as you hit non-trivial data defs.
>>
>>
>>> Or, vice versa :) Since we don't know whether they did the code  
>>> first
>>> and then the template, test, etc. we want to record every change and
>>> replay the session of the student. This will strengthen our case.  
>>> Using
>>> this technique I can tag the students submissions according to the
>>> order of the application of design steps, then use this extra
>>> information for the analysis.
>>
>>
>> And logging won't necessarily prove it. Everyone who has absorbed
>> the DR writes the template in his head and often, though not always
>> writes the tests after the fact.
>>
>> -- Matthias
>>
>>
>>
>>