[plt-scheme] Collecting test data, making science

I've been kicking around the idea of collecting data from
test runs for some time.  I've finally implemented and
released code to do this -- si.plt, coming to a PLaneT near
you.

The idea is simple:

  - the software lets you annotate tests that are for
"bugs": errors found after a release.  i.e. a failure of
your development method

  - every time you run tests, the software records the
number of successes, failures, errors, and bugs

I think analysing this data might show something
interesting.    I'd very much appreciate it if people who
regularly use SchemeUnit would consider using SI, and
making a brief note of their development method.  Changing
method is cool too -- that allows before and after
comparison.  Just note the date you changed.  The more data
you record, the more interesting the analysis can become.

Note that SI doesn't send any data to me -- it's all stored
on your disk.  However, in a few months I intend to ask for
people to send me their data.

Thanks,
Noel

PS:  Comments on SI are welcome.  It is very preliminary at
this stage, though I've been running it locally for a few
months.  Here is an excerpt from the docs, that gives a bit
more info:

Simple Improvement (SI) is a SchemeUnit add-on that makes
the you, the programmer, aware of the effectiveness of your
development process.

There are two main goals for a software development
process:

  - Develop software as quickly as possible

  - Develop software as correctly as possible

Simple Improvement gathers statistics on these two aspects
of software development.  This allows you to see how
effective your process is, and, better yet, tweak your
process to optimise it.

To measure how quickly you are developing software SI
records the number of test cases (and the number of
succcesses, failures, and error) each time you run your
tests.  Assuming you are diligent with your testsing, the
number of tests should be strongly correlated with the
number of features in your software.

To measure how correct your software is SI extends the
SchemeUnit language with a new type of test case called a
'bug-case'.  A bug-case is exactly the same as a test-case,
except it indicates a test for a problem found after a
software release -- in other words, a released bug.  

By observing these two statistics over time you can gain an
idea of the effectiveness of your development process.

To use SI you only need make two changes to your test
suites:

  - replace uses of test/text-ui with test/si-text-ui

  - use bug-case instead of test-case when writing a test
    for a bug (a problem found in released code)



Email: noelwelsh <at> yahoo <dot> com   noel <at> untyped <dot> com
AIM: noelhwelsh
Blogs: http://monospaced.blogspot.com/  http://www.untyped.com/untyping/



 
____________________________________________________________________________________
Sponsored Link

Degrees online in as fast as 1 Yr - MBA, Bachelor's, Master's, Associate
Click now to apply http://yahoo.degrees.info