[plt-scheme] Collecting test data, making science

From: Noel Welsh (noelwelsh at yahoo.com)
Date: Tue Nov 7 07:24:17 EST 2006

I've been kicking around the idea of collecting data from
test runs for some time.  I've finally implemented and
released code to do this -- si.plt, coming to a PLaneT near

The idea is simple:

  - the software lets you annotate tests that are for
"bugs": errors found after a release.  i.e. a failure of
your development method

  - every time you run tests, the software records the
number of successes, failures, errors, and bugs

I think analysing this data might show something
interesting.    I'd very much appreciate it if people who
regularly use SchemeUnit would consider using SI, and
making a brief note of their development method.  Changing
method is cool too -- that allows before and after
comparison.  Just note the date you changed.  The more data
you record, the more interesting the analysis can become.

Note that SI doesn't send any data to me -- it's all stored
on your disk.  However, in a few months I intend to ask for
people to send me their data.


PS:  Comments on SI are welcome.  It is very preliminary at
this stage, though I've been running it locally for a few
months.  Here is an excerpt from the docs, that gives a bit
more info:

Simple Improvement (SI) is a SchemeUnit add-on that makes
the you, the programmer, aware of the effectiveness of your
development process.

There are two main goals for a software development

  - Develop software as quickly as possible

  - Develop software as correctly as possible

Simple Improvement gathers statistics on these two aspects
of software development.  This allows you to see how
effective your process is, and, better yet, tweak your
process to optimise it.

To measure how quickly you are developing software SI
records the number of test cases (and the number of
succcesses, failures, and error) each time you run your
tests.  Assuming you are diligent with your testsing, the
number of tests should be strongly correlated with the
number of features in your software.

To measure how correct your software is SI extends the
SchemeUnit language with a new type of test case called a
'bug-case'.  A bug-case is exactly the same as a test-case,
except it indicates a test for a problem found after a
software release -- in other words, a released bug.  

By observing these two statistics over time you can gain an
idea of the effectiveness of your development process.

To use SI you only need make two changes to your test

  - replace uses of test/text-ui with test/si-text-ui

  - use bug-case instead of test-case when writing a test
    for a bug (a problem found in released code)

Email: noelwelsh <at> yahoo <dot> com   noel <at> untyped <dot> com
AIM: noelhwelsh
Blogs: http://monospaced.blogspot.com/  http://www.untyped.com/untyping/

Sponsored Link

Degrees online in as fast as 1 Yr - MBA, Bachelor's, Master's, Associate
Click now to apply http://yahoo.degrees.info

Posted on the users mailing list.