[plt-scheme] Benchmarking Dracula 2.1

From: Carl Eastlund (cce at ccs.neu.edu)
Date: Thu Nov 1 18:23:55 EDT 2007

I've run several example Dracula files in versions 1.8 and 2.1 of the
language (13 examples, 5 with significant results).  These examples
all come from homework for students; either ACL2 assignments from Rex
Page's course at the University of Oklahoma, or translations from HtDP
(Scheme) assignments here at Northeastern.

For each program, I closed all other applications and other DrScheme
programs, ran it once or twice to get it swapped into memory, then
timed it three or four times until I had a consistent number (I was a
little less precise for programs that consistently ran under 5
seconds).

Here are the results:

File: old time / new time

iEx0.lisp: 3 sec / 3 sec
iEx1.lisp: 3 sec / 3 sec
iEx2.lisp: 15 sec / 5 sec
iEx2thms.lisp: 5 sec / 3 sec
iEx3b.lisp: 36 sec / 6 sec
iEx4.lisp: 3 sec / 3 sec
iEx5.lisp: 21 sec / 5 sec
tEx6.lisp: between 3 min 30 sec and 4 min 30 sec / 10 sec (!!!)
counter-world.lisp: 5 sec / 3 sec
keypress-gui.lisp: 4 sec / 3 sec
eval.lisp: 4 sec / 3 sec
matrix.lisp: 3 sec / 3 sec
worm.lisp: 40 sec / 8 sec

Most of the examples ran in around 5 seconds on both versions;
however, 5 examples (iEx2, iEx3b, iEx5, tEx6, and worm) took 15 or
more seconds in version 1.8, and all ran in 10 or less in version 2.1.
 The biggest example, tEx6.lisp, used to take several minutes, and now
takes only 10 seconds to run.  I don't have accurate memory sizes
across the board, but in one case tEx6 took up to 494 MB on version
1.8 and dropped under 180 MB on 2.1, reducing memory size by almost a
factor of 3.

Most of this speedup comes from a reduction in complexity of compiling
Dracula: some parts of the code used to suffer from quadratic
explosion and are now only linear.  Some of the reduction comes from
teachpacks being compiled ahead of time, where they used to be
compiled on demand at each run.

I hope this helps Dracula users evaluate the new version: if you've
been seeing long load times or lots of swapping out due to memory use,
try version 2.1, it should be much better.

-- 
Carl Eastlund


Posted on the users mailing list.