[plt-scheme] V301.5 Speed Up

I've just be running a few tests.  It seems that time-apply sometimes
returns a negative number for the CPU time.  That does cause havoc with the
variance (and the mean).  

Doug


-----Original Message-----
From: plt-scheme-bounces at list.cs.brown.edu on behalf of Noel Welsh
Sent: Fri 2/10/2006 9:17 AM
To: Gregory Woodhouse
Cc: PLT Scheme
Subject: Re: [plt-scheme] V301.5 Speed Up
 
--- Gregory Woodhouse <gregory.woodhouse at sbcglobal.net>
wrote:

> It would be nice to be able run a test 1000 times,  
> saving the data for statistical analysis.

I've just written code to do this (run code 50 times,
perform test for significance).  It requires a hacked
version of the science collection so it won't work till the
next version of the science collection is out.  If anyone
wants it, email me off list.

Anyway, some observations:

  - GC time is really long compared to run time (for the
silly little benchmarks I tried)

  - unexpectedly, the variance of my measurements was
crazy!  When I made benchmarks (just loops adding up
numbers) long enough to measure the time reliable I got
results like this:

The code:

           (let* ((test1 (lambda ()
                     (for ((i 0 10000) (sum 0))
                           (+ 1000 sum))))
             (test2 (lambda ()
                      (for ((i 0 10000000) (sum 0))
                           (+ 1 sum))))
             (s1 (measure test1))
             (s2 (measure test2)))
        (let-values (((faster? p) (faster s1 s2)))
          (printf "p ~a\n" p)
          (printf "s1 mean: ~a var: ~a\n" (mean s1)
(variance s1))
          (printf "s2 mean: ~a var: ~a\n" (mean s2)
(variance s2))
          (assert-true faster?)))

The output:

p 1.0
s1 mean: 6.799999999999996 var: 22.204081632653068
s2 mean: 13492.2 var: 18376.693877551028

P is the value returned by the t-test (the probability the
means differ by chance).  Incidentally the assumptions for
the t-test are almost certainly violated in this case.

Anyway, I really can't explain the variance being that
large.  Here's how I collect the data:

  ;; measure : ((any ...) -> any  any ...) -> (vector-of
number) 
  (define (measure proc . args)
    (define (prepare)
      (for! (i 0 3)
            (collect-garbage)))
    (list->vector
     (for ((i 0 50) (times null))
          (prepare)
          (let-values (((results cpu-time real-time
gc-time)
                        (time-apply proc args)))
            (cons cpu-time times)))))

Hope that's of interest to someone!

N.

Email: noelwelsh <at> yahoo <dot> com   noel <at> untyped <dot> com
AIM: noelhwelsh
Blogs: http://monospaced.blogspot.com/  http://www.untyped.com/untyping/

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
_________________________________________________
  For list-related administrative tasks:
  http://list.cs.brown.edu/mailman/listinfo/plt-scheme