((( [logo]Racket )))Need Help?
AboutDownloadDocumentationPLaneTCommunityLearning

[plt-scheme] V301.5 Speed Up

From: Williams, M. Douglas (M.DOUGLAS.WILLIAMS at saic.com)
Date: Fri Feb 10 23:38:50 EST 2006		 
I seem to have made a mistake in cutting and pasting the results.  So here
is another run set of runs, with me being more careful.

open-loop.ss
V301
run 0: cpu = 66796 real = 67176 gc = 47558
run 1: cpu = 69060 real = 69400 gc = 49814
run 2: cpu = -357483 real = 72535 gc = -376862
run 3: cpu = 71313 real = 71733 gc = 52016
run 4: cpu = 67597 real = 68128 gc = 48492
run 5: cpu = 67447 real = 67727 gc = 48169
run 6: cpu = 67086 real = 67367 gc = 47933
run 7: cpu = 66987 real = 67647 gc = 47779
run 8: cpu = -362611 real = 67217 gc = -381908
run 9: cpu = 66425 real = 67036 gc = 47159
CPU:   mean = -17738.3, variance = 32552157259.344448
Real:  mean = 68596.6, variance = 3971085.6
GC:    mean = -36985.00000000001, variance = 32569416745.555557

V301.5
run 0: cpu = 15362 real = 15592 gc = 3453
run 1: cpu = 15362 real = 15422 gc = 3476
run 2: cpu = 15132 real = 15191 gc = 3245
run 3: cpu = 14832 real = 14991 gc = 2905
run 4: cpu = 15042 real = 15082 gc = 3025
run 5: cpu = 14882 real = 14941 gc = 3025
run 6: cpu = 14551 real = 14611 gc = 2704
run 7: cpu = 14632 real = 14782 gc = 2744
run 8: cpu = 14671 real = 14701 gc = 2764
run 9: cpu = 14220 real = 14250 gc = 2343
CPU:   mean = 14868.599999999999, variance = 133825.6
Real:  mean = 14956.3, variance = 156329.34444444443
GC:    mean = 2968.4, variance = 125844.04444444446

----------
model-2a.ss
V301
run 0: cpu = 15502 real = 15653 gc = 1442
run 1: cpu = 15422 real = 15452 gc = 1242
run 2: cpu = 16083 real = 16124 gc = 1262
run 3: cpu = 14231 real = 14300 gc = 861
run 4: cpu = 15042 real = 15202 gc = 1082
run 5: cpu = 14741 real = 14801 gc = 842
run 6: cpu = 14100 real = 14181 gc = 1052
run 7: cpu = 13950 real = 13960 gc = 841
run 8: cpu = 14380 real = 14621 gc = 862
run 9: cpu = 14331 real = 14461 gc = 843
CPU:   mean = 14778.2, variance = 501743.5111111111
Real:  mean = 14875.5, variance = 499557.1666666667
GC:    mean = 1032.9, variance = 48270.54444444445

V301.5
run 0: cpu = 7781 real = 7832 gc = 992
run 1: cpu = 7982 real = 8022 gc = 1044
run 2: cpu = 7892 real = 7951 gc = 790
run 3: cpu = 7591 real = 7641 gc = 1041
run 4: cpu = 7681 real = 7831 gc = 791
run 5: cpu = 7220 real = 7260 gc = 560
run 6: cpu = 7341 real = 7381 gc = 811
run 7: cpu = 6940 real = 6970 gc = 551
run 8: cpu = 7130 real = 7320 gc = 551
run 9: cpu = 7391 real = 7420 gc = 560
CPU:   mean = 7494.9, variance = 119088.10000000002
Real:  mean = 7562.8, variance = 118619.28888888888
GC:    mean = 769.1, variance = 42715.211111111115

Doug


-----Original Message-----
From: plt-scheme-bounces at list.cs.brown.edu on behalf of Williams, M. Douglas
Sent: Fri 2/10/2006 3:10 PM
To: PLT Scheme
Subject: RE: [plt-scheme] V301.5 Speed Up
 
I timed (or attempted to) the simulation collection for two different
models.  These are based on just 10 runs each.  Please take these with the
caveats mentioned by several of the previous responders.  However, I would
certainly bank on simulations developed using the simulation collection
running in V301.5 as running significantly faster in V301.5 than in V301.

The first (model-2a.ss) is a combined discrete and continuous simulation and
is both computationally intensive (the continuous part) and continuation
intensive (the discrete part).  There is about a 2x speedup - real times
from ~15000 (V301) to ~7500 (V301.5).

The second model (open-loop.ss) is a pure discrete event simulation.  This
showed even more improvement.  There is about a 4.5x speedup - real times
from ~73000 (V301) to ~15500 (V301.5).  The garbage collection seems to be
the killer in this one for V301.  [You can also see the problem with the
negative cpu times here.  I tried several times and never got 10 runs that
were all 'good'.  The real time and gc time seem to be okay.]

Does anyone know if the time-apply bug always produces negative numbers when
it breaks, or are those just the obvious ones?

Example code from open-loop.ss

(run-simulation 100 1000)
(collect-garbage)

(require (planet "statistics.ss" ("williams" "science.plt")))
(let* ((n 10)
       (cpu-times (make-vector n))
       (real-times (make-vector n))
       (gc-times (make-vector n)))
  (do ((i 0 (+ i 1)))
      ((= i n)(void))
    (collect-garbage)
    (let-values (((result cpu real gc)
                  (time-apply run-simulation '(100 1000))))
      (vector-set! cpu-times i cpu)
      (vector-set! real-times i real)
      (vector-set! gc-times i gc)))
  (do ((i 0 (+ i 1)))
      ((= i n) (void))
    (printf "run ~a: cpu = ~a real = ~a gc = ~a~n"
            i (vector-ref cpu-times i)
            (vector-ref real-times i)
            (vector-ref gc-times i)))
  (printf "CPU:   mean = ~a, variance = ~a~n"
          (mean cpu-times) (variance cpu-times))
  (printf "Real:  mean = ~a, variance = ~a~n"
          (mean real-times) (variance real-times))
  (printf "GC:    mean = ~a, variance = ~a~n"
          (mean gc-times) (variance gc-times)))
  
Data

model-2a.ss
V301
run 0: cpu = 7971 real = 8013 gc = 881
run 1: cpu = 7100 real = 7162 gc = 620
run 2: cpu = 7141 real = 7152 gc = 612
run 3: cpu = 8092 real = 8113 gc = 881
run 4: cpu = 7591 real = 7723 gc = 620
run 5: cpu = 7541 real = 7562 gc = 621
run 6: cpu = 7962 real = 8013 gc = 903
run 7: cpu = 6830 real = 6841 gc = 311
run 8: cpu = 7420 real = 7432 gc = 621
run 9: cpu = 7180 real = 7251 gc = 320
CPU:   mean = 7482.8, variance = 182148.17777777778
Real:  mean = 7526.2, variance = 187229.95555555553
GC:    mean = 638.9999999999999, variance = 44252.0

V301.5
run 0: cpu = 17696 real = 17829 gc = 4016
run 1: cpu = 15051 real = 15064 gc = 2185
run 2: cpu = 14661 real = 14843 gc = 1794
run 3: cpu = 15583 real = 15585 gc = 1412
run 4: cpu = 14821 real = 14944 gc = 1032
run 5: cpu = 15793 real = 15815 gc = 1452
run 6: cpu = 15793 real = 15805 gc = 1062
run 7: cpu = 14381 real = 14393 gc = 852
run 8: cpu = 14792 real = 14923 gc = 852
run 9: cpu = 14841 real = 14853 gc = 852
CPU:   mean = 15341.199999999999, variance = 922566.4
Real:  mean = 15405.4, variance = 938010.2666666667
GC:    mean = 1550.9, variance = 948959.6555555558
----------
open-loop.ss
V301
run 0: cpu = -355100 real = 74959 gc = 54767
run 1: cpu = 74146 real = 74579 gc = 54699
run 2: cpu = 73926 real = 74489 gc = 54420
run 3: cpu = 73666 real = 73876 gc = 54037
run 4: cpu = 73236 real = 73502 gc = 53714
run 5: cpu = 72905 real = 73023 gc = 53377
run 6: cpu = -356852 real = 73172 gc = 53214
run 7: cpu = 72464 real = 72733 gc = 52865
run 8: cpu = 72053 real = 72363 gc = 52465
run 9: cpu = 71853 real = 72233 gc = 52205
CPU:   mean = -12770.300000000001, variance = 32720231896.233334
Real:  mean = 73492.9, variance = 914819.8777777779
GC:    mean = 53576.299999999996, variance = 823737.5666666668

Doug

> -----Original Message-----
> From: plt-scheme-bounces at list.cs.brown.edu [mailto:plt-scheme-
> bounces at list.cs.brown.edu] On Behalf Of Williams, M. Douglas
> Sent: Friday, February 10, 2006 11:12 AM
> To: Noel Welsh; Gregory Woodhouse
> Cc: PLT Scheme
> Subject: RE: [plt-scheme] V301.5 Speed Up
> 
> I've just be running a few tests.  It seems that time-apply sometimes
> returns a negative number for the CPU time.  That does cause havoc with
> the
> variance (and the mean).
> 
> Doug
> 
> 
> -----Original Message-----
> From: plt-scheme-bounces at list.cs.brown.edu on behalf of Noel Welsh
> Sent: Fri 2/10/2006 9:17 AM
> To: Gregory Woodhouse
> Cc: PLT Scheme
> Subject: Re: [plt-scheme] V301.5 Speed Up
> 
> --- Gregory Woodhouse <gregory.woodhouse at sbcglobal.net>
> wrote:
> 
> > It would be nice to be able run a test 1000 times,
> > saving the data for statistical analysis.
> 
> I've just written code to do this (run code 50 times,
> perform test for significance).  It requires a hacked
> version of the science collection so it won't work till the
> next version of the science collection is out.  If anyone
> wants it, email me off list.
> 
> Anyway, some observations:
> 
>   - GC time is really long compared to run time (for the
> silly little benchmarks I tried)
> 
>   - unexpectedly, the variance of my measurements was
> crazy!  When I made benchmarks (just loops adding up
> numbers) long enough to measure the time reliable I got
> results like this:
> 
> The code:
> 
>            (let* ((test1 (lambda ()
>                      (for ((i 0 10000) (sum 0))
>                            (+ 1000 sum))))
>              (test2 (lambda ()
>                       (for ((i 0 10000000) (sum 0))
>                            (+ 1 sum))))
>              (s1 (measure test1))
>              (s2 (measure test2)))
>         (let-values (((faster? p) (faster s1 s2)))
>           (printf "p ~a\n" p)
>           (printf "s1 mean: ~a var: ~a\n" (mean s1)
> (variance s1))
>           (printf "s2 mean: ~a var: ~a\n" (mean s2)
> (variance s2))
>           (assert-true faster?)))
> 
> The output:
> 
> p 1.0
> s1 mean: 6.799999999999996 var: 22.204081632653068
> s2 mean: 13492.2 var: 18376.693877551028
> 
> P is the value returned by the t-test (the probability the
> means differ by chance).  Incidentally the assumptions for
> the t-test are almost certainly violated in this case.
> 
> Anyway, I really can't explain the variance being that
> large.  Here's how I collect the data:
> 
>   ;; measure : ((any ...) -> any  any ...) -> (vector-of
> number)
>   (define (measure proc . args)
>     (define (prepare)
>       (for! (i 0 3)
>             (collect-garbage)))
>     (list->vector
>      (for ((i 0 50) (times null))
>           (prepare)
>           (let-values (((results cpu-time real-time
> gc-time)
>                         (time-apply proc args)))
>             (cons cpu-time times)))))
> 
> Hope that's of interest to someone!
> 
> N.
> 
> Email: noelwelsh <at> yahoo <dot> com   noel <at> untyped <dot> com
> AIM: noelhwelsh
> Blogs: http://monospaced.blogspot.com/  http://www.untyped.com/untyping/
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> _________________________________________________
>   For list-related administrative tasks:
>   http://list.cs.brown.edu/mailman/listinfo/plt-scheme
> 
> _________________________________________________
>   For list-related administrative tasks:
>   http://list.cs.brown.edu/mailman/listinfo/plt-scheme
_________________________________________________
  For list-related administrative tasks:
  http://list.cs.brown.edu/mailman/listinfo/plt-scheme
Posted on the users mailing list.