[plt-scheme] overhead of lots of dynamic closure creation in high-performance environment?

From: Neil Van Dyke (neil at neilvandyke.org)
Date: Wed Feb 11 09:09:05 EST 2009

Is there any rule-of-thumb I could use when reasoning about the 
performance implications of using lots of dynamic creation of closures 
and one-shot application of them?

I need high-performance processing of a protocol that is chunked at 
arbitrary points across packets in a lower protocol layer.  I want to 
integrate the parsing of the higher-level protocol into that of the 
lower-level protocol, rather than assembling the lower-level chunks and 
doing a second pass for the higher-level.

Doing this efficiently in a C-like style requires some headache-inducing 
record-keeping and code.

Doing this by dynamically creating a closure (for immediate application, 
or to be stuck in a hash table keyed on the stream ID), which in turn 
dynamically creates another closure, and so on until the protocol is 
consumed, looks like it will be pretty elegant.  And efficient, if 
closure creation is inexpensive.

For example:

(letrec (;; ...
          (lambda (params name val-size)
            (lambda (content content-i)
              ;; !!! do some processing and return a closure created by
              ;; make-partial-val-paramsproc
          (lambda (params name buf used)
            (lambda (content content-i)
              ;; !!!
So, for each Web page served, this part of the processing would 
typically involve, say, 30 closures created dynamically, applied once, 
and immediately left for GC.

The entire purpose of this mechanism that would use dynamic closures is 
to greatly *speed up* Web serving, so I'm looking for a better intuition 
for how efficient dynamic closures are this is before I code it all. 



Posted on the users mailing list.