[racket] Getting unusual behavior from racket/sandbox: not truly capping memory usage?!

I added some stuff to visualize the dump-memory-stats output. I'm
finding that...

channel-syncer
evt
evt-set
hash
hash-table-bucket
installed-finalizers
semaphore
thread
thread-cell
unknown,154
unknown,225
unknown,226
unknown,231
unknown,234
unknown,253
weak-box

all appear to grow without bound. I made a change to the dispatch
server to not construct new evts on every connection, but it didn't
make any difference.

The code that I'm running doesn't use hashes at all, and it uses the
same evt, so this is suspicious to me.

My guess, based on nothing really, is that inside of the
implementation of sync, there's a call to 'select' or 'poll' that uses
a hash table to map the OS fd back to the Racket evt. This thing is
leaking and causing leaks in the hash table, the evts inside of it,
and maybe the temporary thread that holds the call to select.

I push my dump analysis code to my fork of Danny's repo. Just name one
of the "dump-*" files to "dump" and run "dump.rkt". You'll get a ton
of PNGs in the "dump-pngs" directory.

Jay



On Mon, Apr 22, 2013 at 10:38 AM, Jay McCarthy <jay.mccarthy at gmail.com> wrote:
> I'm looking into this...
>
> I think the connection thing is a bit of red herring. If you allow
> HTTP/1.1 then your test uses 6 connections total until the crash, the
> first 5 to just load the static content. However, if you remove it,
> then every repl test is its own connection. This has the effect of
> each repl test getting its own thread and custodian. So, it's not that
> connections are leaking, but that by re-using the same connection,
> some other thing that is leaking lasts a long time. [BTW, you don't
> need to hack the code to do this, just pass #:connection-close? #t to
> serve/servlet.]
>
> I gathered per-connection memory stats and found that it takes about
> 160 reqs to GC. It dies after 24 such cycles. Most allocate 30mbs.
> Most leak 3 mbs.
>
> I cut out the entire continuation apparatus of the server and had two
> interesting results. First, the memory still grows unboundedly and
> second the sandbox doesn't catch it. This suggests to me that the
> leaked memory is not accounted to the sandbox custodian.
>
> The part of the server that is left uses no global data structures, is
> only using six threads at most, and does not use continuations or
> non-tail recursive loops. It should run in constant memory.
>
> So, now it's my turn to cry wolf and say that something strange is
> happening in the VM.
>
> My results, including logs and modification to the test server are here:
>
> https://github.com/jeapostrophe/web-server-under-sandbox
>
> to get the graphs and analysis, just link one of the logs to "log" and
> run "log.rkt"
>
> Jay
>
> On Sat, Apr 20, 2013 at 5:57 PM, Danny Yoo <dyoo at hashcollision.org> wrote:
>>> Let me post code and see if anyone agrees:
>>>
>>>      https://github.com/dyoo/web-server-under-sandbox
>>>
>>> I create a trivial web server with a servlet that immediately errors
>>> out.  I have it also serve static JavaScript that's essentially a
>>> torture test: it keeps sending requests to the web server, over and
>>> over.  (Ripped out of whalesong's REPL testing code).
>>
>>
>> Followup: I've gotten independent confirmation from Eric Hanchrow that
>> he can replicate my results, so at least I'm not hallucinating.
>>
>> I've been able to trace down the source of the garbage somewhere due
>> to the reuse of the connection to support HTTP/1.1.  If I hack this
>> line in the codebase:
>>
>>     https://github.com/plt/racket/blob/master/collects/web-server/private/dispatch-server-unit.rkt#L106
>>
>>
>> so that it _always_ closes the connection, then the resulting web
>> server survives my torture test.
>>
>>
>> So there's some kind of state that's accumulating due to the
>> connection reuse to support HTTP/1.1.  Still trying to trace the exact
>> nature of the problem down.
>
>
>
> --
> Jay McCarthy <jay at cs.byu.edu>
> Assistant Professor / Brigham Young University
> http://faculty.cs.byu.edu/~jay
>
> "The glory of God is Intelligence" - D&C 93



-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://faculty.cs.byu.edu/~jay

"The glory of God is Intelligence" - D&C 93