[racket-dev] build memory use (was: internal error during gc)

We've so far cut the peak memory use of a build by about 1/4 compared
to a v6.1.1 build. Most of the problems that we fixed were related to
finalizers.

The enclosed "v611.png" and "2014-01-28.png" show memory use plotted
against time during `raco setup -j 1` for v6.1.1 and for the current
development version (64-bit Mac OS X). The graphs have the same
vertical and horizontal scale. The orange area is "making", the blue
area is "running" documentation, and the green area is "rendering"
documentation. The big new peak in the orange area is making "plot",
which has been ported to Typed Racket since v6.1.1 --- using brand new
support for typed classes. The big peaks in blue and green are
rendering the image, math, and plot documentation, all of which include
images generated at document-build time.

As you can see, there's still plenty of room for improvement in
bringing down the peaks, but the line is consistently lower, and the
orange and green areas don't accumulate memory. (The blue area is
supposed to grow, because it's accumulating information about
documents.)

For more information and a tool to explore the data in a little more
detail, see

   http://build-plot.racket-lang.org/

That site is updated daily with a plot of the current development
version (so we'll be able to keep track of memory use going forward).
Follow the "About and Tools" link below the current graph for a link to
the "plot.rkt" tool.

I've put the v6.1.1 data here:

   http://build-plot.racket-lang.org/v611.txt


At Tue, 30 Dec 2014 13:26:40 -0700, Matthew Flatt wrote:
> I think the root of the problem is that `raco setup` gets anywhere
> close to the 1 GB limit. Also, since the "images.scrbl" document
> involves images, the problem may be related to using foreign libraries
> when close to the memory limit (where the foreign library can't force a
> Racket GC to recover from an allocation failure).
> 
> Rendering "images.scrbl" by itself peaks somewhere between 600 MB and
> 700 MB in a 64-bit build, so I wouldn't expect it to be anywhere close
> to 1GB in 32-bit mode. Building within `raco setup` adds some extra
> overhead, and there may be a leak or finalizer interaction that makes
> `raco setup` carry along too much memory by the time it gets to
> "images.scrbl".
> 
> Of course, 700 MB is a lot of memory to build a 100-page manual. It
> takes only 700 MB instead of many GB because I periodically hunt down
> inefficiencies and leaks in an attempt to keep `raco setup` under
> control. It's past time for another hunt.
> 
> At Tue, 30 Dec 2014 08:00:12 -0500, Philippe Meunier wrote:
> > Juan Francisco Cantero Hurtado wrote:
> > >Sincerely, I don't know if the bug is in racket or in openbsd.
> > 
> > Okay...  Well, is anyone interested in trying to locate the cause of
> > the problem?  I can recompile the GC with debugging options turned
> > on, etc., if more information is needed, but I don't know how the GC's
> > code actually works and I don't have the time to dive into that...
> > 
> > Philippe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: v611.png
Type: image/png
Size: 87824 bytes
Desc: not available
URL: <http://lists.racket-lang.org/dev/archive/attachments/20150128/131174a2/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2015-01-28.png
Type: image/png
Size: 85830 bytes
Desc: not available
URL: <http://lists.racket-lang.org/dev/archive/attachments/20150128/131174a2/attachment-0003.png>