[racket-dev] build memory use (was: internal error during gc)

If there are some easy technical details and advice, you can write a
nice blog post about this. Anyway, in one year this data will generate
a nice video.

The difference is clear, but I wanted to see the two graphic
superimposed. The result is in the attached graphic. The colors bars
are from the 2015-01-29 version. (I could have tried to make something
more clever, but it was too much trouble.)

Are PR enabled? At line 109:
+      (define max-val (apply max (map car measurements)))
+      (define max-time (caddr (last measurements)))
If you make multiple png at the same time, all of them get the same
Peak and Duration.

Gustavo


On Wed, Jan 28, 2015 at 1:44 PM, Matthew Flatt <mflatt at cs.utah.edu> wrote:
> We've so far cut the peak memory use of a build by about 1/4 compared
> to a v6.1.1 build. Most of the problems that we fixed were related to
> finalizers.
>
> The enclosed "v611.png" and "2014-01-28.png" show memory use plotted
> against time during `raco setup -j 1` for v6.1.1 and for the current
> development version (64-bit Mac OS X). The graphs have the same
> vertical and horizontal scale. The orange area is "making", the blue
> area is "running" documentation, and the green area is "rendering"
> documentation. The big new peak in the orange area is making "plot",
> which has been ported to Typed Racket since v6.1.1 --- using brand new
> support for typed classes. The big peaks in blue and green are
> rendering the image, math, and plot documentation, all of which include
> images generated at document-build time.
>
> As you can see, there's still plenty of room for improvement in
> bringing down the peaks, but the line is consistently lower, and the
> orange and green areas don't accumulate memory. (The blue area is
> supposed to grow, because it's accumulating information about
> documents.)
>
> For more information and a tool to explore the data in a little more
> detail, see
>
>    http://build-plot.racket-lang.org/
>
> That site is updated daily with a plot of the current development
> version (so we'll be able to keep track of memory use going forward).
> Follow the "About and Tools" link below the current graph for a link to
> the "plot.rkt" tool.
>
> I've put the v6.1.1 data here:
>
>    http://build-plot.racket-lang.org/v611.txt
>
>
> At Tue, 30 Dec 2014 13:26:40 -0700, Matthew Flatt wrote:
>> I think the root of the problem is that `raco setup` gets anywhere
>> close to the 1 GB limit. Also, since the "images.scrbl" document
>> involves images, the problem may be related to using foreign libraries
>> when close to the memory limit (where the foreign library can't force a
>> Racket GC to recover from an allocation failure).
>>
>> Rendering "images.scrbl" by itself peaks somewhere between 600 MB and
>> 700 MB in a 64-bit build, so I wouldn't expect it to be anywhere close
>> to 1GB in 32-bit mode. Building within `raco setup` adds some extra
>> overhead, and there may be a leak or finalizer interaction that makes
>> `raco setup` carry along too much memory by the time it gets to
>> "images.scrbl".
>>
>> Of course, 700 MB is a lot of memory to build a 100-page manual. It
>> takes only 700 MB instead of many GB because I periodically hunt down
>> inefficiencies and leaks in an attempt to keep `raco setup` under
>> control. It's past time for another hunt.
>>
>> At Tue, 30 Dec 2014 08:00:12 -0500, Philippe Meunier wrote:
>> > Juan Francisco Cantero Hurtado wrote:
>> > >Sincerely, I don't know if the bug is in racket or in openbsd.
>> >
>> > Okay...  Well, is anyone interested in trying to locate the cause of
>> > the problem?  I can recompile the GC with debugging options turned
>> > on, etc., if more information is needed, but I don't know how the GC's
>> > code actually works and I don't have the time to dive into that...
>> >
>> > Philippe
>
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2015-01-29-vs-v611.png
Type: image/png
Size: 127722 bytes
Desc: not available
URL: <http://lists.racket-lang.org/dev/archive/attachments/20150129/f64a68f4/attachment-0001.png>