[racket] bitmap% size in memory

At Wed, 19 Dec 2012 12:42:41 -0700, Michael Wilber wrote:
> Just to clarify my understanding: The shadow bytestring takes up memory
> itself though, which means that if a 1MB bitmap is loaded with a 1MB
> shadow, then to the OS, Racket's using 2MB of memory but the garbage
> collector thinks it's only using 1MB. Is that right? Or since it's never
> accessed, does that mean the OS never keeps it resident or something?

That was the idea, but maybe that made more sense years ago when
virtual memory seemed practically infinite.

After experimenting and thinking about this, DrRacket may be failing
because two factors of 2 are multiplied: a factor of 2 for the actual
bitmap plus the shadow, and a factor of 2 (or so) related to triggering
a GC (which also calculates memory use). For a 32-bit Windows run, you
only get 2 GB of address space, and so a 512MB limit wouldn't keep
DrRacket safe if the actual use can be 4 times that.

> In "other-boring-language-land", C# has an
> "AddMemoryPressure(bytesAllocated)" method to "inform the runtime of a
> large allocation of unmanaged memory that should be taken into account
> when scheduling garbage collection."
> http://msdn.microsoft.com/en-us/library/system.gc.addmemorypressure.aspx
> PyPy has a similar internal method that does the same thing.

Yes, I'll have to adjust the GC to support something similar that works
with reachabilty-based accounting.


> Neil Toronto <neil.toronto at gmail.com> writes:
> > The intended purpose is to *fill* memory with an amount that Racket can
> > account for. Without doing that, Racket would think that it uses much
> > less memory than it actually does, and could fill up your RAM long
> > before it decides it's time to collect garbage. All you would need to do
> > is create a few hundred large bitmaps.
> >
> > If DrRacket runs out of memory and closes without warning, that's an
> > error that isn't caused by the shadow byte string in particular. IOW,
> > the shadow byte string is doing what it's supposed to do, but Racket is
> > reacting wrongly to running out of memory.
> >
> > I can't help but think there's a better way than having that shadow byte
> > string, though, especially if Racket's static analysis ever gets good
> > enough to prune it away. But dealing with foreign libraries (like Cairo)
> > is tricky, and they're all quirky in their own special way.
> >
> > Neil ⊥
> >
> > On 12/19/2012 08:26 AM, Bert De Ketelaere wrote:
> >> Thank you for the answer,
> >>
> >> That explains why loading a bitmap (without the shadow byte-string)
> >> didn't raise the the value reported by current-memory-use.
> >>
> >> I'm wondering now in what way this helps the GC and the memory-limit checks?
> >> The reason I started looking into this was that DrRacket kept running
> >> out of memory
> >> and closing down without a warning (even with a memory limit set to 512 MB).
> >> So to me it seems that for current-memory-use and DrRacket's memory
> >> limit check
> >> this trick doesn't really work.  But maybe this is not the intended
> >> purpose of the
> >> shadow byte-string.
> >>
> >> Bert
> >>
> >>
> >>  > From: mflatt at cs.utah.edu
> >>  > To: bedeke at hotmail.com
> >>  > CC: users at racket-lang.org
> >>  > Subject: Re: [racket] bitmap% size in memory
> >>  > Date: Wed, 19 Dec 2012 07:11:24 -0700
> >>  >
> >>  > The "shadow" byte string is indeed unused. It's there only as an
> >>  > accounting trick for Racket's garbage collector.
> >>  >
> >>  > The memory used for the actual bitmap is not visible to the garbage
> >>  > collector, so it doesn't directly count towards a program's (or
> >>  > custodian's) memory use. The shadow byte string is allocated and kept
> >>  > alive during the same time as an actual bitmap to create GC-visible
> >>  > memory use. That is, the shadow byte string triggers full garbage
> >>  > collections and memory-limit checks as needed to keep the system as a
> >>  > whole working right.
> >>  >
> >>  > This trick does use about twice the memory that a bitmap would
> >>  > otherwise use, but it makes the garbage collector and bitmaps play
> >>  > nicely without complex changes to the garbage collector.
> >>  >
> >>  > At Wed, 19 Dec 2012 08:10:37 +0100, Bert De Ketelaere wrote:
> >>  > >
> >>  > > Hello all,
> >>  > >
> >>  > > When loading large pictures (± 4300x2600 pixels) I would expect
> >> that they takes
> >>  > > up around 45Mb of memory.
> >>  > > But when I monitored my program in windows task-manager I noticed
> >> it was more
> >>  > > in the neighborhood of 100Mb (after garbage collection, the
> >> in-between values
> >>  > > goes over 200Mb)
> >>  > > Hoping to find a reason for this I went looking into
> >> racket/draw/bitmap and
> >>  > > noticed that there is a variable "shadow" that is set to a zeroed out
> >>  > > byte-string, the size of the picture. But the only thing happening
> >> with this
> >>  > > "shadow" is it's definition and a set!. It is never read.
> >>  > > I tried removing the two references to "shadow", and in my programs
> >> everythings
> >>  > > seems to keep working, with the added bonus that the picture now
> >> indeed takes
> >>  > > only around 45Mb instead of 100Mb.
> >>  > >
> >>  > > two questions:
> >>  > > As far as I understand classes, since shadow is locally defined, no
> >> parent or
> >>  > > child can access shadow, so removing it should be ok. Is this right?
> >>  > > Is there a way to see how much memory an object occupies?
> >>  > >
> >>  > > Bert
> >>  > >
> >>  > > ____________________
> >>  > > Racket Users list:
> >>  > > http://lists.racket-lang.org/users
> >>
> >>
> >> ____________________
> >>    Racket Users list:
> >>    http://lists.racket-lang.org/users
> >>
> >
> > ____________________
> >   Racket Users list:
> >   http://lists.racket-lang.org/users
> 
> ____________________
>   Racket Users list:
>   http://lists.racket-lang.org/users