[racket] Understanding GC when working with streams

Konrad's exactly right. Your filtered-nums blows up because you named
the stream and then traversed the entire thing. In general, if you
hang onto the head of the stream while traversing then the GC can't
collect anything because since you have a pointer to the head, every
element of the stream is still reachable.

Compare to a common traversal pattern like:

    (let loop ([s <some stream>]) ... (loop (stream-rest s)))

where the head is dropped on each loop iteration.

Other comments:
- (stream-length (gen-filtered-nums)) is fine because there's no
pointer to the head, so the GC collects as you traverse.

- in-range is fine because it's constant space and not a
cons-cell-based stream. It's more like a generator.

- the for/sum is actually collecting while traversing, just more
slowly. I'm not exactly sure why, I may look into it. On my machine,
it got up to 500mb or so but it finished.

On Sat, Sep 7, 2013 at 6:16 AM, Konrad Hinsen
<konrad.hinsen at fastmail.net> wrote:
> Lawrence Woodman writes:
>
>  >   i.  Why does the GC seem to collect more effectively when the stream is
>  >       created in a function as opposed to in a straight definition? i.e
>  >       test-gen-filtered-nums? passes, although I note that
>  >       test-for/sum-gen-filtered-nums? doesn't.
>
> I don't know much about the internals of Racket, but for this one I have
> a strong suspicion: The definitions
>
>   (define nums (in-range max-num))
>   (define filtered-nums
>     (stream-filter (λ (i) (values #t)) nums))
>
> don't allow the garbage collector to ever reclaim the storage occupied
> by the streams, because you could at any time traverse them again.
>
> Streams are conceptually handled like lists with a head and a tail,
> except that the tail is a thunk of code that is evaluated when needed.
> When you create a stream and walk through it, the heads can be
> discarded as they are handled. But if you keep a reference to the head
> of the whole stream, everything has to remain in memory.
>
> Konrad.
>
> ____________________
>   Racket Users list:
>   http://lists.racket-lang.org/users