<html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hello, I'm trying to understand how memory is allocated and collected when working with streams. I recently asked a question about how to limit memory when using streams on Stackoverflow and got two good answers:     <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> <a href="http://stackoverflow.com/questions/18629188/how-to-limit-memory-use-when-using-a-stream">http://stackoverflow.com/questions/18629188/how-to-limit-memory-use-when-using-a-stream</a> However, I'm seeking a better understanding than could really be given through the SO format. I want to use streams because I have too much data to fit in memory and hence want to use them to bring in the data from files and databases sequentially as needed.  However, I'm finding that the GC is not collecting as I would have hoped and hence streams are not quite as straightforward a solution as I expected.  The sort of problems that I am experiencing are demonstrated with the following code   #lang racket   (require rackunit)   ; This program fails with out of memory errors when memory limit set to 128mb   ; It always fails when it comes to testing filtered-nums, regardless of how test-nums?   ; and test-gen-nums? have been set.  However test-for/sum-gen-filtered-nums?   ; also fails if set.   (define max-num 10000000)   (define test-nums? #f)   (define test-gen-filtered-nums? #f)   (define test-for/sum-gen-filtered-nums? #f)   (define nums (in-range max-num))   (define filtered-nums     (stream-filter (λ (i) (values #t)) nums))   (define (gen-filtered-nums)     (stream-filter (λ (i) (values #t)) nums))   (when test-nums?     (displayln "Testing nums")     (check-equal? max-num (stream-length nums)))   (when test-gen-filtered-nums?     (displayln "Testing gen-filtered-nums")     (check-equal? max-num (stream-length (gen-filtered-nums))))   (when test-for/sum-gen-filtered-nums?     (displayln "Testing with for/sum-gen-filtered-nums ")     (check-equal? max-num (for/sum ([i (gen-filtered-nums)]) 1)))   (displayln "Testing filtered-nums")   (check-equal? max-num (stream-length filtered-nums)) I understand that making multiple passes through a big data is inefficient, but here I am trying to gain a better understanding of the GC. So this leads me to a few related questions:   i.  Why does the GC seem to collect more effectively when the stream is       created in a function as opposed to in a straight definition? i.e       test-gen-filtered-nums? passes, although I note that       test-for/sum-gen-filtered-nums? doesn't.   ii.  Is stream-filter inappropriate to use with big data sets?   iii. Is there a better choice than streams for dealing with big data sets, coming from       disparate sources such as files, databases, etc,  within racket? Thanks Lorry <pre class="moz-signature" cols="72">-- vLife Systems Ltd Registered Office: The Meridian, 4 Copthall House, Station Square, Coventry, CV1 2FL Registered in England and Wales No. 06477649 <a class="moz-txt-link-freetext" href="http://vlifesystems.com">http://vlifesystems.com</a> </pre> </body> </html>