[racket] Cleaner way to work with gzipped data?

Figured it out and cleaned it up. It turns out that I was using
with-handlers oddly, but reading further though the documentation it works
as expected. Here's a new version (generalized to any input-port):

(define (with-gunzip thunk)
  (define-values (pipe-from pipe-to) (make-pipe))
  (with-handlers ([exn:fail?
                   (λ (err)
                     (close-output-port pipe-to)
                     (close-input-port pipe-from)
                     (error 'with-gunzip (exn-message err)))])
    (gunzip-through-ports (current-input-port) pipe-to)
    (close-output-port pipe-to)
    (parameterize ([current-input-port pipe-from])
      (thunk))
    (close-input-port pipe-from)))

If anyone's interested in a more in depth write up / source code for this
and with-gzip:
- writeup: http://blog.jverkamp.com/2013/08/06/adventures-in-racket-gzip/
- source:
https://github.com/jpverkamp/small-projects/tree/master/blog/with-gzip.rkt


On Mon, Aug 5, 2013 at 5:36 PM, JP Verkamp <racket at jverkamp.com> wrote:

> Thanks! make-pipe isn't something that I've had to use otherwise, so I
> missed the optional parameter. That does certainly seem to help.
>
> Here's my first take of with-input-from-gzipped-file:
>
> (define (with-input-from-gzipped-file filename thunk #:buffer-size
> [buffer-size #f])
>   (call-with-input-file filename
>     (lambda (file-from)
>       (define-values (pipe-from pipe-to) (make-pipe buffer-size))
>
>       (thread
>           (λ ()
>             (gunzip-through-ports file-from pipe-to)
>             (close-output-port pipe-to)))
>
>       (current-input-port pipe-from)
>       (thunk)
>       (close-input-port pipe-from))))
>
> The main thing missing is that there's no error handling (where the pipe
> should still be closed). At the very least, if I try to call this on a
> non-gzipped file, it breaks on the gunzip-through-ports line.
> Theoretically, some variation of with-handlers should work (error should
> raise an exn:fail?, yes?), but it doesn't seem to be helping.
>
> Any help with that?
>
> Alternatively, I've now found this:
> http://planet.racket-lang.org/display.ss?package=gzip.plt&owner=soegaard
>
> It seems to do exactly what I need, albeit without the call-with-* forms,
> but that's easy enough to wrap. With some very basic testing, it does seem
> to be buffering though, although it is a bit slower than the above. Not
> enough to cause trouble though.
>
>
> On Mon, Aug 5, 2013 at 4:51 PM, Ryan Culpepper <ryanc at ccs.neu.edu> wrote:
>
>> On 08/05/2013 04:29 PM, JP Verkamp wrote:
>>
>>> Is there a nice / idiomatic way to work with gzipped data in a streaming
>>> manner (to avoid loading the rather large files into memory at once). So
>>> far as I can tell, my code isn't doing that. It hangs for a while on the
>>> call to gunzip-through-ports, long enough to uncompress the entire file,
>>> then reads are pretty quick afterwords.
>>>
>>> Here's what I have thus far:
>>>
>>> #lang racket
>>>
>>> (require file/gunzip)
>>>
>>> (define-values (pipe-from pipe-to) (make-pipe))
>>> (with-input-from-file "test.rkt.gz"
>>>    (lambda ()
>>>      (gunzip-through-ports (current-input-port) pipe-to)
>>>      (for ([line (in-lines pipe-from)])
>>>        (displayln line))))
>>>
>>
>> You should probably 1) limit the size of the pipe (to stop it from
>> inflating the whole file at once) and 2) put the gunzip-through-ports call
>> in a separate thread. The gunzip thread will block when the pipe is full;
>> when your program reads some data out of the pipe, the gunzip thread will
>> be able to make some more progress. Something like this:
>>
>> (define-values (pipe-from pipe-to) (make-pipe 4000))
>> (with-input-from-file "test.rkt.gz"
>>   (lambda ()
>>     (thread
>>
>>       (lambda ()
>>         (gunzip-through-ports (current-input-port) pipe-to)
>>         (close-output-port pipe-to)))
>>
>>     (for ([line (in-lines pipe-from)])
>>       (displayln line))))
>>
>>  As an additional problem, that code doesn't actually work.
>>> in-lines seems to be waiting for an eof-object? that
>>> gunzip-through-ports isn't sending. Am I missing something? It ends up
>>> just hanging after reading and printing the file.
>>>
>>
>> The docs don't say anything about closing the port, so you'll probably
>> have to do that yourself. In the code above, I added a call to
>> close-output-port.
>>
>> Ryan
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/users/archive/attachments/20130806/d8a0aabf/attachment-0001.html>