[racket-dev] net/http-client

This looks great!!

A couple suggestions:

1. Support for "Expect: 100-continue" request headers would be
helpful, and I think not too messy to add.

The big use case I'm aware of is Amazon S3. If you make a PUT or POST
request, it might need to redirect you to another URI (outage,
balancing, whatever reason). Expecting and handling 100-continue lets
you avoid transmitting potentially large amount of data that would be
discarded, and you have to send it all over again in the request to
the redirect URI. For (say) a 1 GB upload to S3, this matters.
Although I don't know for sure if other upload-ish web APIs offer
same, I'd guess some do as this is the use case for 100-continue
generally.

How I implemented this in my HTTP package was to have a
`start-request` function that sends the request line and headers,
peeks the response status line, then returns a `boolean?` whether the
PUT/POST/PATCH/whatever data should be transmitted. [1]

I think your `http-conn-send!` could do similar?

2. Support for "Content-Encoding" response headers would also be helpful.

Using the same make-pipe approach as you're doing with chunked
transfer encoding. [2]  Maybe this is mission creep: For HTTP 1.1. you
_must_ support Transfer-Encoding: chunked, whereas Content-Encoding is
just optional. However it's a good option; using compression can
really help out on time as well as bandwidth charges.


IIRC those were the two main things that motivated me to make my HTTP
package at all, to support e.g. my AWS package. If http-client added
them, I might not need my package anymore. (OK, it might take me
awhile to phase it out until I'm ready to de-support older versions of
Racket, but, I and others wouldn't need it for new projects.)


[1]: https://github.com/greghendershott/http/blob/master/http/request.rkt#L142-L189

[2]: By the way, do you want to pass some `limit` optional arg in the
various uses of `make-pipe`? Otherwise IIUC this will suck everything
into RAM, which might not be so great with very large request or
response entities.


On Fri, Aug 23, 2013 at 2:48 PM, Jay McCarthy <jay.mccarthy at gmail.com> wrote:
> Based on a request back in early July to remove the restrictions that
> net/url puts on HTTP communication (vis a vis URL encoding), I have
> just pushed a new HTTP client as net/http-client.
>
> The push also changes net/url to use net/http-client so that we just
> have 1 HTTP request producer, rather than 3.
>
> It passes all of the net/* tests, but these don't use features like
> proxying, HTTP/1.1, etc. I'm slightly nervous that it doesn't do those
> correct, but not super nervous, because I just cut-and-pasted the
> code.
>
> The main approach of the library is best explained by this contract:
>
> [http-sendrecv
>    (->* ((or/c bytes? string?) (or/c bytes? string?))
>         (#:ssl? (or/c boolean? ssl-client-context? symbol?)
>                 #:port (between/c 1 65535)
>                 #:method (or/c bytes? string? symbol?)
>                 #:headers (listof (or/c bytes? string?))
>                 #:data (or/c false/c bytes? string?))
>         (values bytes? (listof bytes?) input-port?))]
>
> Compared to net/url,
> - It supports bytes and strings everywhere
> - It supports data on every method and not just POST
> - It always returns the status line, headers, and content (as a port)
>
> I feel that the only thing it could do better is support two more
> options for #:data:
> - A input-port? to read from and copy to the HTTP connection
> - A (-> output-port? void) function to call with the HTTP connection's
> output port to stream the data
>
> But I'd like a second opinion before adding them.
>
> Jay
>
> --
> Jay McCarthy <jay at cs.byu.edu>
> Assistant Professor / Brigham Young University
> http://faculty.cs.byu.edu/~jay
>
> "The glory of God is Intelligence" - D&C 93
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev