[racket-dev] net/http-client

On Wed, Sep 4, 2013 at 11:03 AM, Jay McCarthy <jay.mccarthy at gmail.com> wrote:
> On Wed, Aug 28, 2013 at 3:30 PM, Greg Hendershott
> <greghendershott at gmail.com> wrote:
>>
>> This looks great!!
>>
>> A couple suggestions:
>>
>> 1. Support for "Expect: 100-continue" request headers would be
>> helpful, and I think not too messy to add.
>>
>> The big use case I'm aware of is Amazon S3. If you make a PUT or POST
>> request, it might need to redirect you to another URI (outage,
>> balancing, whatever reason). Expecting and handling 100-continue lets
>> you avoid transmitting potentially large amount of data that would be
>> discarded, and you have to send it all over again in the request to
>> the redirect URI. For (say) a 1 GB upload to S3, this matters.
>> Although I don't know for sure if other upload-ish web APIs offer
>> same, I'd guess some do as this is the use case for 100-continue
>> generally.
>>
>> How I implemented this in my HTTP package was to have a
>> `start-request` function that sends the request line and headers,
>> peeks the response status line, then returns a `boolean?` whether the
>> PUT/POST/PATCH/whatever data should be transmitted. [1]
>>
>> I think your `http-conn-send!` could do similar?
>
>
> Do you think it is appropriate to expect the http-client user to put in the
> "Expect: 100-continue" or better to always send it if there is a data
> component?

Great question. I took the approach of requiring the client to supply
it if they care.

Instead supplying it always/automatically does seem neat.

But safe? Reading
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html I'm a little
nervous about this section:

  A server that does not understand or is unable to comply with any of the
  expectation values in the Expect field of a request MUST respond with
  appropriate error status. The server MUST respond with a 417
(Expectation Failed)
  status if any of the expectations cannot be met or, if there are
other problems
  with the request, some other 4xx status.

This suggests a scenario where a server might error a request solely
because of the presence of an "Expect: 100-continue" header. That
doesn't strike me as reasonable behavior, but I could imagine some
server doing it.

So I suppose best to default to supplying it automatically, but
provide a way for the client to disable that.

How to specify disabling?

With headers like "Server:" you have a default but let the user's
supplied headers override. But supplying "Expect: " (i.e. "Expect:
<blank>") would feel weird, to a user. And actually sending that -- if
it feels weird to a server causing it to error, well that's the whole
thing we're trying to avoid, see above.

So (tl;dr) perhaps add an optional function parameter that defaults to
#t, e.g. `[expect-100-continue? #t]` ?

>
>>
>> 2. Support for "Content-Encoding" response headers would also be helpful.
>>
>> Using the same make-pipe approach as you're doing with chunked
>> transfer encoding. [2]  Maybe this is mission creep: For HTTP 1.1. you
>> _must_ support Transfer-Encoding: chunked, whereas Content-Encoding is
>> just optional. However it's a good option; using compression can
>> really help out on time as well as bandwidth charges.
>
>
> I just pushed support for this.

Nice, thanks!