<div dir="ltr">I don't know the specifics of your task, but I have been working on a sort of soup to nuts web crawling system. The system itself allows for pretty concise specification of most types of web crawling that I've encountered, and depending on what you need it contains a functional http-browser that manages cookies etc, and that in turn contains a reimplementation of a chunk of net/ that can parse and generate client side cookies. It should be install-able as a racket package. <a href="https://github.com/emdonahu/boris">https://github.com/emdonahu/boris</a> </div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Jan 8, 2014 at 9:33 AM, Matthew Flatt <<a href="mailto:mflatt@cs.utah.edu" target="_blank">mflatt@cs.utah.edu</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In case no one offers a better library, enclosed is a small one that I recently created for a web-scraping task. Start a simulation of a browser with `make-connection`, use `goto!` to follow a link to a relative URL (following redirects), and use `back!` to go back. The `goto!` function returns two values: the headers as a string and the page content as bytes. Beware: My application accessed a single site, so this library doesn't attempt to do the right thing with cookies across sites. <div class="HOEnZb"><div class="h5"> At Wed, 08 Jan 2014 03:48:44 -0800, Duncan Bayne wrote: > Hi All, > > I'm trying to re-write some Common Lisp web-scraping code in Racket. > > In Common Lisp, I'm POSTing a login request, and storing the cookie-jar > for subsequent GETs: > > (defun login (username password) > "Logs in to <a href="http://www.example.com" target="_blank">www.example.com</a>. Returns a cookie-jar containing > authentication details." > (let ((cookie-jar (make-instance 'drakma:cookie-jar))) > (drakma:http-request "<a href="http://www.example.com/login" target="_blank">http://www.example.com/login</a>" > :method :post > :parameters `(("username" . ,username) ("password" . > ,password)) > :cookie-jar cookie-jar) > cookie-jar)) > > ; snip > > (defun get-page (page-num cookie-jar) > "Downloads a potentially invalid HTML page containing data to scrape. > Returns a string containing the HTML." > (let ((url (concatenate 'string "<a href="http://www.example.com/data/" target="_blank">http://www.example.com/data/</a>" > (write-to-string page-num)))) > (let ((body (drakma:http-request url :cookie-jar cookie-jar))) > (if (search "No data found." body) > nil > body)))) > > However, I can't find an equivalent in Racket. The latest HTTP > library[1] makes no mention of cookies at all, and AFAICT the cookie > library[2] seems more about correctly serializing and deserializing > them. > > Can anyone suggest a way of re-writing the above CL in Racket without > having to implement a bunch of header-parsing stuff? > > TIA for any help ... > > [1] > <a href="https://github.com/plt/racket/blob/master/racket/collects/net/http-client.rkt" target="_blank">https://github.com/plt/racket/blob/master/racket/collects/net/http-client.rkt</a> > [2] <a href="http://docs.racket-lang.org/net/cookie.html" target="_blank">http://docs.racket-lang.org/net/cookie.html</a> > > -- > Duncan Bayne > ph: <a href="tel:%2B61%20420817082" value="+61420817082">+61 420817082</a> | web: <a href="http://duncan-bayne.github.com/" target="_blank">http://duncan-bayne.github.com/</a> | skype: > duncan_bayne > > I usually check my mail every 24 - 48 hours. If there's something > urgent going on, please send me an SMS or call me. > ____________________ > Racket Users list: > <a href="http://lists.racket-lang.org/users" target="_blank">http://lists.racket-lang.org/users</a></div></div> ____________________ Racket Users list: <a href="http://lists.racket-lang.org/users" target="_blank">http://lists.racket-lang.org/users</a> </blockquote></div> </div>