[racket] My toy web app's performance: Racket vs Perl vs Ruby

On 04/11/2012 03:03 PM, joshua at anwu.org wrote:
>
> Racketeers,
>
> I'm new to the language, and loving it.  I'm having trouble getting
> good performance out of my little toy web app, however.
>
> I wrote this fake billing app to play with some new languages (like
> Racket) and some other webdev ideas I've had (like client-side
> templating with jQuery and mustache).  As such, I have the same JSON
> api written in Perl, Ruby, and Racket (working on node.js and
> Haskell).  Perl runs under Dancer and Starman, ruby under Sinatra and
> Unicorn, and Racket under nohup and its own included webserver. All
> are running on the same machine.  Each connects to a postgres db,
> executes some queries from a config file, and returns JSON to the
> client.
>
> I've been running apache bench against all three, and the performance
> of Racket vs Perl and Ruby has been... disheartening.  I compiled the
> racket code with 'raco exe' before running it, but Perl and Ruby both
> blow it away.  The racket executable also seems to grab and hold a
> lot of memory, even though I told it to be stateless.  It also tends
> to have failures.

You don't need to use 'raco exe'; it's for ease of distribution, not 
performance.

The stateless option doesn't really apply here, since you aren't using 
send/suspend (or any of the variants thereof).

> ab -c 20 -n 10000 <uri>

What URL(s?) are you testing? It's hard to analyze the numbers below 
without knowing what they're measuring.

> Perl:
>
> Concurrency Level:      20
> Time taken for tests:   86.100 seconds
> Complete requests:      10000
> Failed requests:        0
> Write errors:           0
> Total transferred:      88150000 bytes
> HTML transferred:       86300000 bytes
> Requests per second:    116.14 [#/sec] (mean)
> Time per request:       172.199 [ms] (mean)
> Time per request:       8.610 [ms] (mean, across all concurrent requests)
> Transfer rate:          999.82 [Kbytes/sec] received
>
>
> Ruby:
>
> Concurrency Level:      20
> Time taken for tests:   102.914 seconds
> Complete requests:      10000
> Failed requests:        0
> Write errors:           0
> Total transferred:      88480000 bytes
> HTML transferred:       86050000 bytes
> Requests per second:    97.17 [#/sec] (mean)
> Time per request:       205.827 [ms] (mean)
> Time per request:       10.291 [ms] (mean, across all concurrent requests)
> Transfer rate:          839.60 [Kbytes/sec] received
>
>
> Racket:
>
> Concurrency Level:      20
> Time taken for tests:   139.059 seconds
> Complete requests:      10000
> Failed requests:        687
>     (Connect: 0, Receive: 0, Length: 687, Exceptions: 0)
> Write errors:           0
> Total transferred:      9421469 bytes
> HTML transferred:       7100095 bytes
> Requests per second:    71.91 [#/sec] (mean)
> Time per request:       278.119 [ms] (mean)
> Time per request:       13.906 [ms] (mean, across all concurrent requests)
> Transfer rate:          66.16 [Kbytes/sec] received

Are you sure the "Failed requests" are really failures? A Stackoverflow 
answer suggests that these might be the result of a nondeterministic 
response length. (See http://stackoverflow.com/questions/579450.)

It's odd that the total transferred for the Racket benchmark is almost 
an order of magnitude less than the totals for Perl and Ruby. It would 
help to see the actual URLs used in the tests to make sure this isn't an 
apples-to-oranges comparison.

It would also be useful to run the benchmarks with "-c 1" to measure the 
pure sequential performance. I'm not familiar with the frameworks in 
question, but a brief scan suggests that Starman and Unicorn might be 
forking multiple OS-level workers, which would lead to more parallelism 
on multicore systems. Your Racket setup is limited to a single core.

> I'm hoping it's just inexperience on my part - maybe my Racket code
> just sucks.  Or maybe it's that I'm trying to stay functional and
> avoid mutation, which I don't bother with in the other two. Anyone
> interested in looking at the code and telling me what I'm doing
> wrong, or could do better?  I would love to use Racket for more
> serious projects.
>
> https://github.com/TurtleKitty/CalicoBill

A few of your functions (eg, customer-invoices) look like they suffer 
from the "N+1 problem" (basically, executing a query within a loop 
instead of executing a single query with a join), but I presume you're 
doing the same thing in Perl and Ruby, so that doesn't have anything to 
do with the performance *difference*.

I would have written 'sql-ts->string' using 'format' instead, and I 
would probably use 'for/list' and 'for/hash' instead of 'map' and 
'make-immutable-hash'. But overall your Racket code looks fine to me.

Ryan