[racket] Remote execution in Racket

On Tue, Aug 24, 2010 at 7:08 PM, Richard Cleis <rcleis at mac.com> wrote:
> Can you provide an example? (At this point, one of the profs around here
> asserts, "Code, please." :)

You're right, code will clarify this.  I should have started out
asking what people would recommend to do something.  My users end up
writing things that look like:

~~~
#lang s-exp bein

(execute
  (run "/bin/touch" "boris")
  (import "boris")
  (run-binding ((a 'stdout)) "/bin/echo" "This is a test")
  (import "This is a test"))
~~~

execute sets up a temporary directory, and each command inside works
therein.  run is a wrapper around subprocess that does a bunch of
behind the scenes work.  import pulls a file from the temporary
directory into a managed repository (stored in an SQLite database in
the user's home directory).

execute is nestable as well.  For instance,

~~~
(define (f x) (execute (run "/bin/touch" x)))
(execute
  (f "boris")
  (import "boris"))
~~~

only sets up a single temporary directory.  The execute in the
function f is flattened into the outermost enclosing execute.

This has to run on a cluster.  The cluster consists of a couple of
frontend machines users log into plus fifty or sixty nodes which do
nothing but run batch jobs.  The batch jobs are submitted via the
commandline or via a C API, and generally look like

$ bsub /path/to/program arg1 arg2 ...

Basically, it takes shell commands.  The C API does as well, but takes
an argument of type char**.  I want to write a version of execute that
submits its body as a batch job via LSF.  Once the submission happens
it need not maintain any communication with the original process,
since the disk (and thus the SQLite database that maintains state) is
shared across the cluster.

I would also like to be able to write things like

~~~
(define (f x) (execute-lsf ...stuff with x...))
(map f '(list of things to apply f to)
~~~

Someone pointed out serializable continuations in the stateless
servlets code to me, which seem like just about the perfect solution,
but they seem to be tied into that framework.

Any suggestions would be very welcome.

-- 
Frederick Ross
Bioinformatics and Biostatistics Core Facility
Life Sciences, EPFL
http://bbcf.epfl.ch/
+41 21 693 14 39