[racket-dev] using Racket to build an image-based Lisp: feasible?

On Mar 4, 2013, at 8:09 AM, Matthew Flatt <mflatt at cs.utah.edu> wrote:

> At Sat, 02 Mar 2013 08:44:38 -0600, mikel evins wrote:
>> I'm working on a Lisp. Recently I've been tinkering with an implementation in 
>> Racket. Racket makes it impressively easy to write.
>> 
>> I need it ultimately to be an old-fashioned kernel+image Lisp, with a 
>> platform-specific kernel executable and a platform-independent heap image. 
>> 
>> I'm aware that's not how Racket is designed. The question is: how hard would 
>> it be to use Racket to build such a thing? Easy enough to be worth trying? Too 
>> hard to consider? Can someone see a way to do it that would be easy enough to 
>> be worth a try?
>> 
>> I can imagine building something that might work out of the existing support 
>> for serialization and fasl files, but would be interested to know if folks 
>> more knowledgeable than I can see how to do it better.
> 
> I don't see a better approach than that one.
> 
> Long ago, we tinkered with platform-specific images based on memory
> dumps, and I can see some potential to make that work again via GC
> support. But it wouldn't be easy, it's especially low-level hacking,
> and it probably would not work out for platform-independent images.

Thanks for the response.

Forgive my ignorance about some of Racket's details; I'm not new to Lisp or Scheme, and not *completely* new to Racket, I don't know it particularly well, yet.

Is the bytecode format platform-independent? (I assume it is, but it's always best to check such assumptions). Is it documented? I guess I probably won't need to know gory details about the bytecode format, but I could be wrong.

Does fasl serialization use the bytecode format? Do you see serious problems with building a solution based on serialization and fasls?

I used to work with Will Clinger and John Ulrich's MacScheme; it had the nice feature of supporting a free mixture of bytecode and native-code procedures. Is that a doable thing in Racket, either as it stands right now, or as a feature that could be added without a lot of trouble?

To provide a little context:

The Lisp in question is Bard:

  https://github.com/mikelevins/bard

It's been a hobby project for some years. I started it because I missed the old Newton-era Dylan (I worked on the Dylan-based OS for Newton at Apple). It evolved some over the years, absorbing influences from functional languages and from my own ideas about how to factor type systems.

Last year, for the first time, it became a useful working tool: I used it extensively in a game-programming contract. Last month I finished tidying that version up and removing the contract-specific parts, and released it for anyone's use. That version is still missing some parts of the language, though, and it's a pretty slow AST-walking interpreter. I intend the next release to sport a proper compiler and VM.

Racket fared extremely as I was testing development platforms. It was really easy to implement a good skeleton version of Bard, and the performance of the Racket-based version was very good in my tests. Its main drawback is that it's not a VM+image implementation.

Why do I care about that?

I have some goals for Bard that--well, they don't strictly *require* that kind of architecture, but they make me very much want it. One point is that I plan an interactive development environment loosely modeled on SK8 (another Apple project I used to work on). In that model you build apps by directly modifying the development environment and saving the changes. I miss that way of working and want it back. 

Using a VM+image model, with portable images, as in the Smalltalk world, I can have that kind of environment running on and delivering to every platform I'm interested in. I can save an image on one machine and restart it on another. I can exchange image dumps with my collaborators. Smalltalkers have made this work well. I want those features, but I want to work in Lisp.

So Racket looks pretty attractive, if I can solve the problem of making the VM+image architecture work. Another option would be to start at a lower level and build up the infrastructure myself. I know how to do it, but will choose to go that route only if it looks like it will be less work than leveraging already-existing infrastructure like Racket's. If I didn't have to write the VM, GC, data structures, and so on from scratch, that would be swell. :-)