[racket-dev] using Racket to build an image-based Lisp: feasible?

On Mar 4, 2013, at 9:22 AM, Matthew Flatt <mflatt at cs.utah.edu> wrote:

> At Mon, 04 Mar 2013 08:50:19 -0600, mikel evins wrote:
>> Is the bytecode format platform-independent?
> 
> Yes.
> 
>> Is it documented? I guess I probably won't need to know gory details
>> about the bytecode format, but I could be wrong.
> 
> The gory encoding details are not documented, but the documentation of
> `compile/zo-structs' is essentially documentation of the data
> structure.
> 
> The `compiler/zo-parse' and `compiler/zo-marshal' provide functions for
> parsing and writing bytecode in terms of the `compile/zo-structs'
> structures.
> 
>> Does fasl serialization use the bytecode format? 
> 
> Yes.
> 
>> Do you see serious problems with building a solution based on
>> serialization and fasls?
> 
> No --- that seems like the way to go.

Thanks again; that all sounds pretty encouraging. Racket's mostly a pleasure to work with, so I'm pleased to find that maybe I can make stuff work with it.


>> I used to work with Will Clinger and John Ulrich's MacScheme; it had the nice 
>> feature of supporting a free mixture of bytecode and native-code procedures. 
>> Is that a doable thing in Racket, either as it stands right now, or as a 
>> feature that could be added without a lot of trouble?
> 
> The bytecode format doesn't have a nice way to do that, currently. (You
> can put a native-code shared library in place of a bytecode
> implementation of a module, but I don't think that's what you want.)
> I'm not sure what it would mean to mix native-code procedures with
> bytecode --- where the procedures would come from, for example --- so
> I'm not how difficult it would be to support.

The reason I ask is that MacScheme made it easy to have most of an app compiled to bytecode, which was very compact and reasonably efficient, but to optionally and selectively compile performance-critical procedures to native code. That would be a nice feature to have in Bard, because it would mean that the great majority of an application code be platform independent, but the hot spots could be compiled selectively to native code for the current platform in order to provide better performance.

I know nothing at all about how procedures are represented and invoked in Racket, so I have no diea what would be involved in supporting features like that. With MacScheme, the choice of bytecode or native code was a compile-time decision controlled on a per-procedure basis by optimization parameters. I don't know the representation details, but, in essence, a bytecode procedure and a native-code procedure were records containing different executable code. They were interchangeable from the point of view of calling code, differing only in internal representation. MacScheme did not target multiple platforms, but that feature is great for any piece of software that does target multiple platforms, because it means the entire app can be built as a completely portable binary, with performance-critical pieces optionally and easily adjusted to use platform-specific native code.

> Your project sounds really interesting --- not only on its own terms,
> but to me, also as an experiment in driving Racket in a new direction.
> So far, we have pursed a path at the opposite end of the design
> spectrum from Smalltalk and Lisp images (e.g., with modules, phases,
> and separate module instantiations across phases in Racket). It would
> be great to have a path to the other side. I think Liitin[1] maybe has
> similar goals, and maybe there's an opportunity for some shared
> infrastructure.

Happy to be of service. :-)

Besides the practical reasons I already gave, I also think that there's been a gradual trend away from image-based development over the last twenty years, and I think it's regrettable. I'm not saying it's the greatest thing since sliced bread, or that it's a cure for cancer; I just think it has some useful and pleasant things to bring to software development, and it's so rare these days that those things are in danger of being lost. There is a generation of younger Lisp programmers who don't know what image-based development is, and that seems a shame.