[racket-dev] using Racket to build an image-based Lisp: feasible?

On Mar 4, 2013, at 9:48 PM, Matthias Felleisen <matthias at ccs.neu.edu> wrote:

> 
> On Mar 4, 2013, at 10:35 AM, mikel evins wrote:
> 
>> Besides the practical reasons I already gave, I also think that there's been a gradual trend away from image-based development over the last twenty years, and I think it's regrettable. I'm not saying it's the greatest thing since sliced bread, or that it's a cure for cancer; I just think it has some useful and pleasant things to bring to software development, and it's so rare these days that those things are in danger of being lost. There is a generation of younger Lisp programmers who don't know what image-based development is, and that seems a shame.
> 
> In my younger days, I programmed in Prologs and in Schemes, including Chez, that saved heaps and took off where you left off. I know old Lispers and Smalltalkers (some are younger than me) are still in love with this world view but I will admit that I never regretted having left this part of Lisp behind. 
> 
> What specifically do you think is the allure? 

I don't think; I know. I've done the experiment.

For years, while working at Apple, I worked with Macintosh Common Lisp, and with several tools that were built on it. MCL was an image-based Lisp, and so were the things we built on it.

I grew accustomed to building tools and applications by directly modifying the running Lisp. It was a fast, facile way to work, and I miss it when I don't have it. We could iterate many times more quickly, and debug much more rapidly, than when we had to work with C or C++, or with any other toolchain that didn't offer the same liveness. We still had one of the earliest implementations of Smalltalk-80 around, and it had the same advantages.

The process of building an application was similar to creating a document in a word processor, or in any ordinary application: open the project; make changes; save the project; done. Whole species of ceremony I was accustomed to from other development systems just got out of the way and vanished.

We learned to keep reference images that could be given to others to bring them instantly up-to-date on what we were working on. We kept staged images that recorded our progress at known times, providing running binaries whose function was analogous to that of tagged commits in revision-control systems. In seconds I could switch to a previous known state of the whole system to recover from a mistake, or to compare with a previous known state. In a few minutes I could examine the environment in a dozen incrementally-different states in order to triangulate the cause of a bug.

I generalized the lessons of those development systems to the way I work with whole operating systems. I prefer to work with virtualized systems all the time, because I can keep a backlog of VM snapshots that serve a purpose for my whole working system analogous to what saved heap images served for applications I was working on.

One of the tools I worked on and with for several years was SK8, a system designed to support very rapid development of media-rich applications. SK8 was a sort of logical extremum of the paradigm of rapid development by direct manipulation. Every part of it could be "opened" and modified in-place. People built simple applications in seconds instead of minutes, and complex ones in hours instead of days, by literally snapping elements together and telling them what to do. None of that would have worked anywhere nearly as well if deployment were not as simple as choosing "Save" from a menu.

Sharing working ideas and incremental progress was as simple as handing someone a saved image. They could be running your changes a few seconds after you mentioned them. If they had ideas for changes, you could be running those changes a few seconds later.

Lisp, Smalltalk, and FORTH all give me an environment in which development is a live conversation between me and the machine. Image-based Lisps and Smalltalks give me a better version of that, in which the machine remembers what we talked about, so we can just move on. Working without images is like talking to someone who always needs to read out all of the accumulated minutes of previous conversations before continuing the discussion. It works; I can adapt to it, but it's tedious and stilted by comparison. Rainer Joswig aptly calls it "working with a dead thing on a table", instead of interacting with something living.

I can get along fine without images, and with batch compilation, and whatever. But it's hard to be satisfied with those when I've experienced something better. I constantly miss it when I don't have it.

Less practically, working that way from day to day over a span of years encouraged me to develop a certain perspective about the work we were doing. It's a little hard to articulate, and I became conscious of it only recently, in the last few years. I think it started to become clear while I was working a lot with Clojure. I like Clojure, but something is missing--something fundamental that was hard to put my finger on. 

After thinking about it for a while, it occurred to me that working with Clojure felt more like working with Icon than with MCL, or with other Lisps that I used to work with. As in Icon, there is a fundamental barrier between me and my program that did not exist in image-based systems. Working with Clojure, like working with Icon, was sort of a hands-off, action-at-a-distance affair by comparison with image-based systems. The Oberon system had a similar kind of detachment about it, despite Wirth's admirable efforts to build an engaging interactive system. In a conversation about that, Alan Kay said that Wirth "never got it", meaning that his exposure to Smalltalk had not resulted in a fundamental understanding of what made it work.

Clojure has that same detachment, despite the brilliant work Rich Hickey has done on it, and its community--a bright, inventive, and helpful community--seems to be missing something, too. There are kinds of issues and tools that they just don't seem to ever imagine. I attribute it to their being without any experience of image-based development. The poor guys don't know what they're missing.

I wish that I could articulate more clearly exactly what it is that I'm alluding to here; I can't. I'm still thinking about it, trying to clarify in my mind exactly what it is. I do know that it is intimately tied to the image-based experience, and the way that it promotes an ongoing live conversation with the machine that gradually comes to exist as you develop a program.

As I say, I don't think this is the be-all, end-all of programming. I can work just fine without it, and it's not without its own costs. Working with a gradually-evolving state creates new opportunities for errors; the evolving state can include subtle mistakes that persist over time. Your application can acquire dependencies on dynamic state that is, in truth, ephemeral, and which continues to exist only because you're saving ephemeral state. It's important to have those known-good reference images to fall back to, and it's important to regularly test the process of building your app from the ground up, to ensure that such errors have not accumulated.

It's also not important to all programmers. For some, the liveness I'm talking about seems to make no difference. That's fine; if you're a person who doesn't benefit from that kind of engagement then obviously my argument will leave you cold. But that difference is like the difference between people who can taste quinine and those who can't; or between those who can taste the unpleasant bitter flavor in cilantro, and those who can't. If you can taste it, no argument from someone who can't will make a difference to you, and vice versa.

Without image-based development I can't be as joyful in my work, nor as fast, nor can I know my work as intimately. I've mostly worked on products meant to be shipped to customers, and the advantage of such intimate engagement and knowledge is very large. For the programmer who doesn't benefit from image-based development, of course, it doesn't exist at all, and the absence of image-based systems is of no consequence. But for those who do benefit, being deprived of image-based systems is crippling, and if there aren't any of them around to experience then those people don't even have a way to know that they're crippled.

> Would your Lisp work as a #lang language? 

It already does. 

--me