[racket] DSLs and complexity

A comment on this kind of article, and then on your question...

"Managing complexity" has been a notion in software engineering for a 
very long time.  It was a marketing tagline for a very expensive 
software engineering workstation thing that I worked on 20 years ago, 
for example.

In the spirit of the times, I'll make some crude generalizations:

* The good thing about "best practices" articles on software engineering 
is that reading a lot of the articles exposes to practictioners to many 
ideas.  For example, "the cost of some feature in software can be much 
greater than the initial cost to code it."  You're probably thinking 
"duh", but it's an idea everyone needs to hear at some point, and it's 
good to keep in mind.

* The bad thing about "best practices" articles on software engineering 
is that they often (and I am sometimes guilty of this) latch onto some 
idea and extrapolate from it in ways that are not warranted by the very 
limited data.  This is the realm of Management 'Science'.  Science is 
hard, but we are making pronouncements about complex systems (the 
software engineering process, not the software) without having even the 
same rigor as economists.

>
> For Racket: are DSLs a source of complexity? Or would you argue that 
> they reduce the
> complexity normally introduced with DSLs?

They DSLs *can* be sources of complexity, of course.  A journey through 
three of my *worst* DSLs:

* There is a DSL that I did in grad school, using an early version of 
Java.  It was originally intended to be a part of for a researchy CASE 
system.  (The idea was that you would have different ways of modeling 
parts of an system at a high level in the CASE system, and this DSL the 
back-end for executing models expressed in one of the CASE 
meta-models.)  That DSL took inordinately long to implement, partly 
because Java just was not suited to DSLs, and indeed the DSL arguably 
added complexity, and was not actually a win unless the rest of the 
system was developed.  A while after that, I decided I wanted to be able 
to do DSLs in whatever language I used, and that CL or Scheme was the 
obvious choice for that.  Today, I think Racket is the obvious first 
candidate if you want to do DSLs.

* I was working on a very powerful DSL for XML processing, in my early 
Scheme days, and I was doing it in "syntax-rules" so that it would work 
with almost any Scheme implementation.  It was mostly feature-complete 
before I decided I was doing development in a very expensive way, and 
that I wouldn't want to maintain it.  The language itself was elegant to 
use, but the problem was complexity of the implementation, since I was 
was doing it using "syntax-rules" and necessarily had rule patterns like 
"(_ Z S T O U L E R TV EN EB)".  (More embarrassing info at 
"http://www.neilvandyke.org/weblog/2008/11/#2008-11-03".)  Again, a DSL 
seemed the right way, but I wasn't using good tools for DSLs.

* There is one specialized data format conversion DSL that I made a few 
years ago, that is in daily production use, and I hope the DSL itself 
never needs a bug fix (so far, so good).  Drawbacks to this DSL are that 
the usage takes a while to understand, and that maintenance of the DSL 
implementation itself is not easy (mainly because I used "syntax-rules" 
instead of more modern Racket features, because the problem seemed 
simpler initially).  Still, I strongly suspect that that even that DSL 
saved us from numerous bugs and maintenance problems in implementing the 
details of the data conversion, and was a huge win for this reason.  If 
I had to do it over, I would do the DSL differently, but I would still 
use a DSL.

The theme seems to be that DSLs can go wrong, especially if you use poor 
DSL tools.

Those are my worst DSLs that I recall.  I also have had better DSLs.

Omigosh, I could write a best practices article: "DSLs can go wrong 
sometimes, especially if you don't use Racket. In conclusion, you should 
use Racket and DSLs for absolutely everything."

Neil V.