[racket] Passing info from reader level

At Wed, 12 Dec 2012 01:01:18 -0800, Rajah Mahsohn Omega wrote:
> Hi, I am creating my own language with racket, it's the same as racket
> except that I can embed metadata anywhere in the code. I have implemented a
> read and read-syntax function which detects the metadata and returns
> (make-special-comment metadata) this seems to be the only way to remove it
> from the code. The documentation on special-comments does not give any
> details on what it's used for but I did see references to it in
> parser-tools/lex.rkt.
> My questions are:
> Whats the difference in using the reader and expander layers in racket from
> using lex and yacc?

While there is some similarly, in that a reader and a lexer both
tokenize, a reader takes on the extra responsibility of matching
parentheses, and that extra step better sets up extensibility for the
expander step.

If you're particularly interested in this topic, you may want to see
Jon Rafkind's recent paper on adapting the reader+expander approach to
a less-parenthesized language. The main new ingredient is
"enforestation", which in a sense helps make up for the reader's more
limited role when less parentheses are available:

 http://www.cs.utah.edu/plt/publications/gpce12-rf.pdf


> How are special-comments used in racket?

Mostly for comment boxes in DrRacket.


> Is there any way to access the metadata outside of the reader?

I'm not sure how it plays out in your languages, but I think the right
way to deal with metadata in general is to put it in a submodule.
Then, to access the metadata, you just `require' the submodule.

For example, when you use `scribble/srcdoc' to mix documentation and
code, the documentation is a kind of metadata, and it is pushed out
into a submodule that isn't loaded when you just `require' the module.
An `include-extracted' form `require's documentation metadata to
include it documentation.

To get reader-based metadata into a submodule, you have to get it from
the reader through the expander layer, so that the metadata can be put
in a nested `module' form in the expanded module. That should work if
you're creating a reader and `#%module-begin' that cooperate, since the
`#%module-begin' can immediately shift data into a submodule.


I hope that helps, but let us know if you have more questions!