[racket] Flatten syntax and runtime phases?

Hello again,

I'm following up in case anyone is curious.

After fighting with defmacro for a while I ended up abandoning it and
implementing what I need with syntax/parse instead. This requires
re-writing the arc macros, but that isn't all bad since their new
forms are generally shorter, clearer, and benefit greatly from
hygiene. With my current solution things only need to be bound
for-template, never for-syntax, and work as expected without any
tricks.

Additionally, I just wanted to say that Racket and its community have
really impressed me. Everyone on the lists is helpful and pleasant,
and the documentation is thorough and current. In particular,
syntax/parse is awesome, and has been invaluable so far.

Thanks again,
Nick

On Sat, Jun 16, 2012 at 1:47 PM, Nick Sivo <nick at kogir.com> wrote:
> Thanks for the leads!  I'll look into your suggestions and follow up
> with my results.
>
> -Nick
>
> On Fri, Jun 15, 2012 at 10:54 PM, Ryan Culpepper <ryan at cs.utah.edu> wrote:
>> On 06/15/2012 10:41 PM, Nick Sivo wrote:
>>>
>>> Hi,
>>>
>>> I have the following code:
>>>
>>> #lang racket
>>> (require mzlib/defmacro)
>>>
>>> (defmacro w/uniq (name . body)
>>>   `(let ([,name (gensym)])
>>>      , at body))
>>>
>>> (defmacro in (x . choices)
>>>   (w/uniq g
>>>      `(let ([,g ,x])
>>>         (or ,@(map (lambda (c) `(eqv? ,g ,c)) choices)))))
>>>
>>> Naturally, it fails:
>>>
>>> expand: unbound identifier in module (in phase 1, transformer environment)
>>> in: g
>>>
>>> Additionally defining w/uniq for-syntax fixes it:
>>>
>>> #lang racket
>>> (require mzlib/defmacro
>>>          (for-syntax racket
>>>                      mzlib/defmacro))
>>>
>>> (begin-for-syntax
>>>   (defmacro w/uniq (name . body)
>>>     `(let ([,name (gensym)])
>>>        , at body)))
>>>
>>> (defmacro w/uniq (name . body)
>>>     `(let ([,name (gensym)])
>>>        , at body))
>>>
>>> (defmacro in (x . choices)
>>>   (w/uniq g
>>>      `(let ([,g ,x])
>>>         (or ,@(map (lambda (c) `(eqv? ,g ,c)) choices)))))
>>>
>>> I know this is Racket's phase separation working as intended.  Now, I
>>> appreciate why Racket separates phases as well as the advantages of
>>> hygienic macros, but I'm not writing new code.  I'm trying to enable
>>> existing code* to run natively in Racket and take advantage of its
>>> great JIT and some module optimizations (like inlining).
>>>
>>> Is there any way to hack the first version to work? One option is to
>>> build a deeper nest of duplicate macro definitions each time I
>>> encounter a new one.  It seems possible, and it might even work, but
>>> it sounds awful.  I've already written a custom reader and have an
>>> extensive library of language syntax/parse macros, so I'm open to
>>> anything.
>>
>>
>> I don't know of a way of doing this within Racket other than running each
>> definition at every phase that it might be used at. That might not require
>> duplicating code, though.
>>
>> If you could rely on let*-like scoping for Arc module bodies, then you could
>> do it entirely with submodules (new feature since 5.2.1; see
>> http://blog.racket-lang.org/2012/06/submodules.html for an introduction).
>>
>> The idea would be to use the arc language's #%module-begin macro to
>> transform
>>
>> (module M arc
>>  defn1
>>  defn2
>>  defn3))
>>
>> into
>>
>> (module M arc
>>  (#%plain-module-begin
>>
>>  (module S0 arc)
>>
>>  (module S1 arc
>>    (require (submod ".." S0)
>>             (for-syntax (submod ".." S0)))
>>    defn1
>>    (provide (all-from-out (submod ".." S0))
>>             (all-defined-out)))
>>
>>  (module S2 arc
>>    (require (submod ".." S1)
>>             (for-syntax (submod ".." S1)))
>>    defn2
>>    (provide (all-from-out (submod ".." S1))
>>             (all-defined-out)))
>>
>>  (module S3 arc
>>    (require (submod ".." S2)
>>             (for-syntax (submod ".." S2)))
>>    defn3
>>    (provide (all-from-out (submod ".." S2))
>>             (all-defined-out)))
>>
>>  (require (submod "." S3))
>>  (provide (all-from-out (submod "." S3)))))
>>
>> Each S<N> submodule provides (at phase 0 only) the names defined by all
>> definitions defn1 through defn<N>. (For S0, that means zero definitions.)
>>
>> Each S<N+1> submodule requires S<N> at phases 0 and 1, thus it gets all
>> definitions defn1 though defn<N> for both run time and compile-time.
>>
>> That's if you have strict let*-like scoping. If you want to support mutually
>> recursive modules, you could try to group contiguous run-time definitions
>> together by head-expanding (ie, calling 'local-expand' with a stop list) and
>> accumulating forms as long as they produce 'define-values' forms. I don't
>> see any way to support mutually recursive functions that have macro
>> definitions between them, though.
>>
>> I also don't see any way to support references from a macro body to
>> functions defined later in the module.
>>
>> This will break 'set!', because variables imported from another module
>> cannot be mutated. But you can fix that by defining auxiliary setter
>> procedures and providing proxy set-transformer macros in place of the
>> variables.
>>
>> Mutating a variable (or data structure) in one phase still cannot change the
>> value in other phases, no matter what you do. In fact, compile-time mutation
>> won't work very well at all, since each definition is compiled in a new
>> submodule, which should get its own fresh compile-time state.
>>
>> If you have 'let-syntax' or its equivalent (eg, local 'define-syntax'), then
>> phases 0 and 1 might not be enough, since you can have expressions at phase
>> 2 (and higher):
>>
>>  (let-syntax ([m0
>>                (let-syntax ([m1 <phase-2-expr>]) <phase-1-expr>)])
>>    <phase-0-expr>)
>>
>> If you want to be really *really* crafty, you could try defining your own
>> version of 'let-syntax', 'define-syntax', etc to create requires for higher
>> phases on demand by calling 'syntax-local-lift-require'. That is,
>>
>>  (arc-define-syntax id expr)
>>
>>  == expands to the result of calling ==>
>>
>>  (syntax-local-lift-require
>>   #`(for-meta #,(add1 (syntax-local-phase-level)) S<N>)
>>   #'(racket-define-syntax id expr))
>>
>> I hope that helps. (I haven't tried any of this, so there are surely bugs
>> that need ironing out.)
>>
>> Ryan
>>