[racket] parser tools - defining tokens

On 10/04/2011 01:21 PM, George Neuner wrote:
> Hi all,
>
> I am playing with the parser tools in Racket. I would like to be able
> pass a separately defined list of symbols to define-empty-tokens (which
> is a syntax macro defined in parser-tools/private-lex/token.rkt).
> Unfortunately passing a list to define-empty-tokens doesn't work because
> the macro is expecting to find the token symbols coded in-line.
>
> My lexer uses a hash table to distinguish the reserved words in my
> language from other identifiers. The object is to initialize the parser
> and the hash table from the same list of symbols ... I don't want to
> have to maintain 2 identical lists.
>
> Currently I'm doing the following:
>
> =============================================
>
> (define *reserved-words* '(word1 word2 ...))
>
> (define *reserved-word-hash*
> (for/hash ((sym *reserved-words*))
> (values (string-downcase (symbol->string sym)) sym)
> ))
>
> (eval `(define-empty-tokens RESERVED ,*reserved-words*))
>
> =============================================
>
> This works, but using eval seems like a hack and the code fails syntax
> checking in DrRacket (the parser generator code complains that the
> RESERVED token group is undefined).
>
>
> I probably could modify the macro to return the symbols passed into it
> (I'm sure I don't know how to make it accept a separate list). But I'm
> not sure it is a good idea to mess with the macro anyway ... there is
> enough black magic in the yacc/lex implementation already.

No need to modify any existing macros; just create your own. It's far 
easier to move data from compile time to run time than vice versa. That 
is, instead of generating the define-empty-tokens form from 
*reserved-words*, generate both from a single macro:

(define-syntax-rule
   (define-empty-tokens+list tokens-name list-name (token ...))
   (begin (define-empty-tokens tokens-name (token ...))
          (define list-name '(token ...))))

(define-empty-tokens+list RESERVED *reserved-words* (word1 word2 ...))

Then you can define *reserved-words-hash* the same way.

Ryan