[racket] parse-tools / lexer behavior

From: David Weinstein (dweinst at insitusec.com)
Date: Mon Jun 10 17:21:57 EDT 2013

If I decided to go with the multiple lexers route to basically change
"modes" while I'm in the context of parenthesis... how would one do this?
Should I create an action like

(define parameter-lexer
   [(eof) 'EOF]
   [(:or "V" "Z" "B" "S" "C" "I" "J" "F" "D") (token-SIMPLE_TYPE lexeme)]
   [className (token-CLASS_NAME lexeme)]))

(define node-lexer
   [(eof) 'EOF]
   [".method" 'dirMethod]
   ["(" `(OP ,(parameter-lexer input-port))]
   [")" 'CP]

This would seem to create some artificial hierarchy that the parser would
have to understand (a list within the existing list for example?)? Should I
flatten afterwards to get rid of that?

Would love to see a concrete example of multiple lexers in action if
someone has it.

Thanks much!


On Sat, Jun 8, 2013 at 10:05 PM, Jon Zeppieri <zeppieri at gmail.com> wrote:

> The results you're seeing are consistent with longest match;
> "ILandroid" is longer than "I." The order of the [trigger action-expr]
> pairs won't affect that. The order only affects which action is chosen
> for an ambiguous match, but this case isn't ambiguous.
> I don't think you can do what you're trying to do with only a (single)
> lexer, unless there are some additional rules that aren't expressed in
> your grammar (for example, "qualifiers cannot start with upper-case
> letters").
> On Fri, Jun 7, 2013 at 7:44 PM, David Weinstein <dweinst at insitusec.com>
> wrote:
> > Can anyone explain the behavior that is observed here with the
> > parser-tools/lexer. I think I'm overlooking something but both precedence
> > and trying to make the longest match should be creating the expected
> > behavior (please see comments in the .rkt). Thanks for your help!
> >
> >
> > David
> >
> >
> > ____________________
> >   Racket Users list:
> >   http://lists.racket-lang.org/users
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/users/archive/attachments/20130610/e990706c/attachment.html>

Posted on the users mailing list.