[racket] parse-tools / lexer behavior

I don't know enough about your grammar to know if stringing together
lexers is a good idea or not. That approach is often a bit of a hack
(but sometimes a useful one). The "right" way, of course, is to divide
your grammar into lexical and syntactic grammars, and use a lexer for
the former and a parser for the latter.

As for concrete examples using multiple lexers, the example reader in
collects/parser-tools/examples/read.rkt uses a second lexer
specifically for string tokens.


On Mon, Jun 10, 2013 at 5:21 PM, David Weinstein <dweinst at insitusec.com> wrote:
> If I decided to go with the multiple lexers route to basically change
> "modes" while I'm in the context of parenthesis... how would one do this?
> Should I create an action like
>
> (define parameter-lexer
>   (lexer
>    [(eof) 'EOF]
>    [(:or "V" "Z" "B" "S" "C" "I" "J" "F" "D") (token-SIMPLE_TYPE lexeme)]
>    [className (token-CLASS_NAME lexeme)]))
>
> And
> (define node-lexer
>   (lexer
>    [(eof) 'EOF]
>    [".method" 'dirMethod]
>    ...
>    ["(" `(OP ,(parameter-lexer input-port))]
>    [")" 'CP]
>    ...))
>
> This would seem to create some artificial hierarchy that the parser would
> have to understand (a list within the existing list for example?)? Should I
> flatten afterwards to get rid of that?
>
> Would love to see a concrete example of multiple lexers in action if someone
> has it.
>
> Thanks much!
>
> David
>
> On Sat, Jun 8, 2013 at 10:05 PM, Jon Zeppieri <zeppieri at gmail.com> wrote:
>>
>> The results you're seeing are consistent with longest match;
>> "ILandroid" is longer than "I." The order of the [trigger action-expr]
>> pairs won't affect that. The order only affects which action is chosen
>> for an ambiguous match, but this case isn't ambiguous.
>>
>> I don't think you can do what you're trying to do with only a (single)
>> lexer, unless there are some additional rules that aren't expressed in
>> your grammar (for example, "qualifiers cannot start with upper-case
>> letters").
>>
>>
>>
>> On Fri, Jun 7, 2013 at 7:44 PM, David Weinstein <dweinst at insitusec.com>
>> wrote:
>> > Can anyone explain the behavior that is observed here with the
>> > parser-tools/lexer. I think I'm overlooking something but both
>> > precedence
>> > and trying to make the longest match should be creating the expected
>> > behavior (please see comments in the .rkt). Thanks for your help!
>> >
>> >
>> > David
>> >
>> >
>> > ____________________
>> >   Racket Users list:
>> >   http://lists.racket-lang.org/users
>> >
>
>