[racket] Is there a technical or social reason that parser-tools/lexer doesn't have an "else"?

  On Mon, May 21, 2012 at 7:54 PM, Danny Yoo <dyoo at hashcollision.org> wrote:
> I've been frustrated with trying to compose lexers together.  The
> situation is that I'd like to define a lexer that knows how to give up
> gracefully, and to give another lexer a go at processing a port.
>
> Ideally, I'd like to say something like this:
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> (define lexer-1
>  (lexer ...
>           [else (lexer-2 ip)]))
>
> (define lexer-2
>  (lexer ...
>           [else (error 'i-dunno)]))
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

parser-tools/lex-sre has a sequence regular expression (:), which can
match an empty sequence. (It does warn you that it accepts an empty
sequence, since it's guaranteed to succeed, but it doesn't consume
input.)

#lang racket
(require parser-tools/lex
         (prefix-in : parser-tools/lex-sre))

(define lexer1
  (lexer
   ["a"  lexeme]
   [(::) (lexer2 input-port)]))

(define lexer2
  (lexer
   ["b" lexeme]))

(define (test1)
  (define in (open-input-string "ab"))
  (values (lexer1 in)
          (lexer1 in)))

(define (test2)
  (define in (open-input-string "b"))
  (lexer1 in))