[racket] Is there a technical or social reason that parser-tools/lexer doesn't have an "else"?

From: Danny Yoo (dyoo at hashcollision.org)
Date: Mon May 21 19:54:34 EDT 2012

I've been frustrated with trying to compose lexers together.  The
situation is that I'd like to define a lexer that knows how to give up
gracefully, and to give another lexer a go at processing a port.

Ideally, I'd like to say something like this:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(define lexer-1
  (lexer ...
           [else (lexer-2 ip)]))

(define lexer-2
  (lexer ...
           [else (error 'i-dunno)]))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

As far as I can tell from reading the documentation, I can't make this
work nicely.  That is, I do not want lexer-1 to consume a character
when taking an alternative.

So putting:

;;;;;;
(define lexer-1
  (lexer ...
           [any-char (lexer-2 ip)]))
;;;;;;

despite what the documentation says about how this helps me handle the
error condition, is not a solution that sits right with me.  It
creates a non-composable situation with any other lexers I'd like to
work with.

To make it compose, I need to somehow get that character back into the
port.  I don't know how to efficiently push that character back into
the port besides using something like racket/port's input-port-append,
which is a solution, but not a simple one nor one that immediately
comes to mind.

The design of the parser-tools/lex library feels like it values
simplicity, so I'm wondering: is there a technical reason why
parser-tools/lex doesn't have an "else"?

Posted on the users mailing list.