[racket] matching eof in a parser-tools lex-abbrev

On Mon, Mar 5, 2012 at 4:57 PM, Neil Van Dyke <neil at neilvandyke.org> wrote:
> In a "define-lex-abbrev", how do I match an EOF?

According to the documentation for that library, eof is a special
case.  It's not a regular expression, and is a peer of the other
triggers (special, special-comment) for a rule.  It's meant to signal
to anyone consuming the tokenizer to stop, since no more tokens are
going to come out.


Are you trying to absorb the EOF in one of your lexing rules?  If so,
you can peek at the very next character in your newline rule, and do
something special there, like this:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
#lang racket
(require parser-tools/lex
         parser-tools/lex-sre)

(define-lex-abbrev my-newline
  (or (: #\return #\newline)
      (: #\newline)
      (: #\return)))

(define my-lexer
  (lexer [my-newline (cond [(eof-object? (peek-byte input-port))
                            'newline/eof]
                           [else
                            'newline])]
         [(eof) eof]))

(define ip (open-input-string "\r\n\n"))
(my-lexer ip)
(my-lexer ip)
(my-lexer ip)
(my-lexer ip)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;