[racket] Question about parser-tools/lex

Hi,

I am trying to use the lexer from the parser-tools collection.
I have written a minimal example that exhibits a behavior that seems odd to
me (code is at the end of the email).

The first two tests behave as expected but I would have expected the third
to fail. I understand that it does not, given the way the lexer is
implemented, but I cannot figure out how to change this behavior.

Could someone please give me directions to fix this ?

Thanks in advance.

Regards,
Philippe Mechaï


Sample code:
--------------------
#lang racket/base

(require racket/port
         parser-tools/lex
         (prefix-in : parser-tools/lex-sre))

(define-tokens data (ID NUM))
(define-empty-tokens delim (EOF))

(define sample-lexer
  (lexer
   [(eof) 'EOF]
   [whitespace (sample-lexer input-port)]
   [(:+ alphabetic) (token-ID (string->symbol lexeme))]
   [(:+ numeric) (token-NUM (string->number lexeme))]))

(define (test-lexer instr)
  (printf "~%Test [~a]:~%" instr)
  (with-handlers ([exn:fail? (λ (e)
                               (eprintf "ERROR: ~a~%" (exn-message e)))])
    (with-input-from-string instr
      (λ ()
        (printf "token: ~a~%" (sample-lexer (current-input-port)))))))

;; TEST 1
(test-lexer "*")
;; TEST 2
(test-lexer "4 a")
;; TEST 3
(test-lexer "4a")
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/users/archive/attachments/20121017/e1e17b3b/attachment.html>