[racket] Question about parser-tools/lex

From: Philippe Mechaï (philippe.mechai at gmail.com)
Date: Wed Oct 17 10:01:01 EDT 2012


I am trying to use the lexer from the parser-tools collection.
I have written a minimal example that exhibits a behavior that seems odd to
me (code is at the end of the email).

The first two tests behave as expected but I would have expected the third
to fail. I understand that it does not, given the way the lexer is
implemented, but I cannot figure out how to change this behavior.

Could someone please give me directions to fix this ?

Thanks in advance.

Philippe Mechaï

Sample code:
#lang racket/base

(require racket/port
         (prefix-in : parser-tools/lex-sre))

(define-tokens data (ID NUM))
(define-empty-tokens delim (EOF))

(define sample-lexer
   [(eof) 'EOF]
   [whitespace (sample-lexer input-port)]
   [(:+ alphabetic) (token-ID (string->symbol lexeme))]
   [(:+ numeric) (token-NUM (string->number lexeme))]))

(define (test-lexer instr)
  (printf "~%Test [~a]:~%" instr)
  (with-handlers ([exn:fail? (λ (e)
                               (eprintf "ERROR: ~a~%" (exn-message e)))])
    (with-input-from-string instr
      (λ ()
        (printf "token: ~a~%" (sample-lexer (current-input-port)))))))

;; TEST 1
(test-lexer "*")
;; TEST 2
(test-lexer "4 a")
;; TEST 3
(test-lexer "4a")
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/users/archive/attachments/20121017/e1e17b3b/attachment.html>

Posted on the users mailing list.