[racket] color partial lexemes

Matthew Flatt wrote at 08/01/2011 08:40 PM:
> At Mon, 01 Aug 2011 18:13:11 -0600, Jon Rafkind wrote:
>   
>> How do I deal with partial lexemes when using the color:text colorer? It
>> seems that if my lexer cannot produce a valid token then it will throw
>> an error and the color thread will keep calling `get-token' in a busy loop.
>>     

Speaking generally of parsing for editors here, not of the DrRacket 
syntax-colorer specifically...

You can have the notion of an "error" token, and, if all other rules for 
matching a token fail, have a final rule (or rules) that consume your 
error token.  You can color the error token bright red boldface or 
similar.  Like I think Matthew said, you probably want to recover, not 
abort.

For your error token, you should try to consume a chunk of input such 
that the following text is likely to be a valid token that was likely 
intended.  This is more art than science, and depends very much on the 
language.  For your first attempt, you might try to just consume one 
character followed by any non-whitespace-or-punctuation.  You can also 
be smarter about things like unterminated string literals and other 
patterns of characters (e.g., "#" in Racket). 

-- 
http://www.neilvandyke.org/