[racket] color partial lexemes

On 08/01/2011 06:40 PM, Matthew Flatt wrote:
> At Mon, 01 Aug 2011 18:13:11 -0600, Jon Rafkind wrote:
>> How do I deal with partial lexemes when using the color:text colorer? It
>> seems that if my lexer cannot produce a valid token then it will throw
>> an error and the color thread will keep calling `get-token' in a busy loop.
>>
>> Is there some magic thing I can return from `get-token' to tell the
>> colorer that it should not progress to the next position?
> A coloring lexer's job is to never fail. It should instead categorize
> bad input as 'error and be ready to continue.
>
> Continuing can mean categorizing all further input as 'error, but it's
> usually better to continue in some other way.
>

It seems the colorer re-tokenizes the entire input stream each time I
enter a character. Is that the expected behavior?