[racket] color partial lexemes

At Mon, 01 Aug 2011 18:13:11 -0600, Jon Rafkind wrote:
> How do I deal with partial lexemes when using the color:text colorer? It
> seems that if my lexer cannot produce a valid token then it will throw
> an error and the color thread will keep calling `get-token' in a busy loop.
> 
> Is there some magic thing I can return from `get-token' to tell the
> colorer that it should not progress to the next position?

A coloring lexer's job is to never fail. It should instead categorize
bad input as 'error and be ready to continue.

Continuing can mean categorizing all further input as 'error, but it's
usually better to continue in some other way.