[racket] Inner workings of source annotation

Thank you for the tips pointing me in the right direction.

Apologies for not giving more context; that would certainly help. I’ve been embedded deeply in Racket-space for a couple years now, but lawyers prevent me from talking about my work and I’ve only recently been given permissions to ask questions in public. Please forgive my being indirect and opaque when I’d much rather dish the dirt.

I have a fairly intimate understanding of the macro expander, and understand that the resulting syntax objects retain source location, lexical context, and various properties.  Very soon after sending my email it dawned on me that I’d mistakenly assumed that the editor retained the original source objects and built a visual model atop that, but now I see that’s not the case. It looks like the association between the expanded source objects and the original source text is done solely through the source-location. Fair enough.

Since my last mail I discovered drracket/private/syncheck/traversals which seems to be the meat of it. I see that it recursively parses the expanded code (core syntax forms), builds several indices and cross-references, then sends the results to the editor layer via various syncheck-annotations methods.

In other words I think I’ve got a handle on it now.

Thanks again everyone!

—Todd

From: Matthias Felleisen <matthias at ccs.neu.edu<mailto:matthias at ccs.neu.edu>>
Date: Friday, June 20, 2014 at 5:22 PM
To: "Todd V. Jonker" <jonker at amazon.com<mailto:jonker at amazon.com>>
Cc: "users at racket-lang.org<mailto:users at racket-lang.org>" <users at racket-lang.org<mailto:users at racket-lang.org>>
Subject: Re: [racket] Inner workings of source annotation


On Jun 20, 2014, at 5:07 PM, Jonker, Todd wrote:

Greetings,

I’m interested in learning how DrRacket’s “Check Syntax” feature works.

As I understand it, the process is (1) perform full macro-expansion on the source code; (2) map elements of the expanded code back to the source code; (3) use that mapping to apply annotations/markup to the source code. The documentation at http://docs.racket-lang.org/tools/Check_Syntax.html seems to cover step (3), but I’m struggling to understand how (2) works.

In other words, I’m trying to learn how the lexical information in the expanded syntax objects is mapped back to the appropriate parts of the original syntax objects. I’m reading the code to try to figure out the algorithm but hoping that there’s a paper that describes it more directly.

Thanks in advance for any pointers,



Hi Todd,  good to see you on the list.

(1) local-expand is your friend, (2) the syntax system already propagates source location information (see syntax/loc in the docs), and (3) if I am guessing correctly you may want to look at "syntax property" lists in the docs just in case you want to implement a language of your own that needs to communicate between layers of expansion.

The Typed Racket example is the best documented in this regard. And the best paper to read is

  Languages as Libraries
  http://www.ccs.neu.edu/racket/pubs/#pldi11-thacff

If you play with Typed Racket, you will see that DrRacket 'lifts through' the syntax. It will do so for any language you create in the #lang spirit (with 2) and even macros.

There is also a paper on DrScheme and its implementation, which explains a bit about other aspects of implementing these things. It's probably much less helpful

If this doesn't help, perhaps you can ask more specific questions. -- Matthias


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/users/archive/attachments/20140621/5f5961ae/attachment-0001.html>