[plt-scheme] read-delimited-list

Eli,
Thanks for your elaborated answer. You are right by observing that I wanted 
to connect the reader and syntax expansion phase. OK, you fully convinced me 
that separating the phases is wise (like I very much appreciate the elegant 
and powerfull separation of expansion and run time layers of modules)
Of course the MzScheme manual is not a tutorial and I agree that it must 
certainly not be converted into a tutorial. In fact the manual shows a very 
elaborated example, but this one shows more information than a newby on 
readtables can inhale in a short time (even though I may call myself an 
experienced programmer in many languages). I looked into the cookbook, but 
there I found no examples. Anyway, this thread and and particularly your 
advices have given me a good starting point for reading and understanding 
the documentation on readtables. In the past I have written a lot of parsers 
in languages that do not provide readtables. But do they simplify life a 
lot! Thanks.
Jos Koot

PS1
Indeed, by mistake I sent you a version without unquotesplicing (a version 
that works). What I ment to send was:
(define read-braces
  (case-lambda
   ((ch port src line col pos)
   #`(special-lambda #,@(read/recursive port #\{ #f)))
   ; very nice: unquote-splicing-syntax accepted
   ((ch port) `(special-lambda ,@(read/recursive port #\{ #f))))) ; <---  
mark the @
with another version of syntax special-lambda. The last line results in an 
error because quasiquote/unquote-splicing cannot process the opaque 
placeholder. But after separating reading and expansion, this no longer is a 
problem.

PS2
Matthew, sorry about heading my previous post with 'Matthias and Eli'. Of 
course I ment 'Matthew and Eli'. I owe a lot to Matthias, that's probably 
the cause of my mistake.

----- Original Message ----- 
From: "Eli Barzilay" <eli at barzilay.org>
To: "Jos Koot" <jos.koot at telefonica.net>
Cc: "Matthew Flatt" <mflatt at cs.utah.edu>; <plt-scheme at list.cs.brown.edu>
Sent: Monday, August 21, 2006 8:03 PM
Subject: Re: [plt-scheme] read-delimited-list


> On Aug 20, Jos Koot wrote:
>> Hi Matthias and Eli,
>> My earlier attempts to adapt readtables failed because I failed to
>> understand the docs. Your examples in this thread are very
>> clear. May be they deserve to be included in the help desk.
>
> Perhaps.  The information is all there -- the problem is that it is in
> reference form rather than a tutorial form.  (And a lot of discussions
> went on on that issue, the cookbook is one example, wikifying the docs
> is another possible solution.)
>
>
>> I am facinated by the new world of tools for the definition of
>> customized languages with just a tiny bit of programming. Yet I have
>> two questions as displayed in the example below. Be assured that
>> they are not ment to be interpreted as a critic. I merely wonder
>> whether ot not it would be possible to tune MzScheme's read-tables
>> such as to avoid the pittfalls I fell into.
>
> Keep in mind that using readtables is different than using a new
> syntax using the parser tools -- the main advantage is that they allow
> you to hook onto (Mz)Scheme's syntax and extend it in a local way, but
> that leads to some restrictions too.
>
>> ;;; module my-reader
>> ;;; {var ... expr} ==> (lambda (var ...) expr) ; mark the braces
>> ;;; ~ expr ==> (some-syntax expr)
>
> The second should be very easy to implement, the first could require a
> little more work than Paul's problem because you can't just read the
> whole sub-expression and use it as is -- but you solve this by doing
> the massaging in the `special-lambda' syntax.  In any case, your code
> demonstrates some of the additional issues that I didn't go into in
> the previous post, so hopefully this would be helpful in further
> clarifications (so it will also be verbose).
>
> [re-editing your code for readability]
>
>> (module my-reader mzscheme
>>   (provide special-lambda)
>>   ;; it would be nice not to be obliged to export this syntax
>>   ;; (question 1: is there a way???)
>>   (define-syntax special-lambda
>>     (syntax-rules ()
>>       [(special-lambda (var ... expr)) (lambda (var ...) expr)]))
>
> This is related to the main problem in your code.  When you're using
> any syntactic extensions in PLT, it is best to think about the result
> as adding yet another dimension to the layers that make up your code.
> The usual dimension is macros: in this dimension there is the phase
> that is associated with every piece of code: the run-time code, the
> syntax level, the second syntax level etc.  Matthew's "You want it
> when" paper describes the resulting tower of phases, and the fact that
> MzScheme tries to keep a strict separation of phases.
>
> With the addition of reader code, you get another dimension to this
> story -- the reader code is above (or below, depending on your POV)
> all of these phases.  Because it is in this position, it should be
> kept separated from the rest of the code.  Of course, you *could* mix
> in the levels, and that can lead to some usual mixed messes.  For
> example:
>
>  > (define counter (let ([c 0]) (lambda () (set! c (add1 c)) c)))
>  > (define dollar-reader (lambda _ (counter)))
>  > (current-readtable
>     (make-readtable #f #\$ 'terminating-macro dollar-reader))
>  > (list $ $ $)
>  (1 2 3)
>  > (list $ (counter) $)
>  (4 6 5)
>
> I think that there are very few uses for the mess that you see in that
> last interaction.  To continue with your code -- the problem is that
> you're using a single module for reader functionality, as well as
> bindings that this reader should be using.  It is best to keep the two
> separate: one module to provide the reader extensions and another
> module to give them meaning.  You *could* use the same module for
> both, but if you do so you have to keep in mind that this is a
> hack...  In your case, the `special-lambda' syntax binding is
> irrelevant for the reader code so there is no need for it at
> read-time, and OTOH, when you consider the *meaning* of your code
> (bindings (both values and syntaxes)) there is no need for the reader
> code.
>
>
>>   (define read-braces
>>     (case-lambda
>>       [(ch port src line col pos)
>>        #`(special-lambda #,(read/recursive port #\{ #f))]
>>       ;; very nice: unquote-splicing-syntax accepted
>>       [(ch port) `(special-lambda ,(read/recursive port #\{ #f))]))
>
> Here you have another problem that is related to this issue, and a
> pitfall that would probably confuse several people.  The difference
> between the first and second cases is that you use syntax in the
> first, and a quoted s-expression in the second.  You may have noticed
> that in Matthew's initial reply, he used only a quasiquoted
> s-expression for the reader.  This is an important but subtle point
> that is a problem in your code.  The thing is that the reader should
> return simple syntax, with no lexical information.  In your case, the
> use of `quasisyntax' makes the reader return a syntax that already has
> lexical context.  Here's a simple example:
>
>  > (module foo mzscheme
>      (define a 1)
>      (define (read-dollar . _) #'a)
>      (current-readtable
>       (make-readtable #f #\$ 'terminating-macro read-dollar)))
>  > (require foo)
>  > $
>  stdin::66: compile: access from an uncertified context to unexported
>  variable from module: foo in: a
>
> The problem is that when the reader gets to the "$", it gets a syntax
> that already has lexical information -- in this case it's a direct
> referral to the `a' that is not exported from the module.  (The
> resulting syntax is `3d'.)  The result is a certificate error, similar
> to manually expanding a macro reasult and trying to expose a hidden
> identifier that way.  Another example that shows this:
>
>  > (module foo mzscheme
>      (define a 1) (provide (rename a b))
>      (define (read-dollar . _) #'a)
>      (current-readtable
>       (make-readtable #f #\$ 'terminating-macro read-dollar)))
>  > (require foo)
>  > $
>  1
>  > '$
>  a
>  > (let ([a 123]) $)
>  123
>  > (let ([b 123]) $)
>  1
>
> The rule of thumb is that reader code should *never* use any of the
> `syntax' forms.  It is fine, however, to use `datum->syntax-object' --
> as long as the first argument is `#f' so the result has no lexical
> information.  When you return a non-syntax value, it is converted to
> syntax using `datum->syntax-object' and `#f' for the lexical
> information, so it is fine to do that too.
>
> (This is all described in the "Procedure result" part of section
> 11.2.9 in the MzScheme manual.)
>
>
>>   ;; unquote-splicing not accepted (understandible, but not
>>   ;; consistent, this is question 2)
>
> I don't know what this is referring too -- I see no splicing in your
> code.
>
>
>>   (define read-tilde
>>     (case-lambda
>>       [(ch port src line col pos)
>>        #`(some-syntax
>>           #,(read/recursive port #f (current-readtable)))]
>
> Notes:
>
> * #f and (current-readtable) are defaults, so you could use
>  (read/recursive port)
>
> * Suffers from the same problem as above (use s-expressions instead)
>
> * But you should also change `read/recursive' to
>  `read-syntax/recursive'.  The difference is that the latter will
>  give you syntax objects that contain proper source information.
>
>
>> [...]
>> unquote-splicing: expected argument of type <proper list>; given
>> #<placeholder>
>
> Maybe you pasted code that is different than what you've used?  In any
> case, the deal with `read/recursive' and `read-syntax/recursive' is
> that they almost always return an opaque `placeholder' value that you
> cannot use.  These values are later changed when graph notations (#N=
> and #N#) are resolved.
>
> -- 
>          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
>                  http://www.barzilay.org/                 Maze is Life!