[racket-dev] `_fun' types for higher-order functions in the ffi

This is a design issue in the ffi -- I'd really appreciate comments
from people who use it, particularly if you're using higher-order
functions (= callbacks) with pointer arguments.  It's long because I
don't see a good way out, so it's partly me thinking out loud.

There is a long-standing issue that has recently popped up about the
design of `_fun' types in the ffi.  It came up as a result of pointing
to usage of higher order functions in the qsort() example from the
test suite.  A readable version of the code is:

  (define qsort
    (get-ffi-obj 'qsort #f
                 (_fun (l    : (_list io _int len))
                       (len  : _int = (length l))
                       (size : _int = (ctype-sizeof _int))
                       (compare : (_fun _pointer _pointer -> _int))
                       -> _void -> l)))
  (qsort '(7 1 2 3 5 6 4 8 0 9)
         (λ (x y)
           (let ([x (ptr-ref x _int)] [y (ptr-ref y _int)])
             (cond [(< x y) -1] [(> x y) +1] [else 0]))))

The obvious thing that sticks out here is the fact that the callback
function is defined to receive plain `_pointer' values, and does the
dereferencing manually.  The obvious thing to try (which I did,
obviously not having much experience with it, and Sam did too) is to
use a `_ptr' type:

  (define qsort
    (get-ffi-obj 'qsort #f
                 (_fun (l    : (_list io _int len))
                       (len  : _int = (length l))
                       (size : _int = (ctype-sizeof _int))
                       (compare : (_fun (_ptr i _int) (_ptr i _int) -> _int))
                       -> _void -> l)))

But this doesn't work -- it fails with an obscure error:

  Scheme->C: expects argument of type <int32>; given: #<cpointer>

The thing is that the (_ptr i _int) type is used in a sensible way to
call *out* to C functions.  In this case, what it does is take a
Racket integer value, allocate a pointer and put the integer in it,
then call out to the C function with the pointer.  The problem is that
it tries to do exactly the same thing for a call*back* racket
function, which doesn't make any sense -- at least AFAICT...

Roughly speaking (I didn't really try it), it would take a C integer
argument, and do the same wrapping-in-a-cpointer, and pass that to the
racket function.  I hope that nobody wrote any code that actually uses
that...

Making up a type that does work for callbacks is simple with the
`_fun' machinery:

  (define-fun-syntax _unptr
    (syntax-rules ()
      [(_ t) (type: _pointer pre:  (x => (ptr-ref x t)))]))
  (define qsort
    (get-ffi-obj 'qsort #f
                 (_fun (l    : (_list io _int len))
                       (len  : _int = (length l))
                       (size : _int = (ctype-sizeof _int))
                       (compare : (_fun (_unptr _int) (_unptr _int) -> _int))
                       -> _void -> l)))
  (qsort '(7 1 2 3 5 6 4 8 0 9)
         (λ (x y) (cond [(< x y) -1] [(> x y) +1] [else 0])))

This is an obvious way to "fix" thing, but it leaves the nonsense part
of using `_ptr' for callbacks -- and the whole point of `_fun' was to
make things uniform so that you never need to know which "side" a
particular (part of a) type is on.

Now I'm getting to the part with the open questions about a possible
redesign...

The `_ptr' type is currently defined as follows for the three modes:

  (define-fun-syntax _ptr
    (syntax-rules (i o io)
      [(_ i  t) (type: _pointer
                 pre:  (x => (let ([p (malloc t)]) (ptr-set! p t x) p)))]
      [(_ o  t) (type: _pointer
                 pre:  (malloc t)
                 post: (x => (ptr-ref x t)))]
      [(_ io t) (type: _pointer
                 pre:  (x => (let ([p (malloc t)]) (ptr-set! p t x) p))
                 post: (x => (ptr-ref x t)))]))

I'm thinking of extending `define-fun-syntax' so that it gets either
one or two macros -- if there's only one, it does the same thing as
now.  But when a second one is specified, it is used instead for
wrappers of callbacks.  This definition can now be rewritten as:

  (define-fun-syntax _ptr
    (syntax-rules (i o io) ...same...)
    (syntax-rules (i o io)
      [(_ i  t) (type: _pointer
                 pre:  (x => (ptr-ref x t)))]))

And using this, (_ptr i _int) would work nicely on either side.  But
that doesn't really work...

To see why, consider (_ptr o _int) on a callback: I currently don't
see a sane meaning for this, but regardless, the fact that there are
two syntax transformers is already weird.  The thing is that you can
write (_ptr i _int) right now, and adding a second kind of transformer
makes things assymetric in a way that seems like a bad idea.

In fact, I think that how the whole thing behaves now is already
broken.  For example, it's possible to do this:

  (define _int* (_ptr o _int))
  (define qsort
    (get-ffi-obj 'qsort #f
                 (_fun (l    : (_list io _int len))
                       (len  : _int = (length l))
                       (size : _int = (ctype-sizeof _int))
                       (compare : (_fun _int* _int* -> _int))
                       -> _void -> l)))

and this works -- but using the `_ptr' form directly in the `_fun'
expression is as broken as it was before.

So that's where I got to -- I think that it's a significant problem,
but I don't see a good way to resolve it.

-- 
          ((lambda (x) (x x)) (lambda (x) (x x)))          Eli Barzilay:
                    http://barzilay.org/                   Maze is Life!