[racket-dev] FFI: pointer to an array in a C struct type

On 12/03/2012 07:31 AM, Matthew Flatt wrote:
> At Mon, 3 Dec 2012 12:31:37 +0100, Tobias Hammer wrote:
>> On Mon, 03 Dec 2012 11:45:08 +0100, Neil Toronto <neil.toronto at gmail.com>
>> wrote:
>>
>>> This error seems wrong:
>>>
>>>
>>> #lang racket
>>>
>>> (require ffi/unsafe
>>>            ffi/unsafe/cvector)
>>>
>>> (define-cstruct _mpz ([alloc _int]
>>>                         [size _int]
>>>                         [limbs (_gcable _cvector)]))
>>>
>>> (define z (make-mpz 1 1 (list->cvector '(1) _long)))
>>> (mpz-limbs z)
>>>
>>>   >>> _cvector: cannot automatically convert a C pointer to a cvector
>>>
>>>
>>> The error might be correct, though late and confusing, if a cvector's
>>> "base representation" isn't a pointer type. Is it?
>>
>> The base representation should be a pointer but the length and type
>> information to convert it back to a cvector is missing. Whats stored
>> inside the struct is only a plain pointer, without the information you
>> supplied to make-mpz, and because of this _cvector only has a
>> racket->c-conversion (see ffi/unsafe/cvector.rkt).
>>
>>> If that error is correct, then how are FFI users meant to define a C
>>> struct that has a "long *" field that points to an array? I've not yet
>>> managed to define one whose instances survive a garbage collection cycle
>>> without using (_gcable _cvector). Here's one of my desperate attempts:
>>>
>>>
>>> #lang racket
>>>
>>> (require ffi/unsafe
>>>            ffi/unsafe/cvector)
>>>
>>> (define-cstruct _mpz ([alloc _int] [size _int] [limbs _gcpointer]))
>>>
>>> (define z (make-mpz 1 1 (cvector-ptr (list->cvector '(1) _long))))
>>>
>>>   > (ptr-ref (mpz-limbs z) _long 0)
>>> 1
>>>   > (collect-garbage)
>>>   > (ptr-ref (mpz-limbs z) _long 0)
>>> 139856348920568
>>>
>>>
>>> I mean to be complain-y this time. This shouldn't be this hard to figure
>>> out.
>>
>> I guess the problem here is, that the gc only knows memory with only
>> pointers (to gcable memory) inside and (atomic) memory without any
>> pointers it should follow. You try to mix them up. make-mpz should malloc
>> (see ffi docs) atomic memory and you put a pointer managed by the gc into
>> it. The pointer gets relocated on gc, but the ref inside the struct is not
>> known and not updated.
>>
>> I unfortunately don't know a simple, gc- and typesafe way to put pointers
>> inside structs. My attempts to get around this were pretty hacking (using
>> a raw malloc'd pointer, embed a struct with plain values as pointer inside
>> an interor-malloc'd struct together with all others pointers, etc).
>> I would be really interested in a way to accomplish this, too.
>
> Yes. I've thought about this, but for the libraries I've written, the
> cases where the GC can really solve the problem automatically seem
> relatively rare, so I haven't been sufficiently motivated for my own
> uses.
>
> In the `_mpz' example above, the presence of both `alloc' and `size'
> suggests that a callee might be responsible for malloc()ing `limbs' if
> there's not already enough room. If so, `_gcpointer' isn't the right
> type. Maybe this example doesn't actually go that way, but it's a
> typical problem.
>
> Neil, can you say more about how `_mpz' instances are used with foreign
> functions?

They represent GMP's bignums, and they're used as both input and output 
arguments.

When used as input arguments, GMP's functions never mutate them. It 
should be safe to pass an `mpz' that contains a pointer to memory that 
Racket manages, as long as it doesn't get relocated during the function 
call. In this case, I need to be able to set the `limbs' array's 
elements in Racket code.

When used as output arguments, GMP expects to be able to free the array 
pointed at by the `limbs' field, allocate a new array for the result, 
and set the `limbs' field to point to it. I use the GMP function 
"mpz_init" to initialize the fields of an `mpz' instance before using it 
as an output argument. In this case, I need to be able to read the 
`limbs' array's elements in Racket code.

I think that's everything.

Neil ⊥