[racket-dev] Master-Worker with Places

On 03/29/2012 12:00 PM, Nick Shelley wrote:
> In my limited experience with parallel computing, it seems like the 
> master-worker paradigm is somewhat common. However, it seems like 
> Racket's places (or at least the way events are done with 
> place-channels) makes this inconvenient. Since there is nothing that I 
> can find in the result of syncing on a place-channel that would allow 
> me to then send some new work to the same channel, I have to implement 
> this manually by having each channel keep track of it's place in the 
> list of workers and send that information explicitly. This approach 
> seems hacky and more prone to bugs.

You need to use wrap-evt or handle-evt with a closure that includes the 
channel.
>
> I think it would make more sense for the result of syncing on a 
> place-channel to return the channel itself as a result. This would 
> make an explicit call to place-channel-get necessary, which may be a 
> downside, but the upside is I could then put something on the same 
> channel or do whatever else I may want with it.
This is one way to do it, however we choose to make place-channels work 
like plain channels in Racket.
Its a trade off.  Either you have to explicitly call place-channel-get 
or you have to use wrap-evt or handle-evt to maintain a reference to the 
channel.
>
> If changing how place-channel events are treated isn't feasible, I 
> think it would at least be useful to provide an abstraction that makes 
> master-worker more convenient. One idea might be a sync-channel 
> function that returns the channel.
wrap-evt and handle-evt are those abstractions.

Place-channel is meant to be a simple primitive, enabling more 
complicated parallel constructs to be built on top of place-channels.
We don't yet have a master-worker library. It would be nice if we did.

Some example you can look at include:
The places paper at 
http://www.cs.utah.edu/plt/publications/dls11-tsffd.pdf, demonstrates a 
master-worker parallel build using a jobqueue and handle-evt.
The file collects/setup/parallel-do.rkt is a very complicated example of 
a master-worker implementation using work-queues and wrap-evt

> As a sort of side note, it would be nice to be able to treat this 
> problem similar to a user thread problem where you can conceptually 
> imagine having infinite workers and just giving one chunk of work to 
> each of them. I tried to do this at first, but because of the way 
> places are implemented, I ran out of file descriptors relatively 
> quickly (and the overhead of starting a new VM for each chunk of work 
> might have been too much anyway, I don't know). I don't know if an 
> abstraction like that is possible or useful in general, but it may be 
> something to consider.
>
Places are too resource expensive to spawn one for each work item.  You 
have to use a work-queue and a small number of worker places.

Kevin
>
>
> _________________________
>    Racket Developers list:
>    http://lists.racket-lang.org/dev



On 03/29/2012 12:00 PM, Nick Shelley wrote:
> In my limited experience with parallel computing, it seems like the 
> master-worker paradigm is somewhat common. However, it seems like 
> Racket's places (or at least the way events are done with 
> place-channels) makes this inconvenient. Since there is nothing that I 
> can find in the result of syncing on a place-channel that would allow 
> me to then send some new work to the same channel, I have to implement 
> this manually by having each channel keep track of it's place in the 
> list of workers and send that information explicitly. This approach 
> seems hacky and more prone to bugs.
>
> I think it would make more sense for the result of syncing on a 
> place-channel to return the channel itself as a result. This would 
> make an explicit call to place-channel-get necessary, which may be a 
> downside, but the upside is I could then put something on the same 
> channel or do whatever else I may want with it.
>
> If changing how place-channel events are treated isn't feasible, I 
> think it would at least be useful to provide an abstraction that makes 
> master-worker more convenient. One idea might be a sync-channel 
> function that returns the channel.
>
> As a sort of side note, it would be nice to be able to treat this 
> problem similar to a user thread problem where you can conceptually 
> imagine having infinite workers and just giving one chunk of work to 
> each of them. I tried to do this at first, but because of the way 
> places are implemented, I ran out of file descriptors relatively 
> quickly (and the overhead of starting a new VM for each chunk of work 
> might have been too much anyway, I don't know). I don't know if an 
> abstraction like that is possible or useful in general, but it may be 
> something to consider.
>
> -Nick
>
>
> _________________________
>    Racket Developers list:
>    http://lists.racket-lang.org/dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/dev/archive/attachments/20120329/23b7be95/attachment-0001.html>