[racket] Concurrent execution

Sorry for thinking aload again, but I'm learning these issues myself in the background, and I wonder if something like the following might work?

Concurrent List Processing -idea

Two phase process

1) Distribution phase
Create a recursive procedure combining cons and threads so that each item in the original list will be replaced by a named thread. Sending out new threads won't wait for individual threads to return in this phase. The result of this phase is a list of named threads.

2) Collection phase
This phase is again a recursive cons process, but this time sequential. It starts from the beginning of the named-thread list, and as soon as the first thread returns its value, it moves on. The process goes on until the end of the list is reached. The resulting list is the return value.

A few extra considerations

- I earlier thought that the plain 'thread' procedure was actually doing things truly in parallel and utilising multiple processors as smart as possible where available. I still wish it were so, because at least the way I use sequential and concurrent processing, could be accomblish from the (naive )user's point of view, just by using 'begin' and 'thread'. The way 'thread' could work would be transparent to the user, combining and balancing between 'virtual Racket multiprocessing' within a single OS thread and multiple OS threads among one or more HW processors where available. At the moment, simple 'thread' won't do the trick IIUC.

- If the original list to be processed is very long, then the construction of the second phase list shouldn't wait for bottle neck items to cons the list elsewhere where possible. The speed gain wouldn't be substantial though, since it is just building a list, and you cannot finish it until the last item is available, anyway.

Does this sound even remotely feasible?

br, jukka


> -----Original Message-----
> From: users-bounces at racket-lang.org
> [mailto:users-bounces at racket-lang.org]On Behalf Of Robby Findler
> Sent: 01 October 2011 06:02
> To: Ivanyi Peter
> Cc: Racket mailing list
> Subject: Re: [racket] Concurrent execution
> 
> 
> Futures will let you share data, but you'd have to try to avoid
> allocation in the futures (not completely, just mostly) for good
> performance (so you'd probably want to build vectors to save the
> results into). For places, you cannot share the database, you'd have
> to send if over a place channel to be able to use it in multiple cores
> at once.
> 
> It is hard to say which is going to be better for your task; probably
> futures will be easier to try, so you might start there. They are more
> limited, tho, so you might end up switching to places.
> 
> Have you read the section on performance in the Guide? That provides
> an overview that you might find helpful.
> 
> Robby
> 
> 2011/9/30 Ivanyi Peter <pivanyi at freemail.hu>:
> > Hi All,
> >
> > I would like to ask for some advice.
> > I have a problem in which I have several lists containing some 
> data. These
> > lists are created by one
> > "process" and this part should be sequential. Then these lists should be
> > checked against some
> > large "database", list or hash table, and as a result some new 
> lists must be
> > built, which at the end must
> > be merged. I would like to do this later processing 
> concurrently on several
> > cores.
> > I am a little unsure which method to use: futures, places, threads,
> > processes???
> > (I am kind of familiar with parallel programming in C and MPI, 
> but not this
> > fine grained level.)
> >
> > I am not sure a couple of things here:
> > - how to "distribute" the several lists to the "processes" ? (I 
> do not want
> > to use files for this) Is it possible?
> > - how to merge the resulting lists at the end? Is it possible?
> > - how to share, if it is possible at all, the common "database", list or
> > hash table? If this is not really
> > possible, it is not a problem I can "load" it in on each "process". (At
> > least I hope.)
> >
> > Any suggestions?
> >
> > Thanks in advance,
> >
> > Peter Ivanyi
> > _________________________________________________
> >  For list-related administrative tasks:
> >  http://lists.racket-lang.org/listinfo/users
> >
> 
> _________________________________________________
>   For list-related administrative tasks:
>   http://lists.racket-lang.org/listinfo/users
>