[plt-dev] setup-plt parallelization progress

Howdy,

I've had a student (Petey Aldous) work on parallelizing setup-plt.

Here's how it works:

- Add locks to the compilation manager so that two processes don't
compile at the same time
- Start multiple instances of setup-plt after the first instance gets
through the bootstrapping phase
- Each instance visits the collections in a random order to
probabilistically get work done without stepping on feet.

Here's the results of a few runs:

Processes - Time
1 - 22m
2 - 10m
3 - 11m
4 - 13m
5 - 17m

We're looking at smarter ways that just random collection orders to
make it better. We have lots of ideas but some are more expedient than
others given the reality of the information available from the
compilation manager. If you have anything you think we haven't thought
of, we'd be happy to know. Also, right now we don't do anything to
parallelize the doc building or indexing.

It's actually quite nice that 2 processes works the best, because dual
cores seem like the most common hardware these days. Hopefully we can
do better though.

Jay

-- 
Jay McCarthy <jay at cs.byu.edu>
Assistant Professor / Brigham Young University
http://teammccarthy.org/jay

"The glory of God is Intelligence" - D&C 93