[racket-dev] updated proposal for moving to packages

I've updated the package experiment again:

 https://github.com/mflatt/racket/tree/pkg


State of the Proposal
---------------------

Based on discussion of the initial proposal,

 http://lists.racket-lang.org/dev/archive/2013-May/012364.html

I'm revising the proposal as follows:

 * The details of the repository organization (including where to split
   repositories) should be different.

   As described in next section of this message, the experimental
   repository represents a revised proposal, but there's plenty of room
   for further refinement.

 * The main distribution will not be in binary form; we keep source
   alongside build bytecode and documentation, as before.

   Creating a package catalog that provides binary versions of packages
   (for use in constrained environments) remains a clear possibility,
   but that's now on the backburner.

 * The process for building installers is as originally proposed, but
   substituting built packages (i.e., source plus binary) instead of
   binary packages.

   The big pieces that were missing are now implemented.


Current Proposed Repository Organization
----------------------------------------

In the latest version of the experimental repository, packages are all
in a "pkgs" subdirectory. Each directory in "pkgs" is intended as a
candidate for movement into a separate repository, but each directory
in "pkgs" is either

  * a package directory (single- or multi-collection); or 

  * a directory (named "...-pkgs") containing package directories,
    where the packages in each directory are intended to be together in
    one repository.

It doesn't have to be just two levels: any number of grouping-directory
layers is fine, and the "separate repository" contour can be anywhere
among the grouping layers (as long as each repository is a directory,
and as long as each package is wholly within some repository).


The line between grouping directories and package directories in "pkgs"
is determined by the presence of an "info.rkt" file to indicate a
package. That is, even though a package is not in general required to
have an "info.rkt", each package in "pkgs" is required to have an
"info.rkt" so that it can be recognized as a package directory.

The current "...-pkgs" name for each grouping directory is just a
convention.


The package groupings themselves are not much improved from before ---
lots of work to do there, but that's work that would happen by everyone
if and when we go this direction.


New targets the in Makefile can install package links in the "racket"
directory, so that (on Unix, at least)

 make core
 make pkg-links
 racket/bin/raco setup

is essentially the same as the current

 cd src
 mkdir build
 cd build
 ../configure
 make
 make install

except that the resulting build is in a "racket" subdirectory.

As long as we don't actually create separate repositories, you can
edit/move libraries and collections in "racket/lib/collects" and
"pkgs", push and pull via `git', and rebuild via `racket/bin/raco
setup'. If we do split into separate repositories, then I think it
could be much the same, but with git submodules.

See "INSTALL.txt" for more information.


Your Turn
---------

If we pursue the path demonstrated in this experimental repository,
then we have several infrastructure details to work out yet, to say
nothing of finishing up the organization of our collections and
libraries into packages.

I think we're at the point, though, for you to assess whether this is
the right direction. If it looks like a good direction, then the
follow-up question is how fast to move.

Some possible conclusions:

 1. This is the wrong way.

 2. This is plausible, but the details are not right or not clear; we
    should stick with our current repository structure for at least one
    more release and consider switching after that.

 3. This will work, and we should switch right away and start sorting
    out the details together --- but we should not actually break out
    into separate repositories until after a release or so.

 4. This will work, we should try to switch right away --- AND we should
    split the repository as soon as possible.

Since it's my proposal, hopefully it's clear that I'm in camp 3.

Opinions?