[racket-dev] from packages to installers

 <a class="LargeFriendlyLetters" href="#end">
  DON'T PANIC
 </a>

I've pushed a new version of the package experiment:

 https://github.com/mflatt/racket/tree/pkg

You won't see much difference in terms of package splitting. I haven't
even renamed "-docs" packages to "-doc" packages, yet.

This new version is about creating installers. Normally, the creation
of installers is not of much interest to this list, but one of the
ideas behind the package system is that it should be easy to build a
Racket distribution by combining the core with a selected set of
packages. I think the details of that process both inform and
illustrate details about the package system.


Here are some installers that I built based on the "drracket" package
(includes DrRacket, doesn't include teaching languages, etc.):

 http://www.cs.utah.edu/~mflatt/tmp/installers/

There are still some issues to sort out --- including a missing
"README.txt" and a missing rule to auto-launch "DrRacket.exe" from the
Windows installer --- but I think the installers are close to right.


You can build installers yourself, as long as you have a machine for
each platform where you want an installer. One machine builds the
packages from source (much like the current build process) and then
acts as a server. All machines act as clients to build
platform-specific installers. Each client build is relatively fast
(about 5 minutes): build the core, then download and install pre-built
packages.

See "INSTALL.txt" and "Makefile" for more information, but the process
boils down to this:

 1. On the server machine:
      make server PKGS="..."

 2. On each client machine:
      make client SERVER=... PKGS="..."
    or (on Windows)
      nmake win32-client SERVER=... PKGS="..."


One thing the Makefile illustrates is how to get going from scratch at
the level of Racket sources. Native libraries, such as Cairo, have to
be built and available already, but the process to turn those into
packages is part of the Makefile. The process of consing up a package
catalog from local package directories is also part of the Makefile.

More precisely, all of that happens with the default "local" mode of
the Makefile. You can set `SRC_CATALOG' to a URL for a catalog server,
and then packages are drawn from that catalog instead of local sources.
A non-"local" `SRC_CATALOG' will be a better default after we set up a
catalog server for all of the base packages, where each entry in the
catalog will point to a GitHub repository.

Either way, after a catalog of source packages is available, then the
package system is used to create "built" packages (source plus bytecode
and docs). Finally, a new catalog is created for the built packages so
that they can be served to installer builds.


The Makefile includes `binary-catalog' and `binary-catalog-server'
targets that create and serve a catalog of binary packages. It doesn't
yet work to create installers from binary packages; the first issue is
that `raco setup' wants "installer.rkt" files to exist in source form,
and there are likely a dozen other little problems. I've left
`binary-catalog-server' in place, though, to illustrate how things are
supposed to work for binary packages: binary packages are extracted
automatically from built packages, and creating a binary build is just
a matter of using binary packages.


If you're inclined to look closely at the generated installers or at
the "pkg" branch, you'll see that I made some directory-structure
changes inside the "racket". Most notably, "collects" moved inside
"lib", while configuration items moved from "collects" and "lib" into
"etc". The idea behind moving "collects" to "lib" is to put it next to
"pkgs" and in a more consistent place with respect to Unix conventions.
The move of configuration information to "etc" is also to be more
consistent with Unix conventions, but also to get platform- or
installation-specific information out of "collects", which simplifies
installer creation. All of that is up for discussion, of course.

 <a name="end"/>

I know this all looks complex, the tools are still rough, and we still
have many things about the package system to explain and discuss. I
think we're on the path to a place that will work well, though.

I also think the way to get there is to iterate on the whole path ---
from the simplest `raco pkg' command to the full snapshot and release
process. So, if some earlier step seems terribly wrong to you, rest
assured that an experiment in installer builds doesn't mean that we've
committed to all previous steps and there's no going back. It only
means that we're taking a bunch of steps ahead to make sure that we
know where we're going.