<div dir="ltr">Thanks! (For one, I found the "From Back There to Here" section particularly helpful.) </div><div class="gmail_extra"> <div class="gmail_quote">On Sat, Jul 13, 2013 at 1:56 PM, Matthew Flatt <<a href="mailto:mflatt@cs.utah.edu" target="_blank">mflatt@cs.utah.edu</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Here's a big-picture update of where we are in the new package system and the conversion of the Racket distribution to use packages. This message covers - how I see things working after the package system and reorganization is done, and a report on what pieces are still missing to reach that vision; - a look at how we got to our current design/reorganization choices and whether we're choosing the right place; and - speculation on why the package changes have been so difficult to implement. All of that makes it a long message (sorry!), but I hope this message is useful to bring us more in sync. A Package-Based Racket ---------------------- Let's take a look at how you'll do various things in the new package-based Racket world. (There's no new information here, and parts marked with "[guess]" are especially speculative. Still, some details may be clearer than in earlier accounts, now that much of it is implemented, and I think a comprehensive review may be useful.) ** Downloading release installers from PLT The "<a href="http://www.racket-lang.org" target="_blank">www.racket-lang.org</a>" site's big blue button will provide the same installers that it does now, at least by default. That is, the content provided by the installer --- DrRacket, teaching languages, etc. --- will be pretty much the same as now. The blue button might also provide the option of "Minimal Racket" installers, which gives you something that's a small as we can make it and still provides command-line `raco pkg'. ** Downloading installers from other distributors There are all sorts of reasons that the "main distribution" from PLT might not fit the needs of some group. Maybe the release cycle is too long or at the wrong time. Maybe it includes much too much, much too little, or almost the right amount but missing a crucial package. Maybe the group wants something almost minimal, but still with a graphical package manager. Maybe some group uses a platform for which PLT does not provide an installer. For many of those groups, using a "Minimal Racket" installer plus selective package installations will do the trick. For others, creating a special set of installers might be worthwhile, but there are too many reasons and too many permutations for PLT to provide installers that cover all of them. Fortunately, anyone can build a set of installers and put them on a web page, and we make it as easy as possible to build a set of installers that start with a given set of packages. PLT could host a web page or wiki that points to other distributors. PLT might even be able to provide an automated service that generates a set of installers for a basic set of platforms. ** Compiling a release from source In addition to installers, a download site can provide a source-code option (not specific to any platform, unlike the current source packages), which would mainly be used for building Racket on additional platforms. This option is mostly a snapshot of the source-code repository for the core, but it includes a pre-built "collects" tree (see "technical detail", below) and a default configuration that points back to the distributor's site for pre-built packages. ** Adding or upgrading supported packages In much the same way that you can easily install a set of supported packages on your current OS, you'll be able to easily install a set of packages that are supported by your distributor. Those packages are pre-built, so they install quickly, along with any included documentation. Depending on the distributor and installer, packages might be downloaded and installed in "binary" form, which means that tests and source code (for libraries and documentation) are omitted from the package. PLT seems unlikely to provide such installers in the near future. The default package scope configured by a distribution tends to be "user", which means that packages are installed in a user-specific location. Package updates can be made available by distributors for whatever reason and on whatever timetable see they fit. If your distribution is from PLT, then the supported packages are called "ring-0" packages. Ring-0 packages include contributions from third parties (i.e., not just packages implemented by PLT) that are vetted and regularly tested by PLT. [Guess:] The "Racket" and "Minimal Racket" distributions might point to different pre-built package catalogs. Possibly, the "Racket" catalog never updates packages that were included in the installer (on the grounds that the user may not have write permission to the install), while the "Minimal Racket" catalog includes more frequent updates for bug fixes (on the grounds that the user can update any installed package). A distributor doesn't necessarily have to provide its own package catalog. It can instead supply an installer that works with packages as served by some other distributor's catalog, such as PLT's catalog. (See "technical detail" below.) A user can also redirect `raco pkg' to a different catalog server, instead of using the configuration that was supplied by the installer. Binary, pre-built, and source variants of a package can be "updated" to each other in any direction. ** Adding or upgrading other packages An installer-provided configuration will normally point to a catalog of packages that are not specifically supported by the distributor but are still readily available --- probably mostly in source form and directly pulled from a git repository. In particular, "<a href="http://pkg.racket-lang.org" target="_blank">pkg.racket-lang.org</a>" provides packages in source form. ** Reading documentation A distribution site provides online documentation (including all supported packages) alongside installers and packages. Many installers and packages include documentation to be installed on a user's machine, but there are some packages that provide libraries without documentation. For example, "gui-lib" provides GUI libraries without local documentation, while "gui" combines "gui-lib" local documentation and the libraries. Sometimes, documentation that is installed locally will still refer to documentation that is not downloaded. Such links are directed back to the distributor's site. That situation won't happen often for pre-built packages, because links that go to other packages will tend to go to packages that are dependencies. It will happen more for binary packages, because the dependency can be build-time only. ** Creating new packages A minimal package is a directory. So, let's suppose that you have some modules in a directory that you want to turn into a package. Suppose that your directory is called "potato", and it has module a file "eat.rkt". Turn your directory into a locally installed package with raco pkg install --link potato Then, you can use "eat.rkt" with (require potato/eat) To give your package to someone else, you could zip up the "potato" directory as "potato.zip", and the other person would install with raco pkg install potato.zip Note that you can use any zip archiving tool, or you can use rack pkg create --form-install potato to create the ".zip" file, which has the advantage that directories like "compiled" and ".git" are omitted. Even better, maybe your directory is already on GitHub at "<a href="http://github.com/idaho/potato" target="_blank">http://github.com/idaho/potato</a>". Then, others can install your package with raco pkg install github://<a href="http://github.com/idaho/potato/master" target="_blank">github.com/idaho/potato/master</a> If you push changes to your GitHub repository, others can get them with raco pkg update potato If you're ready for the world to use your package, then go to "<a href="http://pkg.racket-lang.org" target="_blank">pkg.racket-lang.org</a>" and point the package name "potato" at your GitHub repository. Then, not only will others know about your package, they'll be able to install it with raco pkg install potato Finally, if you'd like PLT to include your package as a pre-built package with each snapshot and release, then go back to "<a href="http://pkg-racket-lang.org" target="_blank">pkg-racket-lang.org</a>" and request ring-0 status for the package. Ring-0 status may require a few bureaucratic improvements to your package, such as including an "info.rkt" file if you don't have one already, because those details are needed to keep your package in working order. ** Using the cutting edge PLT provides one or more snapshot sites that work the same as the release site, except that each snapshot's catalog expires after a few days. When that catalog goes away, you can continue to use the snapshot, but you'll have to get packages and updates via source. ** Using the bleeding edge A user who wants to work with the minute-by-minute latest can start by cloning the core Racket git repository, `configure', `make', and `make install' to get a Minimum Racket build. Then, start installing packages with `raco pkg'. The default package catalog in built-from-source Racket is "<a href="http://pkg.racket-lang.org" target="_blank">pkg.racket-lang.org</a>", which means that you get all packages in source form from various git repositories, including for PLT-maintained packages. The default package scope is "installation". If you run `raco pkg update -a', then you likely get updates and trigger many compiles. Eventually, an update will fail, because your core Racket version is too old, and you'll need to `git pull', `configure', `make', and `make install' --- if you haven't been doing that, anyway. Since packages were added with installation-wide scope, `make install' rebuilds your previously installed packages, too. ** Using the bleeding edge as a PLT developer As a convenience to PLT developers, who tend to work on a particular set of packages, there is an alternate way of working on the bleeding edge (which anyone can use, if they prefer). [Guess #1:] Instead of cloning the core Racket repo, clone a "main distribution" repo that has the core Racket repo as a submodule, plus git submodules for each of the packages that are dependencies of "main-distribution". In other words, you get something that looks like the current Racket repo, but that uses git submodules. [Guess #2:] Instead of cloning the core Racket repo from GitHub, you clone from the "main distribution" repository, just like now. In addition to being mirrored to GitHub directly, individual parts of the "main distribution" repo are mirrored as GitHub repositories, and the mirrors are the ones that "<a href="http://pkg.racket-lang.org" target="_blank">pkg.racket-lang.org</a>" references. GitHub repositories that correspond to packages (submodules in guess #1, mirrored subtrees in guess #2) are registered with "<a href="http://pkg.racket-lang.org" target="_blank">pkg.racket-lang.org</a>", which is how users on the bleeding-edge might normally get the packages. ** Becoming a distributor If you want to create installers like PLT's, then it's simplest to clone the git repo like a PLT developer, and then use `make installers'. Alternatively, you can use `make installers-from-catalog' to create a set of installers based on packages pulled from a specified catalog. Either way, if you want to piggy-back on some other installer's set of pre-built packages, then configuration options and/or makefile targets to do that. (This is more sketchy; see below.) ** Taking your own snapshot of Racket and packages: Sometimes, you don't need to build installers, but you'd still like a snapshot of the current Racket core and package. You might want to edit the snapshot to upgrade some packages while keeping others the same. The `raco pkg catalog-copy' command is one of many tools to manipulate catalog servers. For packages that are mapped to GitHub repositories, merely copying a catalog doesn't archive the code, but it archives a particular commit id. It's always possible to grab a copy of a package repository and reference the copy from a catalog. A Technical Detail ------------------ Starting from scratch twice with the same Racket sources does not lead to compatible pre-built packages, unfortunarely, because bytecode files are generated deterministically. Maybe we'll be able to fix that, one day. Meanwhile, pre-built packages depend on a particular build of the libraries in "collects", as well as a particular build of any dependencies. So, if a distributor wants to enable other distributors that use the same catalog of pre-built packages, the distributor must serve a "collects" tarball, too. Providing the "collects" will be built into the snapshot support.