No subject
From: ()
Date: Mon Dec 3 19:58:15 EST 2012 |
|
three categories:<br>
<br>
=C2=A0* Package-system design<br>
<br>
=C2=A0* Repository organization<br>
<br>
=C2=A0* Concerns that a more distributed ecosystem means a less unified one=
<br>
<br>
Let's take them one at a time.<br>
<br>
** Package-system design<br>
<br>
We all appreciate the work that Jay did to design the package<br>
system. I hear lingering concern about the design, including its<br>
limited support for versioning (just dependency checks), the fact that<br>
the package system is outside the module system (no built-in<br>
auto-download of packages, although a tool like DrRacket can suggest<br>
package installs in response to missing-library exceptions), its<br>
stance on conflicts (simply disallowed), and its flat namespace (which<br>
could make conflicts more frequent).<br>
<br>
On some of the points, I think reasonable people will disagree. We've<b=
r>
had a years-long discussion, and we've been paying attention to<br>
precedents. We've explored some nearby alternatives to the current<br>
design (I'm thinking of single-collection versus multi-collection<br>
packages). I think we've gotten as close to consensus as possible.<br>
<br>
** Repository organization<br>
<br>
As we try to split the Racket repository into packages, the questions<br>
concern how finely to split the repository and how to eventually<br>
allocate packages to source-code repositories.<br>
<br>
I think the initial split of the Racket repository went more smoothly<br>
than anyone expected. It was fairly easy, for example, to extract a<br>
relatively small core to run `raco pkg', or to draw a line between<br>
DrRacket and the teaching languages. I chalk that up to general<br>
competence among the Racket implementors: big systems must be<br>
developed in layers, whether the layers are declared or not.<br>
<br>
In fact, it has worked out so well that the splitting of Racket into<br>
packages has taken a more aggressive form than I expected. At this<br>
point, we've split the Racket repository into 137(!) packages, and<br>
that number is still growing. Two of us tried to make a coarser split,<br>
and it didn't feel right. Others have since started shuffling packages<=
br>
and continue to split things further. We seem to really like declaring<br>
dependencies and reducing unrequested functionality.<br>
<br>
Given that packages are going to be split finely, the question of<br>
allocating packages to repositories is less straightforward. We've<br>
concluded that "scribble-lib" and "scribble-doc" are go=
od to have<br>
separate as separate packages, but we don't want Scribble's<br>
implementation and its documentation to end up in a separate<br>
source-code repositories. At the same time, putting everything in one<br>
big repository is intractable, at least at the point where we want<br>
packages downloaded directly from a repository. (A package can be a<br>
subdirectory of a repository, but the package manager has to download<br>
a tarball of the entire tree to extract the subdirectory.) So, under<br>
"pkgs", we have an extra layer in the directory hierarchy to refl=
ect<br>
an intended organization into repositories. Using a layer of<br>
directories is consistent with git submodules, if we choose to go that<br>
way.<br>
<br>
The fact that many of us have tried and arrived at the same conclusion<br>
on granularity gives me confidence that it's a reasonable conclusion,<b=
r>
but the current Racket repository organization really does feel<br>
complex. For example, the core of `raco setup' is<br>
<br>
=C2=A0 =C2=A0racket/lib/collects/setup/setup-unit.rkt<br>
<br>
while the Scribble part of `raco setup' is in<br>
<br>
=C2=A0 =C2=A0pkgs/racket-pkgs/racket-index/setup/scribble.rkt<br>
<br>
Those paths reflect that `raco setup' is mostly core functionality,<br>
but you don't get documentation setup until you install the<br>
"racket-index" package, which is currently grouped with other<br>
almost-core packages.<br>
<br>
This example also illustrates how the current organization relies on<br>
collection splicing in a big way. In the long run, not many<br>
collections are going to be spliced so much as, say, "racket" and=
<br>
"data", but splicing two or three times to separate modules,<br>
documentation, and tests may turn out to be common.<br>
<br>
And then there's<br>
<br>
=C2=A0 =C2=A0pkgs/drracket-pkgs/drracket/drracket/drracket.rkt<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^ =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0^ =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0^<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 repo =C2=A0 =C2=A0 =C2=A0package =
=C2=A0collection =C2=A0module<br>
<br>
Every layer before a "/" has multiple descendents, so they layers=
are<br>
not trivially collapsed. If you just look at the path, it seems<br>
crazy. But if you're expecting <repo>/<package>/<collect=
ion>/<module>,<br>
then hopefully it seems reasonable.<br>
<br>
In short, the current layout is driven by three factors: a bias toward<br>
fine-grained packages, a sense that it's good to reflect layers and<br>
dependencies via separate filesystem directories, and some constraints<br>
on how directories relate to git repositories. Unless we change those<br>
driving factors, I don't see us arriving at a simpler organization.<br>
<br>
** Distributed versus unified ecosystem<br>
<br>
While less prominent than the other categories, I'm also hearing some<b=
r>
concern that splitting up the Racket repository and reorganizing<br>
various pieces of infrastructure will lead to a less unified system<br>
--- or even a less unified community.<br>
<br>
Moving our products and infrastructure into a more distributed form is<br>
one of my main goals, but I don't think that "distributed" ha=
s to mean<br>
"fragmented". It seems to me that the more distributed we are abl=
e to<br>
make our world (the Internet, git, etc.), the more closely we are able<br>
to work together. The math behind that effect eludes me, but I believe<br>
in it, anyway.<br>
<br>
At the same time, the sudden emphasis on reorganizing the Racket<br>
repository could also give the impression that the new package system<br>
is primarily about distributing Racket, and not about "third-party&quo=
t;<br>
libraries and packages. I think we're trying to make our as much code<b=
r>
as possible treated as "third-party", and thus ensure that all pa=
rties<br>
are well supported.<br>
<br>
<br>
Why Aren't We There, Yet?<br>
-------------------------<br>
<br>
We're hardly the first to design a package system or apply it to a big<=
br>
system, and I can't shake the sense most of the time that we're jus=
t<br>
reinventing the wheel. Along those lines, implementing the mechanics<br>
of the package system has been suspiciously difficult.<br>
<br>
I hope that part of the reason is our commitment to documentation ---<br>
that it exists, that it builds reliably, that it's richly formatted,<br=
>
and that it is pervasively cross-referenced and hyperlinked. I don't<br=
>
think that any package system delivers documentation that's anything<br=
>
like ours.<br>
<br>
Could it also be an unusual commitment to relative paths, especially<br>
when distribution pre-built items? A lot of problems go away if you<br>
know that the library is going to be in "/usr/local/lib".<br>
<br>
Surely part of it is trying to make `raco setup' fast for installing<br=
>
packages. It's complex and fragile to performing an incremental<br>
computation based on changes inferred from filesystem state.<br>
<br>
Bootstrapping, at least, is known to be tricky. The Racket compiler<br>
isn't written in Racket, yet, but the installer-creator installs<br>
Racket packages to create a local installation that is used to set up<br>
packages on a remote installation that runs a Racket script to build<br>
an installer. It took many days to make that work and make it<br>
configurable.<br>
<br>
On the plus side, `raco setup' can usefully check package dependencies<=
br>
and sort them into "build-time" and "run-time" dependen=
cies, even for<br>
documentation links, and that checking was relatively easy to<br>
implement. Since module collection references can be synthesized at<br>
run time, there's no way to completely check dependencies statically,<b=
r>
but I think we may end up with something that's more reliable and<br>
complete than checking in other package systems. If so, maybe that<br>
helps explain why it was hard.<br>
<br>
_________________________<br>
=C2=A0 Racket Developers list:<br>
=C2=A0 <a href=3D"http://lists.racket-lang.org/dev" target=3D"_blank">http:=
//lists.racket-lang.org/dev</a><br>
</blockquote></div><br></div>
--089e0149c8007eb4c804e1781315--
![[logo]](/logo.png){kind=logo}