[racket-dev] Planet and Packages (was Re: PLaneT(2): Single vs multi-collection packages)

Just my $0.01: it seems to me that the direction we're headed with the pkg
system would make a great platform for exploring all kinds of practical
security issues surrounding code and distribution (building a very nice
research program, I mean). We already have good sandboxing facilities and
this lets us explore how to best set those up and how to build web-of-trust
type stuff on top of that.

Robby


On Thu, Jul 4, 2013 at 1:02 PM, Sam Tobin-Hochstadt <samth at ccs.neu.edu>wrote:

> Neil,
>
> You clearly put a bunch of thought into this email, so I think it
> needs a response. I've changed the subject to put this in a new
> thread.
>
> On Fri, Jun 14, 2013 at 10:48 PM, Neil Van Dyke <neil at neilvandyke.org>
> wrote:
>
> > For all of my packages, as well as any package I can imagine, I think
> that
> > the original PLaneT got many things right or close to right:
>
> Some of these features the new package system has:
>
> > * Files from a package-version end up grouped together in the directory
> > structure, specific to that package-version, and certainly not mixed into
> > directories with files from package-versions of different package-names.
>
> While collections now "splice", this is still true.  When I install a
> package, it creates a new directory and everything goes there. For
> example, 'raco pkg install -i gcstats' produces the directory
> `/home/samth/sw/plt/racket/lib/pkgs/gcstats`.
>
> > * Flat namespace (let's ignore the PLaneT package-owner part for now),
> > without attempt to name packages according to some topical ontology.
>
> This is still true.
>
> > * Metadata in "info.rkt".
>
> Also still true.
>
> > * Some kind of unique package-name controlled by a developer.
>
> This is true to the same degree it was with Planet. That is, if you
> want to register your package at `pkg.racket-lang.org`, then you have
> to have a unique name. Otherwise, you can locally install whatever you
> want and give it a name that conflicts, or run your own server with
> similar consequences.
>
> Some things are different:
>
> > * Multiple versions for a given package-name (I'll call them
> > package-versions in this email) can be installed, and there is some
> version
> > selection mechanism.
>
> This is no longer true, and this is where the new system differs from
> other systems such as 'npm' as well. I think this was, in fact, a
> major _problem_ with Planet, and one that I'm personally glad the new
> system doesn't have.  I think there are a few problems with what
> Planet did:
>
> 1. Many packages can't really work with multiple versions in the same
> program, whether because of generative structs, or files used, or many
> other issues.
>
> 2. The way Planet versions were selected was mostly hardwired, and
> everything was hard to upgrade.
>
> There were other problems, too, but these are the ones I remember
> being most significant.
>
> However, I _think_ (the documentation is pretty terse) that the
> `--scope-dir` option to `raco pkg install` can help you simulate a
> more npm-like workflow.
>
> > I was expecting to use this original PLaneT as a starting point, and
> evolve
> > it in ways like the following...
> >
> > * In addition to the "(planet ...)" require-specs, package-versions also
> can
> > come from "http:", "https:", and "git:" URLs.  ("github:" would also be
> OK.)
> > Each such URL would identify trees or a tarball.  Then we see how people
> > choose the PLaneT server vs. HTTP vs. Git over time.
>
> Package dependencies can be specified with URLs, which can specify
> remote directories, or remote ZIP (or tar etc) files, or local files
> or directories, or GitHub repositories.  I hope to add support for
> arbitrary git remotes soon (if the `git` binary is available).
>
> > * Maybe improve the version-selection and compatibility support.
> > Investigate whether it's worthwhile to separate out the
> > backward-compatibility information from the static package-version
> > distribution (and especially from the version number), or whether in
> > practice there are simpler ways that are satisfactory.
>
> I think decoupling the source code from the version specification is
> exactly the improvement wanted here.
>
> > * Maybe a facility in "info.rkt" to provide aliases for require specs.
> > Otherwise, people writing nontrivial multi-file code that uses other
> > packages from PLaneT/whatever end up having to make wrapper modules so
> that
> > we don't goof our require-specs and accidentally depend on multiple
> > package-versions for the same package-name.  Note that, with URLs, these
> > aliases *might* be the only actual package-name construct in the HTTP/Git
> > system as distinct from URL similarities of package-versions.  This info
> > might be implicit in a package-version's "info.rkt"'s reference to a
> > previous package-version, perhaps coming from an assertion of
> compatibility
> > info.  This might be simpler than it might sound, but it has some
> > interesting implications, including for forking and web-of-trust.)
>
> Again, this is addressed by no longer having Planet's treatment of
> versions and require specs.
>
> > * Simple web-of-trust package-version public-key signing of
> package-versions
> > (e.g., URLs plus hashes of contents), to start with, perhaps initially
> with
> > only centralized repository for signatures.  Soon build distributed
> > web-of-trust, plus multiple repositories so organizations have option to
> > keep their signatures separate.  Build mechanisms atop that, including
> > advancing the state of the art.
>
> I think package signing is something we'll eventually need, but I also
> don't think it's on the critical path.  It's also possible to add to
> the package system in the future.
>
>
> > * Automate and simplify releasing in general.  With PLaneT, it's been
> > not-unusual for even core Racket developers to avoid releasing some
> add-on
> > code to PLaneT, perhaps because the clerical stuff was a headache.  For
> the
> > old PLaneT, I was simplifying this with McFly, but with new a package
> > mechanism, I would start with that and then ask what clerical parts still
> > need help.  (For example, if doing development in an SCM repository
> that's
> > accessed directly via require-specs, then releasing a package-version
> might
> > consist mainly of adding a tag/label.  planet-lang.org's directory might
> > even update automatically, given info about a previously-released
> > package-version of the same package-name.)
>
> The new system is much much much easier to release packages for.  For
> starters, this is a working package:
>
>     https://github.com/samth/add-blaster
>
> You can install it with `raco pkg install
> github://github.com/samth/add-blaster/master` (I hope we can make that
> command line shorter).
>
> Second, you can add this to `pkg.racket-lang.org` with a few clicks.
> This can be automated once pkg.racket-lang.org has an API. Further, it
> automatically updates when the GitHub repository changes.
>
> > * Use submodule support to support single-file packages, at least for the
> > HTTP/Git package-versions.  "(module+ info ...)".  It seems from Emacs
> > history that some people really like the single-file module, it lowers
> > barriers, and now submodules give us an easy way to finally do it.
>
> This is potentially an interesting idea, although the info.rkt file
> format is quite restricted to enable reading it without running
> arbitrary code, and this might be hard to integrate with submodules.
>
> > * Do whatever is necessary to avoid blocking the program for
> > few/several-minutes while documentation is reformatted, when requiring an
> > uncached package-version.  Maybe even moving it to an async process
> that's
> > run when idle (Unix "nice"?) would work.
>
> First, the split between documentation and code for many packages will
> help here. Second, I think this is a lower-level issue than the
> package system -- in-tree builds have this issue too.
>
> Also, what program is being blocked here?  The installation process?
> DrRacket?
>
> > * To put it vaguely: keep things simple in most cases, but don't
> dumb-down
> > in practically restrictive ways, and keep an eye out for places to
> > experiment with potential big wins for immediate practice or research.
>  Some
> > things I just mentioned above would surely need refinement/exclusion
> based
> > on this principle.  For another example, I heard some comments at one
> point
> > about a package name being an interface, and multiple sources being able
> to
> > provide implementations matching that interface.  I don't know the
> current
> > plans for that, but I wouldn't make any special mechanism for that.  For
> > another example, don't try to dumb-down package-names, as if the first
> > person to make a package concerning the generic concept "foo" has the
> > be-all-end-all package for all things "foo".
>
> Package names aren't interfaces, but modules names (like `data/list`)
> can be thought of that way, and multiple packages can provide the same
> one.  However, this kind of overlap is discouraged.
>
> I'm not sure what would be a less 'dumbed-down' but still flat
> namespace. Do you have example suggestions?  The flat namespace has
> worked well with other languages with huge numbers of packages.
>
> > * Some Web directory of software on "planet-lang.org" (with JSON dump
> and
> > maybe query), which includes both PLaneT packages together with the
> HTTP/Git
> > packages that people have chosen to list in the directory.  It's a "this
> is
> > all the Racket packages we know about, and probably easier to find via
> > search here than via Google."  (Eventually, this would be hooked up to
> the
> > site-wide search feature for "planet-lang.org", together with other
> > categories of other searchable Racket-related info that we identify.
>  Then
> > DrRacket search could be hooked up to that.)
>
> Currently, `pkg.racket-lang.org` doesn't include the contents of
> `planet-compat.racket-lang.org`, but I suppose it could. It's already
> searchable.
>
> I hope that's helpful for clarifying some of the current design, and
> it's great to have feedback from a heavy user of Planet.
>
> Sam
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.racket-lang.org/dev/archive/attachments/20130705/d00d845d/attachment.html>