[racket] Worried about the new package manager not storing each version of a package

To the extent this is relying on package providers to do the right
thing -- relying somewhat on convention and best practices:

I think now would be a great time for a blog post (and maybe a
RacketCon presentation) stating the best practices.

Including things that seem obvious to core Racket devs. And including
things that Jay might understandably want to be _optional_, and it's
great that the system is flexible enough to allow more than just
what's strongly recommend, but it's OK to strongly recommend.

So maybe (IIUC) the recommended practices would include:

1. Host on GitHub.com.
2. Use Semantic Versioning (although Matthew IIRC it would need to
support versions like "2.0.0" and "0.0.1" that it currently rejects?).
3. Once you reach 1.0.0, you _must_ start using Git tags for each new
version (before then maybe it's not required).

If I don't have that quite right... well then I think that goes to my
point that we need something clearer. :) Whatever the guidelines, just
make it super clear and checklist-oriented. That way, someone with a
useful package for domain X can know exactly what's best, and do it
without needing to understand the domain and history of package
management so deeply. That is one thing I think that Planet 1 got
right, and would be great to preserve, even if via different means.

The new pkg mgr is technically still in beta, but it's getting used
much more recently. As a result I think this kind of best practices
guidance feels like it's "suddenly" necessary in a way it maybe didn't
seem even just a month or two ago? At least that's my impression.

p.s. Maybe Lawrence you'd still prefer a system that forces packages
to do the right thing, but, you'd at least be more comfortable where
the right thing is crystal clear to package providers and you can see
if they're doing it or not?


On Tue, Aug 27, 2013 at 8:55 AM, Matthew Flatt <mflatt at cs.utah.edu> wrote:
> Hi Lorry,
>
> Thanks for your comments.
>
> For what it's worth, I think you're worried about problems that will
> have simple solutions in practice. For example, I don't think you
> should rely a package whose author isn't invested enough to at least
> put it on GitHub or otherwise archive old versions.
>
> If one of your scenarios turns out to be a problem in practice,
> however, then you should expect that we will improve the package system
> to address the problem. That is, don't expect the package system to be
> perfect, but do expect us to respond to concrete problems.
>
> That opinion points to my general answer about relying on Racket for a
> commercial enterprise:
>
>  * You *cannot* rely on on Racket version X.Y.Z as the prefect design.
>
>  * You *can* rely on Racket to keep changing and keep improving. It's
>    natural and appropriate to worry about change and/or new designs,
>    but we are committed to change and improvement, anyway.
>
>  * You *can* rely on Racket to provide backward compatibility and/or a
>    migration path so that your programs keep working as Racket
>    improves.
>
> As a case in point, this thread is about the new package system. We
> have an old package system (PLaneT), and it is still supported. We're
> building on the experience of the old system to solve some of its
> problems and to create new opportunities. I have various reasons to
> believe that the new package system is a step in the right direction
> (not the least of which is the fact that DrRacket can now be a
> package), but I'm even more sure that's the design is as good as we can
> do without further practical experience.
>
> Past performance is no guarantee of future results, as they say --- but
> maybe past performance is the best indicator available, anyway. Racket
> has been around for a while, so you can see how we work, and that
> should help you decide whether Racket is stable enough to build on.
>
> Matthew
>
> At Tue, 27 Aug 2013 07:37:58 +0100, Lawrence Woodman wrote:
>> On 08/26/13 14:27, Matthew Flatt wrote:
>>
>> > A _package name_ is something like "mischief", which you use for
>> > installing and declaring dependencies. A _package implementation_ is
>> > something that you download from, say,
>> >
>> >   https://github.com/carl-eastlund/mischief/tarball/
>> fe7119517a4dcd3f5c509735a7b5a5664151c14f
>> >
>> > Note that a package implementation in this sense corresponds to
>> > specific revision of a pile of code, such as a particular commit in a
>> > git repository. The package manager includes the concepts of a "package
>> > source" and a "checksum", which together tell you how to get a package
>> > implementation. (That implementation may have its own version number,
>> > but such a version number is in principle orthogonal to the package
>> > implementation's checksum.)
>> >
>> > The mapping from a package name to a package implementation is provided
>> > by a "catalog". PLT provides a catalog server at pkg.racket-lang.org,
>> > but you can make your own catalog (as a server or on a local
>> > filesystem), and so you can precisely control the mapping from package
>> > names to packages.
>> >
>> > Furthermore, we've added tools to `raco pkg' to make it easier to
>> > manage catalogs. For example, if you want to take a snapshot of the
>> > current pkg.racket-lang.org and use that from now on (so that the
>> > mapping from package names to packages doesn't change), use these
>> > commands:
>> >
>> >   raco pkg catalog-copyhttps://pkg.racket-lang.org  /full/path/to/catalog/
>> >   raco pkg config --set catalogsfile:///full/path/to/catalog/
>> >
>> > You can modify the files generated at "/full/path/to/catalog/" by hand
>> > in a fairly obvious way. Or you can upload the directory to a
>> > file-serving HTTP site and point installations to the uploaded
>> > directory as the catalog. There's also an option to use an SQLite
>> > database as the format for a catalog, which is a better option if you
>> > want to modify the catalog programmatically via `pkg/db', but an SQLite
>> > database is less easy to use from a file-serving HTTP site.
>> >
>> > In particular, I can imagine having a project whose source code
>> > includes a package catalog. To upgrade a particular package, I'd change
>> > the catalog and `raco pkg update'. When I commit a particular revision
>> > of the source code to a git repository, the package catalog is saved;
>> > then, I can roll pack the project (including its references to specific
>> > package implementations) to any previous version with its associated
>> > package implementation via a `git checkout' (or whatever) plus `raco
>> > pkg update'. Working this way, the package catalog acts a lot like git
>> > submodules.
>> >
>>
>> This would all rely on being able to access previous revisions to point
>> the catalogue to.
>> Not too big a problem if accessing a package through github, but how
>> would you do this
>> if the source is just a zip file, unless the package author is going to
>> host each and every
>> revision that they made with a unique name?  Yes, you could download
>> these revisions
>> and point to them locally.  However, if you wanted to distribute your
>> packages you
>> would then also have to be prepared to distribute revisions of other
>> authors' packages.
>>
>> I can create stability for me by backing up revisions of packages that
>> work with an
>> application and pointing an application's package catalogue to those
>> revisions.  However,
>> this would mean running package update before each run of a racket
>> application.  It would
>> also mean that to develop the application with multiple contributors
>> that I would have to have
>> a method to distribute other authors' package revisions.
>>
>> With all this package updating, what happens if I want to run two racket
>> applications at
>> the same time that rely on different conflicting releases of a package?
>>
>> This all seems like a real pain and incredibly brittle and frustrating.
>> One little slip, such as
>> forgetting to backup the packages before an update and you could break
>> an application
>> or package with potentially no way of recovering other then
>> re-implementing the functionality
>> of an updated package that you were relying on.
>>
>> Everyday that I use Racket I love it more and more, but I'm scared to
>> invest money building and
>> marketing an application, written in Racket, to be used commercially
>> while these problems remain.
>>
>> I want to build my business, like my house, on rock and not sand.
>>
>> I'm sure that I am not and will not be the only person who worries about
>> this.  Racket has some
>> fantastic contributors with a great academic background, but
>> occasionally it may be useful to hear
>> the concerns of businesses wishing to invest in Racket and develop with it.
>>
>>
>> Thanks
>>
>>
>> Lorry
>>
>> --
>> vLife Systems Ltd
>> Registered Office: The Meridian, 4 Copthall House, Station Square, Coventry,
>> CV1 2FL
>> Registered in England and Wales No. 06477649
>> http://vlifesystems.com
> ____________________
>   Racket Users list:
>   http://lists.racket-lang.org/users