<div dir="ltr">Thanks, Jay. I have two high-level negative comments.<div><br></div><div style>1) I don't think that it is feasible to aim for a world where 'raco setup' never errors. (The real-world variation of your matthew-isnt-looking example is system-specific code and we do do that a bunch.)</div>
<div style><br></div><div style>2) You are currently making a distinction between 'raco pkg' and 'raco setup' as if these were the only two things. But this is actually unfair. Really, when you are at this level of granularity, 'raco pkg' may actually be one thing but 'raco setup' is actually N things (zo file building, info.rkt cache building, documentation building, documentation index building (a different thing) executable creation, and some stuff I'm forgetting (oh: and all in parallel and on a per-collection basis, etc etc)). Despite that, we have hammered on 'raco setup' enough that it now, mostly, seems to be a single thing that succeeds or fails in a somewhat understandable way for all kinds of common failures. At least that's one conclusion I take from earlier messages in this thread. :)</div>
<div style><br></div><div style>That is where I'd like to us to aim for raco pkg to be. I think this is about smoothing out edges in common error situations and not about fundamental changes.</div><div style><br></div>
<div style>On the positive side, I really like how you can be precise about your guarantee #6. This seems like a fantastic foundation on which to build a system where 'raco pkg' might trigger a 'raco setup' that spews error messages but that various queries afterwards can help the user explore what has happened and how to fix it. I appreciate your "stubbornness" on keeping the internal/logical integrity of the pkg system. We need that.</div>
<div style><br></div><div style>Robby</div><div style><br></div><div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 14, 2013 at 9:13 AM, Jay McCarthy <span dir="ltr"><<a href="mailto:jay.mccarthy@gmail.com" target="_blank">jay.mccarthy@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'll try to respond to these four messages at the same time...<br>
<br>
Sam said,<br>
<div class="im">> In addition to the larger point Robby makes, this can be pretty<br>
> confusing. For example, you can fail to install enough dependencies,<br>
> I think.<br>
><br>
> Another problem is that there's no way to know what to do to fix<br>
> things. Say there's an error in `raco setup` that's transient (the<br>
> machine lost power, for example). What command do I run to 'fix' the<br>
> setup? I don't even know what collections were being installed.<br>
<br>
</div>On Thu, Jun 13, 2013 at 9:27 PM, Robby Findler<br>
<div class="im"><<a href="mailto:robby@eecs.northwestern.edu">robby@eecs.northwestern.edu</a>> wrote:<br>
> I know it runs it. I don't know why Jay writes "The package system says<br>
> something is installed when the files are in place and the link is made.<br>
> From some perspective, that's its job.". I can't tell if there's some<br>
> technical piece I'm missing or not (on the surface, these words sound almost<br>
> lazy but I *KNOW* Jay is anything but lazy!)<br>
><br>
> Robby<br>
><br>
><br>
> On Thu, Jun 13, 2013 at 10:21 PM, Carl Eastlund <<a href="mailto:cce@ccs.neu.edu">cce@ccs.neu.edu</a>> wrote:<br>
>><br>
>> It does run 'raco setup', it just doesn't have much to do in response to a<br>
>> failure, at least right now.<br>
>><br>
>> Carl Eastlund<br>
>><br>
>> On Thu, Jun 13, 2013 at 11:14 PM, Robby Findler<br>
>> <<a href="mailto:robby@eecs.northwestern.edu">robby@eecs.northwestern.edu</a>> wrote:<br>
>>><br>
>>> Also, Jay: can you explain more why 'raco setup' isn't something that we<br>
>>> should think about as running "inside" the pkg manager? (I'm not saying that<br>
>>> automatically rolling back packages is the right thing to do or anything<br>
>>> like that, but I would like to understand the model you have better.)<br>
<br>
</div>Short:<br>
<br>
I interpret these messages to be saying, "raco pkg printed error<br>
messages and didn't say what was wrong or what I should do about<br>
it". The messages came from 'raco setup', of course, and 'raco pkg'<br>
gave no interpretation.<br>
<br>
As a simple first step, we can have 'raco pkg' observe the output of<br>
'raco setup' (which might require changing it) and if it gives any<br>
error message of any kind, end with a message like:<br>
<br>
"raco pkg detected strange output during setup. This could mean there<br>
was a problem. If you would like to undo this command, run:<br>
<br>
raco pkg remove the-package-you-installed"<br>
<br>
This sounds like a simple to change to make, but there's one big<br>
problem. After 'raco pkg update' runs, the undo command is much more<br>
complicated and scary in its own way<br>
<br>
raco pkg remove --force the-package-you-updated<br>
raco pkg install the-source-for-the-package-you-updated-at-the-old-git-commit<br>
<br>
But some package sources may not be possible to get back, such as an<br>
old tar ball that you downloaded and don't have any more.<br>
<br>
When raco pkg update runs normally, it leaves the old files in place<br>
until the end and then deletes them. It could put these into some sort<br>
of 'archive' location to facilitate that. But it's a big change and<br>
easy to get very wrong.<br>
<br>
Long:<br>
<br>
I also interpret these messages to be saying<br>
<br>
"When raco pkg returns with exit status 0, the packages I asked to be<br>
install/updated should 'work'."<br>
<br>
This is a reasonable wish, but I don't think it can be enforced<br>
technically. Let's un-ravel worked-ness:<br>
<br>
1. Does the task I thought it would do<br>
<br>
Clearly no way to ensure this. No package system does this.<br>
<br>
2. Every file executes without errors<br>
<br>
This is what DrDr normally does and is too burdensome for every<br>
package, and includes problems of the next stage. No package system<br>
does this.<br>
<br>
3. Passes its test cases on my system<br>
<br>
CPAN is fairly unique in doing this upon install, but virtually no<br>
other package system does this. I don't think it is actually useful,<br>
because there's no reason to have confidence in the test cases and<br>
it's hard from a test case writer perspective to get everything in<br>
place. (For example, a Mongo DB interface needs a running Mongo DB<br>
server to test against.)<br>
<br>
4. Compiles on my system<br>
<br>
Most package systems don't need to think about this, because they<br>
ship only binaries. Languages like Perl and Javascript also don't<br>
really do this, because they don't compile in any real<br>
sense. Languages that do can typically make a pretty simple promise<br>
based on their languages inability to have bizarre and<br>
unpredicatable compile-time dependencies.<br>
<br>
Racket has the ability to provide bizarre and unpredictable<br>
compile-time dependencies:<br>
<br>
#lang racket/base<br>
<br>
(define-syntax (macro-invocation stx)<br>
(if matthew-isn't-looking<br>
#'(require #,(random-file))<br>
#'(void)))<br>
(macro-invocation)<br>
<br>
Since we can't really offer this, the best we can do is trust the<br>
dependencies listed. We have ideas on how to enforce those<br>
dependencies, but it isn't implemented yet.<br>
<br>
5. Its actual dependencies are on my system<br>
<br>
This is like the above.<br>
<br>
6. Its listed dependencies are on my system<br>
<br>
This is the standard that 'raco pkg' uses. It guarantees that either<br>
all dependencies are installed or that no changes were made. This<br>
means that you can never run 'raco pkg update', download a new<br>
version of an installed package that has a new dependency that<br>
conflicts with a current package, and then end up with a failure<br>
that results in some new packages and some old packages.<br>
<br>
This, by the way, is the most common standard of 'worked-ness' on<br>
Linux package systems, like pacman, apt-get, etc, that 'raco pkg' is<br>
based on. These end-user tools don't enforce anything when an<br>
install happens and sometimes there are mistakes by package<br>
developers that cause something to break on an end-user system and<br>
it has to be dealt with somewhere on the developer side.<br>
<br>
In other words, there is *no* error that can happen during 'raco<br>
setup' that breaks the package system. Sam's question about what do I<br>
run to "fix" things doesn't make sense to me, because there can't be<br>
anything wrong at that point. If you are using links and then go<br>
change the code in response to a 'raco setup' error, you don't need to<br>
tell the package system at all, you just run 'raco setup' (with no<br>
arguments so it runs on everything) again.<br>
<br>
--<br>
<br>
Given that I think we can't enforce higher standards technically, my<br>
goal has been to enforce standard (2) with the concept of ring-0<br>
packages. To remind you,<br>
<br>
ring-2: No constraints<br>
<br>
ring-1: Cannot conflict with any package in ring-1 or ring-0.<br>
<br>
ring-0: Must contain tests that I can run on DrDr and that pass<br>
reliably<br>
<br>
Once that is in place, the next step would to make it "difficult" to<br>
install anything other than ring-0 packages. (For instance, not<br>
offering auto-install and not having them in the central documentation<br>
site.) The goal would be to make it so that 'raco setup' never errors.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
--<br>
Jay McCarthy <<a href="mailto:jay@cs.byu.edu">jay@cs.byu.edu</a>><br>
Assistant Professor / Brigham Young University<br>
<a href="http://faculty.cs.byu.edu/~jay" target="_blank">http://faculty.cs.byu.edu/~jay</a><br>
<br>
"The glory of God is Intelligence" - D&C 93<br>
</div></div></blockquote></div><br></div>