[racket-dev] [plt] Push #29677: master branch updated

From: Matthew Flatt (mflatt at cs.utah.edu)
Date: Sat Jan 31 11:47:56 EST 2015
Previous message: [racket-dev] [plt] Push #29677: master branch updated
Messages sorted by: [date] [thread] [subject] [author]
I noticed when I made that change that the ".dep" format is still not
documented anywhere. I'll document it soon.

Here's a first cut at the grammar of the value that is stored in a
".dep" file via `write`:

 <deps> = (list <version-string>
                (cons <hash-of-source> <hash-of-dependencies>)
                <dep> ...)
 <dep> = (cons 'indirect <direct-dep>)
       | <direct-dep>
 <direct-dep> = (cons 'ext <plain-dep>) ; non-module dependency
              | <plain-dep>             ; module dependency
 <plain-dep> = (cons 'collects <byte-string>) ; a collection-relative path
             | <byte-string>                  ; other path (not portable)
   

At Fri, 30 Jan 2015 22:29:11 -0800, Eric Dobson wrote:
> This change seemed to change the format of .dep files, likely as intended
> to add the indirect dependencies. Is there any documentation of what the
> format is supposed to be? Currently I've just been trying to read cm.rkt
> and understand how it treats them.
> 
> On Thu, Jan 8, 2015 at 9:31 AM, <mflatt at racket-lang.org> wrote:
> 
> > mflatt has updated `master' from c56c9250f1 to 95e85ec5bd.
> >   http://git.racket-lang.org/plt/c56c9250f1..95e85ec5bd
> >
> > =====[ 2 Commits ]======================================================
> > Directory summary:
> >   45.1% pkgs/racket-doc/scribblings/raco/
> >    4.7% pkgs/racket-doc/scribblings/reference/
> >   47.5% racket/collects/compiler/
> >
> > ~~~~~~~~~~
> >
> > fe9a04d Matthew Flatt <mflatt at racket-lang.org> 2015-01-08 09:11
> > :
> > | doc tweaks for `raco {setup,make}`
> > :
> >   M pkgs/racket-doc/scribblings/raco/make.scrbl  |  4 ++--
> >   M pkgs/racket-doc/scribblings/raco/setup.scrbl | 22
> > ++++++++++++----------
> >
> > ~~~~~~~~~~
> >
> > 95e85ec Matthew Flatt <mflatt at racket-lang.org> 2015-01-08 09:57
> > :
> > | add support for indirect CM dependencies; use in `lazy-require`
> > |
> > | If module M in package P imports module N from package Q,
> > | and if N has a `lazy-require` for a module in R that is
> > | triggered during the compilation of M, then P doesn't really
> > | depend on R; P depends on Q, and Q depends on R, and P
> > | shoudn't necessarily know anything about Q. At the same time,
> > | a change to the file in R means that M must be recompiled.
> > | So, continue to track the compilation dependency, but mark
> > | it as "indirect" so that the package-dependency checker can
> > | ignore the dependency.
> > :
> >   M pkgs/racket-doc/scribblings/raco/make.scrbl       | 33 ++++++++-----
> >   M racket/collects/compiler/cm-accomplice.rkt        | 14 +++---
> >   M racket/collects/compiler/cm.rkt                   | 49
> > ++++++++++++++------
> >   M racket/collects/racket/lazy-require.rkt           |  2 +-
> >   M racket/collects/setup/private/pkg-deps.rkt        |  1 +
> >   M .../racket-doc/scribblings/reference/syntax.scrbl |  8 ++--
> >
> > =====[ Overall Diff ]===================================================
> >
> > pkgs/racket-doc/scribblings/raco/make.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/raco/make.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/raco/make.scrbl
> > @@ -123,7 +123,7 @@ would create only @filepath{compiled/b_rkt.zo} and
> >
> >  @; ----------------------------------------------------------------------
> >
> > - at section{Dependency Files}
> > + at section[#:tag "Dependency Files"]{Dependency Files}
> >
> >  In addition to a bytecode file, @exec{raco make} creates a file
> >  @filepath{compiled/@nonterm{name}_ at nonterm{ext}.dep} that records
> > @@ -538,7 +538,7 @@ messages are instances of a
> > @racket[parallel-compile-event] prefab structure:
> >
> >  @racketblock[
> >    (struct parallel-compile-event (worker event) #:prefab)
> > -].
> > +]
> >
> >  The worker field is the index of the worker that the created the event.
> > The event
> >  field is a @racket[compile-event] as document in
> > @@ -550,25 +550,36 @@ field is a @racket[compile-event] as document in
> >
> >  @defmodule[compiler/cm-accomplice]
> >
> > - at defproc[(register-external-file [file (and path? complete-path?)])
> > void?]{
> > + at defproc[(register-external-file [file (and path? complete-path?)]
> > +                                 [#:indirect? indirect? any/c #f])
> > +         void?]{
> >
> > -Logs a message (see @racket[log-message]) at level @racket['info] to
> > -a logger named @racket['cm-accomplice]. The
> > -message data is a @racketidfont{file-dependency} prefab structure type
> > -with two fields; the first field's value is @racket[file] and the second
> > -field's value is @racket[#f] (to indicate a non-module dependency).
> > +Logs a message (see @racket[log-message]) at level @racket['info] to a
> > +logger named @racket['cm-accomplice]. The message data is a
> > + at racketidfont{file-dependency} prefab structure type with two fields;
> > +the first field's value is @racket[file] and the second field's value
> > +is @racket[#f] (to indicate a non-module dependency). If the
> > + at racket[indirect?] argument is true, the data is more specifically an
> > +instance of a @racketidfont{file-dependency/indirect} prefab structure
> > +type that is a subtype of @racketidfont{file-dependency} with no new
> > +fields.
> >
> >  A compilation manager implemented by @racketmodname[compiler/cm] looks
> > -for such messages to register an external dependency. The compilation
> > -manager records (in a @filepath{.dep} file) the path as contributing
> > -to the implementation of the module currently being
> > +for such messages to register an external dependency. In response, the
> > +compilation manager records (in a @filepath{.dep} file) the path as
> > +contributing to the implementation of the module currently being
> >  compiled. Afterward, if the registered file is modified, the
> > -compilation manager will know to recompile the module.
> > +compilation manager will know to recompile the module. An ``indirect''
> > +dependency has no effect on recompilation, but it can signal to other
> > +tools, such as a package-dependency checker, that the dependency is
> > +indirect (and should not imply a direct package dependency).
> >
> >  The @racket[include] macro, for example, calls this procedure with the
> >  path of an included file as it expands an @racket[include] form.}
> >
> > - at defproc[(register-external-module [file (and path? complete-path?)])
> > void?]{
> > + at defproc[(register-external-module [file (and path? complete-path?)]
> > +                                   [#:indirect? indirect? any/c #f])
> > +         void?]{
> >
> >  Like @racket[register-external-file], but logs a message with a
> >  @racketidfont{file-dependency} prefab structure type whose second
> >
> > pkgs/racket-doc/scribblings/raco/setup.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/raco/setup.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/raco/setup.scrbl
> > @@ -744,13 +744,14 @@ Optional @filepath{info.rkt} fields trigger
> > additional actions by
> >     module. More specifically, used modules are determined when
> >     deleting a @filepath{.dep} file, which would have been created to
> >     accompany a @filepath{.zo} file when the @filepath{.zo} was built
> > -   by @exec{raco setup}. If the @filepath{.dep} file indicates another
> > -   module, that module's @filepath{.zo} is deleted only if it also has
> > -   an accompanying @filepath{.dep} file. In that case, the
> > -   @filepath{.dep} file is deleted, and additional used modules are
> > -   deleted based on the used module's @filepath{.dep} file, etc.
> > -   Supplying a specific list of collections to @exec{raco setup} disables
> > -   this dependency-based deletion of compiled files.}
> > +   by @exec{raco setup} or @exec{raco make} (see
> > +   @secref["Dependency\x20Files"]). If the @filepath{.dep} file
> > +   indicates another module, that module's @filepath{.zo} is deleted
> > +   only if it also has an accompanying @filepath{.dep} file. In that
> > +   case, the @filepath{.dep} file is deleted, and additional used
> > +   modules are deleted based on the used module's @filepath{.dep}
> > +   file, etc. Supplying a specific list of collections to @exec{raco
> > +   setup} disables this dependency-based deletion of compiled files.}
> >
> >  ]
> >
> > @@ -816,9 +817,10 @@ with fewer dependencies.
> >  @subsection{How Dependency Checking Works}
> >
> >  Dependency checking uses @filepath{.zo} files, associated
> > - at filepath{.dep} files, and the documentation index. Dynamic
> > -references, such as through @racket[dynamic-require], are not visible
> > -to the dependency checker; only dependencies via @racket[require],
> > + at filepath{.dep} files (see @secref["Dependency Files"]), and the
> > +documentation index. Dynamic references, such as through
> > + at racket[dynamic-require], are not visible to the dependency checker;
> > +only dependencies via @racket[require],
> >  @racket[define-runtime-module-path-index], and other forms that
> >  cooperate with @racket[raco make] are visible for dependency checking.
> >
> >
> > pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > @@ -3005,10 +3005,10 @@ submodule). Introduced submodules have the names
> >  @racket[lazy-require-]@racket[_n]@racketidfont{-}@racket[_m], where
> >  @racket[_n] is a phase-level number and @racket[_m] is a number.
> >
> > -When the use of a lazily-required function triggers module loading,
> > - at racket[register-external-module] declares a potential compilation
> > -dependency (in case the function is used in the process of compiling a
> > -module).
> > +When the use of a lazily-required function triggers module loading, it
> > +also triggers a use of @racket[register-external-module] to declare an
> > +indirect compilation dependency (in case the function is used in the
> > +process of compiling a module).
> >
> >  @examples[#:eval lazy-require-eval
> >  (lazy-require
> >
> > racket/collects/compiler/cm-accomplice.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/compiler/cm-accomplice.rkt
> > +++ NEW/racket/collects/compiler/cm-accomplice.rkt
> > @@ -3,16 +3,18 @@
> >  (provide register-external-file
> >           register-external-module)
> >
> > -(define (register-external-file f)
> > -  (register-external 'register-external-file f #f))
> > -(define (register-external-module f)
> > -  (register-external 'register-external-module f #t))
> > +(define (register-external-file f #:indirect? [indirect? #f])
> > +  (register-external 'register-external-file f #f indirect?))
> > +(define (register-external-module f #:indirect? [indirect? #f])
> > +  (register-external 'register-external-module f #t indirect?))
> >
> > -(define (register-external who f module?)
> > +(define (register-external who f module? indirect?)
> >    (unless (and (path? f) (complete-path? f))
> >      (raise-type-error who "complete path" f))
> >    (log-message (current-logger)
> >                 'info
> >                 'cm-accomplice
> >                 (format "file dependency: ~s" f)
> > -               `#s(file-dependency ,f ,module?)))
> > +               (if indirect?
> > +                   `#s((file-dependency/indirect file-dependency 2) ,f
> > ,module?)
> > +                   `#s(file-dependency ,f ,module?))))
> >
> > racket/collects/compiler/cm.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/compiler/cm.rkt
> > +++ NEW/racket/collects/compiler/cm.rkt
> > @@ -231,9 +231,13 @@
> >  (define (get-dep-sha1s deps up-to-date collection-cache read-src-syntax
> > mode roots must-exist? seen)
> >    (let ([l (for/fold ([l null]) ([dep (in-list deps)])
> >               (and l
> > +                  ;; (cons 'indirect dep) => indirect dependency (for
> > pkg-dep checking)
> >                    ;; (cons 'ext rel-path) => a non-module file, check
> > source
> >                    ;; rel-path => a module file name, check cache
> > -                  (let* ([ext? (and (pair? dep) (eq? 'ext (car dep)))]
> > +                  (let* ([dep (if (and (pair? dep) (eq? 'indirect (car
> > dep)))
> > +                                  (cdr dep)
> > +                                  dep)]
> > +                         [ext? (and (pair? dep) (eq? 'ext (car dep)))]
> >                           [p (collects-relative*->path (if ext? (cdr dep)
> > dep) collection-cache)])
> >                      (cond
> >                       [ext? (let ([v (get-source-sha1 p)])
> > @@ -273,19 +277,26 @@
> >                                           external-module-deps ; can
> > create cycles if misused!
> >                                           reader-deps))]
> >          [external-deps (remove-duplicates external-deps)])
> > +    (define (path*->collects-relative/maybe-indirect dep)
> > +      (if (and (pair? dep) (eq? 'indirect (car dep)))
> > +          (cons 'indirect (path*->collects-relative (cdr dep)))
> > +          (path*->collects-relative dep)))
> >      (with-compile-output dep-path
> >        (lambda (op tmp-path)
> >          (let ([deps (append
> > -                     (map path*->collects-relative deps)
> > +                     (map path*->collects-relative/maybe-indirect deps)
> >                       (map (lambda (x)
> > -                            (cons 'ext (path*->collects-relative x)))
> > +                            (define d
> > (path*->collects-relative/maybe-indirect x))
> > +                            (if (and (pair? d) (eq? 'indirect d))
> > +                                (cons 'indirect (cons 'ext (cdr d)))
> > +                                (cons 'ext d)))
> >                            external-deps))])
> > -        (write (list* (version)
> > -                      (cons (or src-sha1 (get-source-sha1 path))
> > -                            (get-dep-sha1s deps up-to-date
> > collection-cache read-src-syntax mode roots #t #hash()))
> > -                      deps)
> > -               op)
> > -        (newline op))))))
> > +          (write (list* (version)
> > +                        (cons (or src-sha1 (get-source-sha1 path))
> > +                              (get-dep-sha1s deps up-to-date
> > collection-cache read-src-syntax mode roots #t #hash()))
> > +                        deps)
> > +                 op)
> > +          (newline op))))))
> >
> >  (define (format-time sec)
> >    (let ([d (seconds->date sec)])
> > @@ -311,6 +322,7 @@
> >  (define-struct ext-reader-guard (proc top)
> >    #:property prop:procedure (struct-field-index proc))
> >  (define-struct file-dependency (path module?) #:prefab)
> > +(define-struct (file-dependency/indirect file-dependency) () #:prefab)
> >
> >  (define (compile-zo* mode roots path src-sha1 read-src-syntax zo-name
> > up-to-date collection-cache)
> >    ;; The `path' argument has been converted to .rkt or .ss form,
> > @@ -322,10 +334,14 @@
> >    (define reader-deps null)
> >    (define deps-sema (make-semaphore 1))
> >    (define done-key (gensym))
> > -  (define (external-dep! p module?)
> > +  (define (external-dep! p module? indirect?)
> > +    (define bstr (path->bytes p))
> > +    (define dep (if indirect?
> > +                    (cons 'indirect bstr)
> > +                    bstr))
> >      (if module?
> > -        (set! external-module-deps (cons (path->bytes p)
> > external-module-deps))
> > -        (set! external-deps (cons (path->bytes p) external-deps))))
> > +        (set! external-module-deps (cons dep external-module-deps))
> > +        (set! external-deps (cons dep external-deps))))
> >    (define (reader-dep! p)
> >      (call-with-semaphore
> >       deps-sema
> > @@ -386,7 +402,8 @@
> >                     (file-dependency? (vector-ref l 2))
> >                     (path? (file-dependency-path (vector-ref l 2))))
> >            (external-dep! (file-dependency-path (vector-ref l 2))
> > -                         (file-dependency-module? (vector-ref l 2))))
> > +                         (file-dependency-module? (vector-ref l 2))
> > +                         (file-dependency/indirect? (vector-ref l 2))))
> >          (loop))))
> >
> >    ;; Write the code and dependencies:
> > @@ -627,9 +644,13 @@
> >                ;; If `sha1-only?', then `maybe-compile-zo' returns a #f or
> > thunk:
> >                (maybe-compile-zo sha1-only? deps mode roots path orig-path
> > read-src-syntax up-to-date collection-cache new-seen)]
> >               [(ormap
> > -               (lambda (p)
> > +               (lambda (raw-p)
> > +                 ;; (cons 'indirect dep) => indirect dependency (for
> > pkg-dep checking)
> >                   ;; (cons 'ext rel-path) => a non-module file (check date)
> >                   ;; rel-path => a module file name (check transitive
> > dates)
> > +                 (define p (if (and (pair? raw-p) (eq? 'indirect (car
> > raw-p)))
> > +                               (cdr raw-p)
> > +                               raw-p))
> >                   (define ext? (and (pair? p) (eq? 'ext (car p))))
> >                   (define d (collects-relative*->path (if ext? (cdr p) p)
> > collection-cache))
> >                   (define t
> >
> > racket/collects/racket/lazy-require.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/racket/lazy-require.rkt
> > +++ NEW/racket/collects/racket/lazy-require.rkt
> > @@ -110,4 +110,4 @@
> >                   modpath
> >                   (variable-reference->resolved-module-path vr))))])
> >      (when (path? path)
> > -      (register-external-module path))))
> > +      (register-external-module path #:indirect? #t))))
> >
> > racket/collects/setup/private/pkg-deps.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/setup/private/pkg-deps.rkt
> > +++ NEW/racket/collects/setup/private/pkg-deps.rkt
> > @@ -489,6 +489,7 @@
> >              ;; Treat everything in ".dep" as 'build mode...
> >              (define deps (cddr (call-with-input-file* (build-path dir f)
> > read)))
> >              (for ([dep (in-list deps)])
> > +              ;; Note: indirect dependencies (which start with 'indirect)
> > are ignored
> >                (when (and (pair? dep)
> >                           (eq? 'collects (car dep)))
> >                  (define path-strs (map bytes->string/utf-8 (cdr dep)))
> >
> _________________________
>   Racket Developers list:
>   http://lists.racket-lang.org/dev
Posted on the dev mailing list.
Previous message: [racket-dev] [plt] Push #29677: master branch updated
Messages sorted by: [date] [thread] [subject] [author]