+@subsection Source Registry Configuration
+
+Configuring ASDF used to require special magic
+to be applied just at the right moment,
+between the moment ASDF is loaded and the moment it is used,
+in a way that is specific to the user,
+the implementation he is using and the application he is building.
+
+This made for awkward configuration files and startup scripts
+that could not be shared between users, managed by administrators
+or packaged by distributions.
+
+ASDF 2 provides a well-documented way to configure ASDF,
+with sensible defaults, adequate configuration languages,
+and a coherent set of configuration files and hooks.
+
+We believe it's a vast improvement because it decouples
+application distribution from library distribution.
+The application writer can avoid thinking where the libraries are,
+and the library distributor (dpkg, clbuild, advanced user, etc.)
+can configure them once and for every application.
+Yet settings can be easily overridden where needed,
+so whoever needs control has exactly as much as required.
+
+At the same time, ASDF 2 remains compatible
+with the old magic you may have in your build scripts
+(using @code{*central-registry*} and
+@code{*system-definition-search-functions*})
+to tailor the ASDF configuration to your build automation needs,
+and also allows for new magic, simpler and more powerful magic.
+
+@xref{Controlling where ASDF searches for systems}.
+
+
+@subsection Usual operations are made easier to the user
+
+In ASDF 1, you had to use the awkward syntax
+@code{(asdf:oos 'asdf:load-op :foo)}
+to load a system,
+and similarly for @code{compile-op}, @code{test-op}.
+
+In ASDF 2, you can use shortcuts for the usual operations:
+@code{(asdf:load-system :foo)}, and
+similarly for @code{compile-system}, @code{test-system}.
+
+
+@subsection Many bugs have been fixed
+
+The following issues and many others have been fixed:
+
+@itemize
+@item
+The infamous TRAVERSE function has been revamped significantly,
+with many bugs squashed.
+In particular, dependencies were not correctly propagated
+across submodules within a system but now are.
+The :version and :feature features and
+the :force (system1 .. systemN) feature have been fixed.
+
+@item
+Performance has been notably improved for large systems
+(say with thousands of components) by using
+hash-tables instead of linear search,
+and linear-time list accumulation
+instead of quadratic-time recursive appends.
+
+@item
+Many features used to not be portable,
+especially where pathnames were involved.
+Windows support was notably quirky because of such non-portability.
+
+@item
+The internal test suite used to massively fail on many implementations.
+While still incomplete, it now fully passes
+on all implementations supported by the test suite,
+except for GCL (due to GCL bugs).
+
+@item
+Support was lacking for some implementations.
+ABCL and GCL were notably wholly broken.
+ECL extensions were not integrated in the ASDF release.
+
+@item
+The documentation was grossly out of date.
+
+@end itemize
+
+
+@subsection ASDF itself is versioned
+
+Between new features, old bugs fixed, and new bugs introduced,
+there were various releases of ASDF in the wild,
+and no simple way to check which release had which feature set.
+People using or writing systems had to either make worst-case assumptions
+as to what features were available and worked,
+or take great pains to have the correct version of ASDF installed.
+
+With ASDF 2, we provide a new stable set of working features
+that everyone can rely on from now on.
+Use @code{#+asdf2} to detect presence of ASDF 2,
+@code{(asdf:version-satisfies (asdf:asdf-version) "2.345.67")}
+to check the availability of a version no earlier than required.
+
+
+@subsection ASDF can be upgraded
+
+When an old version of ASDF was loaded,
+it was very hard to upgrade ASDF in your current image
+without breaking everything.
+Instead you had to exit the Lisp process and
+somehow arrange to start a new one from a simpler image.
+Something that can't be done from within Lisp,
+making automation of it difficult,
+which compounded with difficulty in configuration,
+made the task quite hard.
+Yet as we saw before, the task would have been required
+to not have to live with the worst case or non-portable
+subset of ASDF features.
+
+With ASDF 2, it is easy to upgrade
+from ASDF 2 to later versions from within Lisp,
+and not too hard to upgrade from ASDF 1 to ASDF 2 from within Lisp.
+We support hot upgrade of ASDF and any breakage is a bug
+that we will do our best to fix.
+There are still limitations on upgrade, though,
+most notably the fact that after you upgrade ASDF,
+you must also reload or upgrade all ASDF extensions.
+
+@subsection Decoupled release cycle
+
+When vendors were releasing their Lisp implementations with ASDF,
+they had to basically never change version
+because neither upgrade nor downgrade was possible
+without breaking something for someone,
+and no obvious upgrade path was visible and recommendable.
+
+With ASDF 2, upgrade is possible, easy and can be recommended.
+This means that vendors can safely ship a recent version of ASDF,
+confident that if a user isn't fully satisfied,
+he can easily upgrade ASDF and deal
+with a supported recent version of it.
+This means that release cycles will be causally decoupled,
+the practical consequence of which will mean faster convergence
+towards the latest version for everyone.
+
+
+@subsection Pitfalls of the transition to ASDF 2
+
+The main pitfalls in upgrading to ASDF 2 seem to be related
+to the output translation mechanism.
+
+@itemize
+
+@item
+Output translations is enabled by default. This may surprise some users,
+most of them in pleasant way (we hope), a few of them in an unpleasant way.
+It is trivial to disable output translations.
+@xref{FAQ,,``How can I wholly disable the compiler output cache?''}.
+
+@item
+Some systems in the large have been known not to play well with output translations.
+They were relatively easy to fix.
+Once again, it is also easy to disable output translations,
+or to override its configuration.
+
+@item
+The new ASDF output translations are incompatible with ASDF-Binary-Locations.
+They replace A-B-L, and there is compatibility mode to emulate
+your previous A-B-L configuration.
+See @code{enable-asdf-binary-locations-compatibility} in
+@pxref{Controlling where ASDF saves compiled files,,Backward Compatibility}.
+But thou shalt not load ABL on top of ASDF 2.
+
+@end itemize
+
+Other issues include the following:
+
+@itemize
+
+@item
+ASDF pathname designators are now specified in places where they were unspecified,
+and a few small adjustments have to be made to some non-portable defsystems.
+Notably, in the @code{:pathname} argument to a @code{defsystem} and its components,
+a logical pathname (or implementation-dependent hierarchical pathname)
+must now be specified with @code{#p} syntax
+where the namestring might have previously sufficed;
+moreover when evaluation is desired @code{#.} must be used,
+where it wasn't necessary in the toplevel @code{:pathname} argument
+(but necessary in other @code{:pathname} arguments).
+
+@item
+There is a slight performance bug, notably on SBCL,
+when initially searching for @file{asd} files,
+the implicit @code{(directory "/configured/path/**/*.asd")}
+for every configured path @code{(:tree "/configured/path/")}
+in your @code{source-registry} configuration can cause a slight pause.
+Try to @code{(time (asdf:initialize-source-registry))}
+to see how bad it is or isn't on your system.
+If you insist on not having this pause,
+you can avoid the pause by overriding the default source-registry configuration
+and not use any deep @code{:tree} entry but only @code{:directory} entries
+or shallow @code{:tree} entries.
+Or you can fix your implementation to not be quite that slow
+when recursing through directories.
+@emph{Update}: This performance bug fixed the hard way in 2.010.
+
+@item
+On Windows, only LispWorks supports proper default configuration pathnames
+based on the Windows registry.
+Other implementations make do with environment variables,
+that you may have to define yourself if you're using an older version of Windows.
+Windows support is somewhat less tested than Unix support.
+Please help report and fix bugs.
+
+@item
+The mechanism by which one customizes a system so that Lisp files
+may use a different extension from the default @file{.lisp} has changed.
+Previously, the pathname for a component was lazily computed when operating on a system,
+and you would
+@code{(defmethod source-file-type ((component cl-source-file) (system (eql (find-system 'foo))))
+ (declare (ignorable component system)) "lis")}.
+Now, the pathname for a component is eagerly computed when defining the system,
+and instead you will @code{(defclass cl-source-file.lis (cl-source-file) ((type :initform "lis")))}
+and use @code{:default-component-class cl-source-file.lis} as argument to @code{defsystem},
+as detailed in a @pxref{FAQ,How do I create a system definition where all the source files have a .cl extension?} below.
+
+@findex source-file-type
+
+
+@end itemize
+
+
+@section Issues with installing the proper version of ASDF
+
+@subsection ``My Common Lisp implementation comes with an outdated version of ASDF. What to do?''
+
+We recommend you upgrade ASDF.
+@xref{Loading ASDF,,Upgrading ASDF}.
+
+If this does not work, it is a bug, and you should report it.
+@xref{FAQ, report-bugs, Where do I report a bug}.
+In the meantime, you can load @file{asdf.lisp} directly.
+@xref{Loading ASDF,Loading an otherwise installed ASDF}.
+
+
+@subsection ``I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?''
+
+Starting with current candidate releases of ASDF 2,
+it should always be a good time to upgrade to a recent ASDF.
+You may consult with the maintainer for which specific version they recommend,
+but the latest @code{release} should be correct.
+We trust you to thoroughly test it with your implementation before you release it.
+If there are any issues with the current release,
+it's a bug that you should report upstream and that we will fix ASAP.
+
+As to how to include ASDF, we recommend the following:
+
+@itemize
+@item
+If ASDF isn't loaded yet, then @code{(require "asdf")}
+should load the version of ASDF that is bundled with your system.
+If possible so should @code{(require "ASDF")}.
+You may have it load some other version configured by the user,
+if you allow such configuration.
+
+@item
+If your system provides a mechanism to hook into @code{CL:REQUIRE},
+then it would be nice to add ASDF to this hook the same way that
+ABCL, CCL, CLISP, CMUCL, ECL, SBCL and SCL do it.
+
+@item
+You may, like SBCL, have ASDF be implicitly used to require systems
+that are bundled with your Lisp distribution.
+If you do have a few magic systems that come with your implementation
+in a precompiled way such that one should only use the binary version
+that goes with your distribution, like SBCL does,
+then you should add them in the beginning of @code{wrapping-source-registry}.
+
+@item
+If you have magic systems as above, like SBCL does,
+then we explicitly ask you to @emph{NOT} distribute
+@file{asdf.asd} as part of those magic systems.
+You should still include the file @file{asdf.lisp} in your source distribution
+and precompile it in your binary distribution,
+but @file{asdf.asd} if included at all,
+should be secluded from the magic systems,
+in a separate file hierarchy;
+alternatively, you may provide the system
+after renaming it and its @file{.asd} file to e.g.
+@code{asdf-ecl} and @file{asdf-ecl.asd}, or
+@code{sb-asdf} and @file{sb-asdf.asd}.
+Indeed, if you made @file{asdf.asd} a magic system,
+then users would no longer be able to upgrade ASDF using ASDF itself
+to some version of their preference that
+they maintain independently from your Lisp distribution.
+
+@item
+If you do not have any such magic systems, or have other non-magic systems
+that you want to bundle with your implementation,
+then you may add them to the @code{default-source-registry},
+and you are welcome to include @file{asdf.asd} amongst them.
+
+@item
+Please send us upstream any patches you make to ASDF itself,
+so we can merge them back in for the benefit of your users
+when they upgrade to the upstream version.
+
+@end itemize
+
+
+
+@section Issues with configuring ASDF
+
+@subsection ``How can I customize where fasl files are stored?''
+
+@xref{Controlling where ASDF saves compiled files}.
+
+Note that in the past there was an add-on to ASDF called
+@code{ASDF-binary-locations}, developed by Gary King.
+That add-on has been merged into ASDF proper,
+then superseded by the @code{asdf-output-translations} facility.
+
+Note that use of @code{asdf-output-translations}
+can interfere with one aspect of your systems
+--- if your system uses @code{*load-truename*} to find files
+(e.g., if you have some data files stored with your program),
+then the relocation that this ASDF customization performs
+is likely to interfere.
+Use @code{asdf:system-relative-pathname} to locate a file
+in the source directory of some system, and
+use @code{asdf:apply-output-translations} to locate a file
+whose pathname has been translated by the facility.
+
+@subsection ``How can I wholly disable the compiler output cache?''
+
+To permanently disable the compiler output cache
+for all future runs of ASDF, you can:
+
+@example
+mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
+echo ':disable-cache' > ~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf
+@end example
+
+This assumes that you didn't otherwise configure the ASDF files
+(if you did, edit them again),
+and don't somehow override the configuration at runtime
+with a shell variable (see below) or some other runtime command
+(e.g. some call to @code{asdf:initialize-output-translations}).
+
+To disable the compiler output cache in Lisp processes
+run by your current shell, try (assuming @code{bash} or @code{zsh})
+(on Unix and cygwin only):
+
+@example
+export ASDF_OUTPUT_TRANSLATIONS=/:
+@end example
+
+To disable the compiler output cache just in the current Lisp process,
+use (after loading ASDF but before using it):
+
+@example
+(asdf:disable-output-translations)
+@end example
+
+@section Issues with using and extending ASDF to define systems
+
+@subsection ``How can I cater for unit-testing in my system?''
+
+ASDF provides a predefined test operation, @code{test-op}.
+@xref{Predefined operations of ASDF, test-op}.
+The test operation, however, is largely left to the system definer to specify.
+@code{test-op} has been
+a topic of considerable discussion on the
+@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel,asdf-devel mailing list},
+and on the
+@url{https://launchpad.net/asdf,launchpad bug-tracker}.
+
+Here are some guidelines:
+
+@itemize
+@item
+For a given system, @var{foo}, you will want to define a corresponding
+test system, such as @var{foo-test}. The reason that you will want this
+separate system is that ASDF does not out of the box supply components
+that are conditionally loaded. So if you want to have source files
+(with the test definitions) that will not be loaded except when testing,
+they should be put elsewhere.
+
+@item
+The @var{foo-test} system can be defined in an asd file of its own or
+together with @var{foo}. An aesthetic preference against cluttering up
+the filesystem with extra asd files should be balanced against the
+question of whether one might want to directly load @var{foo-test}.
+Typically one would not want to do this except in early stages of
+debugging.
+
+@item
+Record that testing is implemented by @var{foo-test}. For example:
+@example
+(defsystem @var{foo}
+ :in-order-to ((test-op (test-op @var{foo-test})))
+ ....)
+
+(defsystem @var{foo-test}
+ :depends-on (@var{foo} @var{my-test-library} ...)
+ ....)
+@end example
+@end itemize
+
+This procedure will allow you to support users who do not wish to
+install your test framework.
+
+One oddity of ASDF is that @code{operate} (@pxref{Operations,operate})
+does not return a value. So in current versions of ASDF there is no
+reliable programmatic means of determining whether or not a set of tests
+has passed, or which tests have failed. The user must simply read the
+console output. This limitation has been the subject of much
+discussion.
+
+@subsection ``How can I cater for documentation generation in my system?''
+
+The ASDF developers are currently working to add a @code{doc-op}
+to the set of predefined ASDF operations.
+@xref{Predefined operations of ASDF}.
+See also @url{https://bugs.launchpad.net/asdf/+bug/479470}.
+
+
+
+@subsection ``How can I maintain non-Lisp (e.g. C) source files?''
+
+See @code{cffi}'s @code{cffi-grovel}.
+
+@anchor{report-bugs}
+
+
+@subsection ``I want to put my module's files at the top level. How do I do this?''
+
+By default, the files contained in an asdf module go
+in a subdirectory with the same name as the module.
+However, this can be overridden by adding a @code{:pathname ""} argument
+to the module description.
+For example, here is how it could be done
+in the spatial-trees ASDF system definition for ASDF 2:
+
+@example
+(asdf:defsystem :spatial-trees
+ :components
+ ((:module base
+ :pathname ""
+ :components
+ ((:file "package")
+ (:file "basedefs" :depends-on ("package"))
+ (:file "rectangles" :depends-on ("package"))))
+ (:module tree-impls
+ :depends-on (base)
+ :pathname ""
+ :components
+ ((:file "r-trees")
+ (:file "greene-trees" :depends-on ("r-trees"))
+ (:file "rstar-trees" :depends-on ("r-trees"))
+ (:file "rplus-trees" :depends-on ("r-trees"))
+ (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
+ (:module viz
+ :depends-on (base)
+ :pathname ""
+ :components
+ ((:static-file "spatial-tree-viz.lisp")))
+ (:module tests
+ :depends-on (base)
+ :pathname ""
+ :components
+ ((:static-file "spatial-tree-test.lisp")))
+ (:static-file "LICENCE")
+ (:static-file "TODO")))
+@end example
+
+All of the files in the @code{tree-impls} module are at the top level,
+instead of in a @file{tree-impls/} subdirectory.
+
+Note that the argument to @code{:pathname} can be either a pathname object or a string.
+A pathname object can be constructed with the @file{#p"foo/bar/"} syntax,
+but this is discouraged because the results of parsing a namestring are not portable.
+A pathname can only be portably constructed with such syntax as
+@code{#.(make-pathname :directory '(:relative "foo" "bar"))},
+and similarly the current directory can only be portably specified as
+@code{#.(make-pathname :directory '(:relative))}.
+However, as of ASDF 2, you can portably use a string to denote a pathname.
+The string will be parsed as a @code{/}-separated path from the current directory,
+such that the empty string @code{""} denotes the current directory, and
+@code{"foo/bar"} (no trailing @code{/} required in the case of modules)
+portably denotes the same subdirectory as above.
+When files are specified, the last @code{/}-separated component is interpreted
+either as the name component of a pathname
+(if the component class specifies a pathname type),
+or as a name component plus optional dot-separated type component
+(if the component class doesn't specifies a pathname type).
+
+@subsection How do I create a system definition where all the source files have a .cl extension?
+
+Starting with ASDF 2.014.14, you may just pass
+the builtin class @code{cl-source-file.cl} as
+the @code{:default-component-class} argument to @code{defsystem}:
+
+@lisp
+(defsystem my-cl-system
+ :default-component-class cl-source-file.cl
+ ...)
+@end lisp
+
+Another builtin class @code{cl-source-file.lsp} is offered
+for files ending in @file{.lsp}.
+
+If you want to use a different extension
+for which ASDF doesn't provide builtin support,
+or want to support versions of ASDF
+earlier than 2.014.14 (but later than 2.000),
+you can define a class as follows:
+
+@lisp
+;; Prologue: make sure we're using a sane package.
+(defpackage :my-asdf-extension
+ (:use :asdf :common-lisp)
+ (:export #:cl-source-file.lis))
+(in-package :my-asdf-extension)
+
+(defclass cl-source-file.lis (cl-source-file)
+ ((type :initform "lis")))
+@end lisp
+
+Then you can use it as follows:
+@lisp
+(defsystem my-cl-system
+ :default-component-class my-asdf-extension:cl-source-file.lis
+ ...)
+@end lisp
+
+Of course, if you're in the same package, e.g. in the same file,
+you won't need to use the package qualifier before @code{cl-source-file.lis}.
+Actually, if all you're doing is defining this class
+and using it in the same file without other fancy definitions,
+you might skip package complications:
+
+@lisp
+(in-package :asdf)
+(defclass cl-source-file.lis (cl-source-file)
+ ((type :initform "lis")))
+(defsystem my-cl-system
+ :default-component-class cl-source-file.lis
+ ...)
+@end lisp
+
+It is possible to achieve the same effect
+in a way that supports both ASDF 1 and ASDF 2,
+but really, friends don't let friends use ASDF 1.
+Please upgrade to ASDF 2.
+In short, though: do same as above, but
+@emph{before} you use the class in a @code{defsystem},
+you also define the following method:
+
+@lisp
+(defmethod source-file-type ((f cl-source-file.lis) (m module))
+ (declare (ignorable f m))
+ "lis")
+@end lisp
+
+@comment FIXME: Add a FAQ about how to use a new system class...
+
+
+@node TODO list, Inspiration, FAQ, Top
+@comment node-name, next, previous, up
+@chapter TODO list
+
+Here is an old list of things to do,
+in addition to the bugs that are now tracked on launchpad:
+@url{https://launchpad.net/asdf}.
+
+@section Outstanding spec questions, things to add
+
+** packaging systems
+
+*** manual page component?
+
+** style guide for .asd files
+
+You should either use keywords or be careful
+with the package that you evaluate defsystem forms in.
+Otherwise @code{(defsystem partition ...)}
+being read in the @code{cl-user} package
+will intern a @code{cl-user:partition} symbol,
+which will then collide with the @code{partition:partition} symbol.
+
+Actually there's a hairier packages problem to think about too.
+@code{in-order-to} is not a keyword:
+if you read @code{defsystem} forms in a package that doesn't use ASDF,
+odd things might happen.
+
+
+** extending defsystem with new options
+
+You might not want to write a whole parser,
+but just to add options to the existing syntax.
+Reinstate @code{parse-option} or something akin.
+
+
+** document all the error classes
+
+** what to do with compile-file failure
+
+Should check the primary return value from compile-file and see if
+that gets us any closer to a sensible error handling strategy
+
+** foreign files
+
+lift unix-dso stuff from db-sockets
+
+** Diagnostics
+
+A ``dry run'' of an operation can be made with the following form:
+
+@lisp
+(traverse (make-instance '<operation-name>)
+ (find-system <system-name>)
+ 'explain)
+@end lisp
+
+This uses unexported symbols.
+What would be a nice interface for this functionality?
+
+@section Missing bits in implementation
+
+** reuse the same scratch package whenever a system is reloaded from disk
+
+** proclamations probably aren't
+
+** when a system is reloaded with fewer components than it previously had, odd things happen
+
+We should do something inventive when processing a @code{defsystem} form,
+like take the list of kids and @code{setf} the slot to @code{nil},
+then transfer children from old to new list as they're found.
projects / sbcl.git / blobdiff