Fix make-array transforms.

[sbcl.git] / contrib / asdf / asdf.texinfo

\input texinfo          @c -*- texinfo -*-
@c %**start of header
@setfilename asdf.info
@settitle ASDF Manual
@c %**end of header

@c We use @&key, etc to escape & from TeX in lambda lists --
@c so we need to define them for info as well.
@macro AallowOtherKeys
&allow-other-keys
@end macro
@macro Aoptional
&optional
@end macro
@macro Arest
&rest
@end macro
@macro Akey
&key
@end macro
@macro Abody
&body
@end macro

@c for install-info
@dircategory Software development
@direntry
* asdf: (asdf).           Another System Definition Facility (for Common Lisp)
@end direntry

@copying
This manual describes ASDF, a system definition facility
for Common Lisp programs and libraries.

You can find the latest version of this manual at
@url{http://common-lisp.net/project/asdf/asdf.html}.

ASDF Copyright @copyright{} 2001-2013 Daniel Barlow and contributors.

This manual Copyright @copyright{} 2001-2013 Daniel Barlow and contributors.

This manual revised @copyright{} 2009-2013 Robert P. Goldman and Francois-Rene Rideau.

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
``Software''), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@end copying



@titlepage
@title ASDF: Another System Definition Facility

@c The following two commands start the copyright page.
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@c Output the table of contents at the beginning.
@contents

@c -------------------

@ifnottex

@node Top, Introduction, (dir), (dir)
@top asdf: another system definition facility

@insertcopying

@menu
* Introduction::
* Loading ASDF::
* Configuring ASDF::
* Using ASDF::
* Defining systems with defsystem::
* The object model of ASDF::
* Controlling where ASDF searches for systems::
* Controlling where ASDF saves compiled files::
* Error handling::
* Miscellaneous additional functionality::
* Getting the latest version::
* FAQ::
* TODO list::
* Inspiration::
* Concept Index::
* Function and Class Index::
* Variable Index::

@c @detailmenu
@c  --- The Detailed Node Listing ---

@c Defining systems with defsystem

@c * The defsystem form::
@c * A more involved example::
@c * The defsystem grammar::
@c * Other code in .asd files::

@c The object model of ASDF

@c * Operations::
@c * Components::
@c * Functions::

@c Operations

@c * Predefined operations of ASDF::
@c * Creating new operations::

@c Components

@c * Common attributes of components::
@c * Pre-defined subclasses of component::
@c * Creating new component types::

@c properties

@c * Pre-defined subclasses of component::
@c * Creating new component types::

@c @end detailmenu
@end menu

@end ifnottex

@c -------------------

@node Introduction, Loading ASDF, Top, Top
@comment  node-name,  next,  previous,  up
@chapter Introduction
@cindex ASDF-related features
@vindex *features*
@cindex Testing for ASDF
@cindex ASDF versions
@cindex :asdf
@cindex :asdf2
@cindex :asdf3

ASDF is Another System Definition Facility:
a tool for specifying how systems of Common Lisp software
are comprised of components (sub-systems and files),
and how to operate on these components in the right order
so that they can be compiled, loaded, tested, etc.

ASDF presents three faces:
one for users of Common Lisp software who want to reuse other people's code,
one for writers of Common Lisp software who want to specify how to build their systems,
one for implementers of Common Lisp extensions who want to extend the build system.
@xref{Using ASDF,,Loading a system},
to learn how to use ASDF to load a system.
@xref{Defining systems with defsystem},
to learn how to define a system of your own.
@xref{The object model of ASDF}, for a description of
the ASDF internals and how to extend ASDF.

@emph{Nota Bene}:
We have released ASDF 2.000 on May 31st 2010,
and ASDF 3.0 on January 31st 2013.
Releases of ASDF 2 and later have since then been included
in all actively maintained CL implementations that used to bundle ASDF 1,
plus some implementations that didn't use to,
and has been made to work with all actively used CL implementations and a few more.
@xref{FAQ,,``What has changed between ASDF 1 and ASDF 2?''}.
Furthermore, it is possible to upgrade from ASDF 1 to ASDF 2 or ASDF 3 on the fly.
For this reason, we have stopped supporting ASDF 1 and ASDF 2.
If you are using ASDF 1 or ASDF 2 and are experiencing any kind of issues or limitations,
we recommend you upgrade to ASDF 3
--- and we explain how to do that. @xref{Loading ASDF}.

Also note that ASDF is not to be confused with ASDF-Install.
ASDF-Install is not part of ASDF, but a separate piece of software.
ASDF-Install is also unmaintained and obsolete.
We recommend you use Quicklisp instead,
which works great and is being actively maintained.
If you want to download software from version control instead of tarballs,
so you may more easily modify it, we recommend clbuild.


@node Loading ASDF, Configuring ASDF, Introduction, Top
@comment  node-name,  next,  previous,  up
@chapter Loading ASDF
@vindex *central-registry*
@cindex link farm
@findex load-system
@findex require-system
@findex compile-system
@findex test-system
@cindex system directory designator
@findex operate
@findex oos

@c @menu
@c * Installing ASDF::
@c @end menu


@section Loading a pre-installed ASDF

Most recent Lisp implementations include a copy of ASDF 2, and soon ASDF 3.
You can usually load this copy using Common Lisp's @code{require} function:

@lisp
(require "asdf")
@end lisp

As of the writing of this manual,
the following implementations provide ASDF 2 this way:
abcl allegro ccl clisp cmucl ecl lispworks mkcl sbcl xcl.
The following implementation doesn't provide it yet but will in an upcoming release:
scl.
The following implementations are obsolete, not actively maintained,
and most probably will never bundle it:
cormanlisp gcl genera mcl.

If the implementation you are using doesn't provide ASDF 2 or ASDF 3,
see @pxref{Loading ASDF,,Loading an otherwise installed ASDF} below.
If that implementation is still actively maintained,
you may also send a bug report to your Lisp vendor and complain
about their failing to provide ASDF.

NB: all implementations except clisp also accept
@code{(require "ASDF")}, @code{(require 'asdf)} and @code{(require :asdf)}.
For portability's sake, you probably want to use @code{(require "asdf")}.


@section Checking whether ASDF is loaded

To check whether ASDF is properly loaded in your current Lisp image,
you can run this form:

@lisp
(asdf:asdf-version)
@end lisp

If it returns a string,
that is the version of ASDF that is currently installed.

If it raises an error,
then either ASDF is not loaded, or
you are using an old version of ASDF.

You can check whether an old version is loaded
by checking if the ASDF package is present.
The form below will allow you to programmatically determine
whether a recent version is loaded, an old version is loaded,
or none at all:

@lisp
(when (find-package :asdf)
  (let ((ver (symbol-value (or (find-symbol (string :*asdf-version*) :asdf)
                               (find-symbol (string :*asdf-revision*) :asdf)))))
    (etypecase ver
      (string ver)
      (cons (with-output-to-string (s)
              (loop for (n . m) on ver do (princ n s) (when m (princ "." s)))))
      (null "1.0"))))
@end lisp

If it returns @code{NIL} then ASDF is not installed.
Otherwise it should return a string.
If it returns @code{"1.0"}, then it can actually be
any version before 1.77 or so, or some buggy variant of 1.x.

If you are experiencing problems with ASDF,
please try upgrading to the latest released version,
using the method below,
before you contact us and raise an issue.


@section Upgrading ASDF

If your implementation provides ASDF 3 or later,
you only need to @code{(require "asdf")}:
ASDF will automatically look whether an updated version of itself is available
amongst the regularly configured systems, before it compiles anything else.
See @pxref{Configuring ASDF} below.

If your implementation does provide ASDF 2 or later,
but not ASDF 3 or later,
and you want to upgrade to a more recent version,
you need to install and configure your ASDF as above,
and additionally, you need to explicitly tell ASDF to load itself,
right after you require your implementation's old ASDF 2:

@lisp
(require "asdf")
(asdf:load-system :asdf)
@end lisp

If on the other hand, your implementation only provides an old ASDF,
you will require a special configuration step and an old-style loading.
Take special attention to not omit the trailing directory separator
@code{/} at the end of your pathname:

@lisp
(require "asdf")
(push #p"@var{/path/to/new/asdf/}" asdf:*central-registry*)
(asdf:oos 'asdf:load-op :asdf)
@end lisp

Note that ASDF 1 won't redirect its output files,
or at least won't do it according to your usual ASDF 2 configuration.
You therefore need write access on the directory
where you install the new ASDF,
and make sure you're not using it
for multiple mutually incompatible implementations.
At worst, you may have to have multiple copies of the new ASDF,
e.g. one per implementation installation, to avoid clashes.
Note that to our knowledge all implementations that provide ASDF
provide ASDF 2 in their latest release, so
you may want to upgrade your implementation rather than go through that hoop.

Finally, if you are using an unmaintained implementation
that does not provide ASDF at all,
see @pxref{Loading ASDF,,Loading an otherwise installed ASDF} below.

Note that there are some limitations to upgrading ASDF:
@itemize
@item
Previously loaded ASDF extension becomes invalid, and will need to be reloaded.
This applies to e.g. CFFI-Grovel, or to hacks used by ironclad, etc.
Since it isn't possible to automatically detect what extensions are present
that need to be invalidated,
ASDF will actually invalidate all previously loaded systems
when it is loaded on top of a different ASDF version,
starting with ASDF 2.014.8 (as far as releases go, 2.015);
and it will automatically attempt this self-upgrade as its very first step
starting with ASDF 3.

@item
For this an many other reasons,
it important reason to load, configure and upgrade ASDF (if needed)
as one of the very first things done by your build and startup scripts.
Until all implementations provide ASDF 3 or later,
it is safer if you upgrade ASDF and its extensions as a special step
at the very beginning of whatever script you are running,
before you start using ASDF to load anything else;
even afterwards, it is still a good idea, to avoid having to
load and reload code twice as it gets invalidated.

@item
Until all implementations provide ASDF 3 or later,
it is unsafe to upgrade ASDF as part of loading a system
that depends on a more recent version of ASDF,
since the new one might shadow the old one while the old one is running,
and the running old one will be confused
when extensions are loaded into the new one.
In the meantime, we recommend that your systems should @emph{not} specify
@code{:depends-on (:asdf)}, or @code{:depends-on ((:version :asdf "2.010"))},
but instead that they check that a recent enough ASDF is installed,
with such code as:
@example
(unless (or #+asdf2 (asdf:version-satisfies
                     (asdf:asdf-version) *required-asdf-version*))
  (error "FOO requires ASDF ~A or later." *required-asdf-version*))
@end example
@item
Until all implementations provide ASDF 3 or later,
it is unsafe for a system to transitively depend on ASDF
and not directly depend on ASDF;
if any of the system you use either depends-on asdf,
system-depends-on asdf, or transitively does,
you should also do as well.
@end itemize


@section Loading an otherwise installed ASDF

If your implementation doesn't include ASDF,
if for some reason the upgrade somehow fails,
does not or cannot apply to your case,
you will have to install the file @file{asdf.lisp}
somewhere and load it with:

@lisp
(load "/path/to/your/installed/asdf.lisp")
@end lisp

The single file @file{asdf.lisp} is all you normally need to use ASDF.

You can extract this file from latest release tarball on the
@url{http://common-lisp.net/project/asdf/,ASDF website}.
If you are daring and willing to report bugs, you can get
the latest and greatest version of ASDF from its git repository.
@xref{Getting the latest version}.

For maximum convenience you might want to have ASDF loaded
whenever you start your Lisp implementation,
for example by loading it from the startup script or dumping a custom core
--- check your Lisp implementation's manual for details.


@node Configuring ASDF, Using ASDF, Loading ASDF, Top
@comment  node-name,  next,  previous,  up

@chapter Configuring ASDF

@section Configuring ASDF to find your systems

So it may compile and load your systems, ASDF must be configured to find
the @file{.asd} files that contain system definitions.

Since ASDF 2, the preferred way to configure where ASDF finds your systems is
the @code{source-registry} facility,
fully described in its own chapter of this manual.
@xref{Controlling where ASDF searches for systems}.

The default location for a user to install Common Lisp software is under
@file{~/.local/share/common-lisp/source/}.
If you install software there (it can be a symlink),
you don't need further configuration.
If you're installing software yourself at a location that isn't standard,
you have to tell ASDF where you installed it. See below.
If you're using some tool to install software (e.g. Quicklisp),
the authors of that tool should already have configured ASDF.

The simplest way to add a path to your search path,
say @file{/home/luser/.asd-link-farm/}
is to create the directory
@file{~/.config/common-lisp/source-registry.conf.d/}
and there create a file with any name of your choice,
and with the type @file{conf},
for instance @file{42-asd-link-farm.conf}
containing the line:

@kbd{(:directory "/home/luser/.asd-link-farm/")}

If you want all the subdirectories under @file{/home/luser/lisp/}
to be recursively scanned for @file{.asd} files, instead use:

@kbd{(:tree "/home/luser/lisp/")}

Note that your Operating System distribution or your system administrator
may already have configured system-managed libraries for you.

The required @file{.conf} extension allows you to have disabled files
or editor backups (ending in @file{~}), and works portably
(for instance, it is a pain to allow both empty and non-empty extension on CLISP).
Excluded are files the name of which start with a @file{.} character.
It is customary to start the filename with two digits
that specify the order in which the directories will be scanned.

ASDF will automatically read your configuration
the first time you try to find a system.
You can reset the source-registry configuration with:

@lisp
(asdf:clear-source-registry)
@end lisp

And you probably should do so before you dump your Lisp image,
if the configuration may change
between the machine where you save it at the time you save it
and the machine you resume it at the time you resume it.
Actually, you should use @code{(asdf:clear-configuration)}
before you dump your Lisp image, which includes the above.


@section Configuring ASDF to find your systems --- old style

The old way to configure ASDF to find your systems is by
@code{push}ing directory pathnames onto the variable
@code{asdf:*central-registry*}.

You must configure this variable between the time you load ASDF
and the time you first try to use it.
Loading and configuring ASDF presumably happen
as part of some initialization script that builds or starts
your Common Lisp software system.
(For instance, some SBCL users used to put it in their @file{~/.sbclrc}.)

The @code{asdf:*central-registry*} is empty by default in ASDF 2 or ASDF 3,
but is still supported for compatibility with ASDF 1.
When used, it takes precedence over the above source-registry@footnote{
It is possible to further customize
the system definition file search.
That's considered advanced use, and covered later:
search forward for
@code{*system-definition-search-functions*}.
@xref{Defining systems with defsystem}.}.

For instance, if you wanted ASDF to find the @file{.asd} file
@file{/home/me/src/foo/foo.asd} your initialization script
could after it loads ASDF with @code{(require "asdf")}
configure it with:

@lisp
(push "/home/me/src/foo/" asdf:*central-registry*)
@end lisp

Note the trailing slash: when searching for a system,
ASDF will evaluate each entry of the central registry
and coerce the result to a pathname@footnote{
ASDF will indeed call @code{EVAL} on each entry.
It will also skip entries that evaluate to @code{NIL}.

Strings and pathname objects are self-evaluating,
in which case the @code{EVAL} step does nothing;
but you may push arbitrary SEXP onto the central registry,
that will be evaluated to compute e.g. things that depend
on the value of shell variables or the identity of the user.

The variable @code{asdf:*central-registry*} is thus a list of
``system directory designators''.
A @dfn{system directory designator} is a form
which will be evaluated whenever a system is to be found,
and must evaluate to a directory to look in.
By ``directory'' here, we mean
``designator for a pathname with a supplied DIRECTORY component''.
}
at which point the presence of the trailing directory name separator
is necessary to tell Lisp that you're discussing a directory
rather than a file.

Typically, however, there are a lot of @file{.asd} files, and
a common idiom was to have to put
a bunch of @emph{symbolic links} to @file{.asd} files
in a common directory
and push @emph{that} directory (the ``link farm'')
to the
@code{asdf:*central-registry*}
instead of pushing each of the many involved directories
to the @code{asdf:*central-registry*}.
ASDF knows how to follow such @emph{symlinks}
to the actual file location when resolving the paths of system components
(on Windows, you can use Windows shortcuts instead of POSIX symlinks;
if you try aliases under MacOS, we are curious to hear about your experience).

For example, if @code{#p"/home/me/cl/systems/"} (note the trailing slash)
is a member of @code{*central-registry*}, you could set up the
system @var{foo} for loading with asdf with the following
commands at the shell:

@example
$ cd /home/me/cl/systems/
$ ln -s ~/src/foo/foo.asd .
@end example

This old style for configuring ASDF is not recommended for new users,
but it is supported for old users, and for users who want to programmatically
control what directories are added to the ASDF search path.


@section Configuring where ASDF stores object files
@findex clear-output-translations

ASDF lets you configure where object files will be stored.
Sensible defaults are provided and
you shouldn't normally have to worry about it.

This allows the same source code repository may be shared
between several versions of several Common Lisp implementations,
between several users using different compilation options
and without write privileges on shared source directories, etc.
This also allows to keep source directories uncluttered
by plenty of object files.

Starting with ASDF 2, the @code{asdf-output-translations} facility
was added to ASDF itself, that controls where object files will be stored.
This facility is fully described in a chapter of this manual,
@ref{Controlling where ASDF saves compiled files}.

The simplest way to add a translation to your search path,
say from @file{/foo/bar/baz/quux/}
to @file{/where/i/want/my/fasls/}
is to create the directory
@file{~/.config/common-lisp/asdf-output-translations.conf.d/}
and there create a file with any name of your choice and the type @file{conf},
for instance @file{42-bazquux.conf}
containing the line:

@kbd{("/foo/bar/baz/quux/" "/where/i/want/my/fasls/")}

To disable output translations for source under a given directory,
say @file{/toto/tata/}
you can create a file @file{40-disable-toto.conf}
with the line:

@kbd{("/toto/tata/")}

To wholly disable output translations for all directories,
you can create a file @file{00-disable.conf}
with the line:

@kbd{(t t)}

Note that your Operating System distribution or your system administrator
may already have configured translations for you.
In absence of any configuration, the default is to redirect everything
under an implementation-dependent subdirectory of @file{~/.cache/common-lisp/}.
@xref{Controlling where ASDF searches for systems}, for full details.

The required @file{.conf} extension allows you to have disabled files
or editor backups (ending in @file{~}), and works portably
(for instance, it is a pain to allow both empty and non-empty extension on CLISP).
Excluded are files the name of which start with a @file{.} character.
It is customary to start the filename with two digits
that specify the order in which the directories will be scanned.

ASDF will automatically read your configuration
the first time you try to find a system.
You can reset the source-registry configuration with:

@lisp
(asdf:clear-output-translations)
@end lisp

And you probably should do so before you dump your Lisp image,
if the configuration may change
between the machine where you save it at the time you save it
and the machine you resume it at the time you resume it.
(Once again, you should use @code{(asdf:clear-configuration)}
before you dump your Lisp image, which includes the above.)

Finally note that before ASDF 2,
other ASDF add-ons offered the same functionality,
each in subtly different and incompatible ways:
ASDF-Binary-Locations, cl-launch, common-lisp-controller.
ASDF-Binary-Locations is now not needed anymore and should not be used.
cl-launch 3.000 and common-lisp-controller 7.2 have been updated
to just delegate this functionality to ASDF.

@node Using ASDF, Defining systems with defsystem, Configuring ASDF, Top
@comment  node-name,  next,  previous,  up


@section Resetting Configuration

When you dump and restore an image, or when you tweak your configuration,
you may want to reset the ASDF configuration.
For that you may use the following function:

@defun clear-configuration
   undoes any ASDF configuration,
   regarding source-registry or output-translations.
@end defun

If you use SBCL, CMUCL or SCL, you may use this snippet
so that the ASDF configuration be cleared automatically as you dump an image:

@example
#+(or cmu sbcl scl)
(pushnew 'clear-configuration
         #+(or cmu scl) ext:*before-save-initializations*
         #+sbcl sb-ext:*save-hooks*)
@end example

For compatibility with all Lisp implementations, however,
you might want instead your build script to explicitly call
@code{(asdf:clear-configuration)} at an appropriate moment before dumping.


@chapter Using ASDF

@section Loading a system

The system @var{foo} is loaded (and compiled, if necessary)
by evaluating the following Lisp form:

@example
(asdf:load-system :@var{foo})
@end example

On some implementations (namely recent versions of
ABCL, Allegro CL, Clozure CL, CMUCL, ECL, GNU CLISP,
LispWorks, MKCL, SBCL and XCL),
ASDF hooks into the @code{CL:REQUIRE} facility
and you can just use:

@example
(require :@var{foo})
@end example

In older versions of ASDF, you needed to use
@code{(asdf:oos 'asdf:load-op :@var{foo})}.
If your ASDF is too old to provide @code{asdf:load-system} though
we recommend that you upgrade to ASDF 3.
@xref{Loading ASDF,,Loading an otherwise installed ASDF}.

Note the name of a system is specified as a string or a symbol,
typically a keyword.
If a symbol (including a keyword), its name is taken and lowercased.
The name must be a suitable value for the @code{:name} initarg
to @code{make-pathname} in whatever filesystem the system is to be found.
The lower-casing-symbols behaviour is unconventional,
but was selected after some consideration.
Observations suggest that the type of systems we want to support
either have lowercase as customary case (unix, mac, windows)
or silently convert lowercase to uppercase (lpns),
so this makes more sense than attempting to use @code{:case :common},
which is reported not to work on some implementations


@section Other Operations

ASDF provides three commands for the most common system operations:
@code{load-system}, @code{compile-system} or @code{test-system}.
It also provides @code{require-system}, a version of @code{load-system}
that skips trying to update systems that are already loaded.

Because ASDF is an extensible system
for defining @emph{operations} on @emph{components},
it also provides a generic function @code{operate}
(which is usually abbreviated by @code{oos}).
You'll use @code{oos} whenever you want to do something beyond
compiling, loading and testing.

Output from ASDF and ASDF extensions are supposed to be sent
to the CL stream @code{*standard-output*},
and so rebinding that stream around calls to @code{asdf:operate}
should redirect all output from ASDF operations.

Reminder: before ASDF can operate on a system, however,
it must be able to find and load that system's definition.
@xref{Configuring ASDF,,Configuring ASDF to find your systems}.

For the advanced users, note that
@code{require-system} calls @code{load-system}
with keyword arguments @code{:force-not (loaded-systems)}.
@code{loaded-systems} returns a list of the names of loaded systems.
@code{load-system} applies @code{operate} with the operation from
@code{*load-system-operation*}, which by default is @code{load-op},
the system, and any provided keyword arguments.

@section Summary

To use ASDF:

@itemize
@item
Load ASDF itself into your Lisp image, either through
@code{(require "asdf")} or else through
@code{(load "/path/to/asdf.lisp")}.

@item
Make sure ASDF can find system definitions
thanks to proper source-registry configuration.

@item
Load a system with @code{(asdf:load-system :my-system)}
or use some other operation on some system of your choice.

@end itemize

@section Moving on

That's all you need to know to use ASDF to load systems written by others.
The rest of this manual deals with writing system definitions
for Common Lisp software you write yourself,
including how to extend ASDF to define new operation and component types.


@node Defining systems with defsystem, The object model of ASDF, Using ASDF, Top
@comment  node-name,  next,  previous,  up
@chapter Defining systems with defsystem

This chapter describes how to use asdf to define systems and develop
software.


@menu
* The defsystem form::
* A more involved example::
* The defsystem grammar::
* Other code in .asd files::
@end menu

@node  The defsystem form, A more involved example, Defining systems with defsystem, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section The defsystem form

Systems can be constructed programmatically
by instantiating components using @code{make-instance}.
Most of the time, however, it is much more practical to use
a static @code{defsystem} form.
This section begins with an example of a system definition,
then gives the full grammar of @code{defsystem}.

Let's look at a simple system.
This is a complete file that would
usually be saved as @file{hello-lisp.asd}:

@lisp
(in-package :asdf)

(defsystem "hello-lisp"
  :description "hello-lisp: a sample Lisp system."
  :version "0.2.1"
  :author "Joe User <joe@@example.com>"
  :licence "Public Domain"
  :components ((:file "packages")
               (:file "macros" :depends-on ("packages"))
               (:file "hello" :depends-on ("macros"))))
@end lisp

Some notes about this example:

@itemize

@item
The file starts with an @code{in-package} form
to use package @code{asdf}.
You could instead start your definition by using
a qualified name @code{asdf:defsystem}.

@item
If in addition to simply using @code{defsystem},
you are going to define functions,
create ASDF extension, globally bind symbols, etc.,
it is recommended that to avoid namespace pollution between systems,
you should create your own package for that purpose,
for instance replacing the above @code{(in-package :asdf)} with:

@lisp
(defpackage :foo-system
  (:use :cl :asdf))

(in-package :foo-system)
@end lisp

@item
The @code{defsystem} form defines a system named @code{hello-lisp}
that contains three source files:
@file{packages}, @file{macros} and @file{hello}.

@item
The file @file{macros} depends on @file{packages}
(presumably because the package it's in is defined in @file{packages}),
and the file @file{hello} depends on @file{macros}
(and hence, transitively on @file{packages}).
This means that ASDF will compile and load @file{packages} and @file{macros}
before starting the compilation of file @file{hello}.

@item
The files are located in the same directory
as the file with the system definition.
ASDF resolves symbolic links (or Windows shortcuts)
before loading the system definition file and
stores its location in the resulting system@footnote{
It is possible, though almost never necessary, to override this behaviour.}.
This is a good thing because the user can move the system sources
without having to edit the system definition.

@c FIXME: Should have cross-reference to "Version specifiers" in the
@c defsystem grammar, but the cross-referencing is so broken by
@c insufficient node breakdown that I have not put one in.
@item
Make sure you know how the @code{:version} numbers will be parsed!
They are parsed as period-separated lists of integers.
I.e., in the example, @code{0.2.1} is to be interpreted,
roughly speaking, as @code{(0 2 1)}.
In particular, version @code{0.2.1}
is interpreted the same as @code{0.0002.1} and
is strictly version-less-than version @code{0.20.1},
even though the two are the same when interpreted as decimal fractions.
Instead of a string representing the version,
the @code{:version} argument can be an expression that is resolved to
such a string using the following trivial domain-specific language:
in addition to being a literal string, it can be an expression of the form
@code{(:read-file-form <pathname-or-string> :at <access-at-specifier>)},
which will be resolved by reading a form in the specified pathname
(read as a subpathname of the current system if relative or a unix-namestring).
You may use a @code{uiop:access-at} specifier
with the (optional) @code{:at} keyword,
by default the specifier is @code{0}, meaning the first form is returned.

@cindex :version

@end itemize

@node  A more involved example, The defsystem grammar, The defsystem form, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section A more involved example

Let's illustrate some more involved uses of @code{defsystem} via a
slightly convoluted example:

@lisp
(defsystem "foo"
  :version "1.0.0"
  :components ((:module "mod"
                            :components ((:file "bar")
                                                  (:file"baz")
                                                  (:file "quux"))
                            :perform (compile-op :after (op c)
                                                  (do-something c))
                            :explain (compile-op :after (op c)
                                            (explain-something c)))
                         (:file "blah")))
@end lisp

The @code{:module} component named @code{"mod"} is a collection of three files,
which will be located in a subdirectory of the main code directory named
@file{mod} (this location can be overridden; see the discussion of the
@code{:pathname} option in @ref{The defsystem grammar}).

The method-form tokens provide a shorthand for defining methods on
particular components.  This part

@lisp
                :perform (compile-op :after (op c)
                          (do-something c))
                :explain (compile-op :after (op c)
                          (explain-something c))
@end lisp

has the effect of

@lisp
(defmethod perform :after ((op compile-op) (c (eql ...)))
           (do-something c))
(defmethod explain :after ((op compile-op) (c (eql ...)))
           (explain-something c))
@end lisp

where @code{...} is the component in question.
In this case @code{...} would expand to something like

@lisp
(find-component (find-system "foo") "mod")
@end lisp

For more details on the syntax of such forms, see @ref{The defsystem
grammar}.
For more details on what these methods do, @pxref{Operations} in
@ref{The object model of ASDF}.

@c The following plunge into the weeds is not appropriate in this
@c location. [2010/10/03:rpg]
@c note that although this also supports @code{:before} methods,
@c they may not do what you want them to ---
@c a @code{:before} method on perform @code{((op compile-op) (c (eql ...)))}
@c will run after all the dependencies and sub-components have been processed,
@c but before the component in question has been compiled.

@node  The defsystem grammar, Other code in .asd files, A more involved example, Defining systems with defsystem
@comment  node-name,  next,  previous,  up
@section The defsystem grammar

@c FIXME: @var typesetting not consistently used here.  We should either expand
@c its use to everywhere, or we should kill it everywhere.


@example
system-definition := ( defsystem system-designator @var{system-option}* )

system-option := :defsystem-depends-on system-list
                 | :weakly-depends-on @var{system-list}
                 | :class class-name (see discussion below)
                 | module-option
                 | option

module-option := :components component-list
                 | :serial [ t | nil ]

option :=
        | :pathname pathname-specifier
        | :default-component-class class-name
        | :perform method-form
        | :explain method-form
        | :output-files method-form
        | :operation-done-p method-form
        | :if-feature feature-expression
        | :depends-on ( @var{dependency-def}* )
        | :in-order-to ( @var{dependency}+ )


system-list := ( @var{simple-component-name}* )

component-list := ( @var{component-def}* )

component-def  := ( component-type simple-component-name @var{option}* )

component-type := :module | :file | :static-file | other-component-type

other-component-type := symbol-by-name (@pxref{The defsystem grammar,,Component types})

dependency-def := simple-component-name
               | ( :feature name )
               | ( :version simple-component-name version-specifier)

dependency := (dependent-op @var{requirement}+)
requirement := (required-op @var{required-component}+)
             | (feature feature-name)
dependent-op := operation-name
required-op := operation-name | feature

simple-component-name := string
                      |  symbol

pathname-specifier := pathname | string | symbol

method-form := (operation-name qual lambda-list @Arest body)
qual := method qualifier

component-dep-fail-option := :fail | :try-next | :ignore

feature-expression := keyword | (:and @var{feature-expression}*)
                      | (:or @var{feature-expression}*) | (:not @var{feature-expression})
@end example


@subsection Component names

Component names (@code{simple-component-name})
may be either strings or symbols.

@subsection Component types

Component type names, even if expressed as keywords, will be looked up
by name in the current package and in the asdf package, if not found in
the current package.  So a component type @code{my-component-type}, in
the current package @code{my-system-asd} can be specified as
@code{:my-component-type}, or @code{my-component-type}.

@code{system} and its subclasses are @emph{not}
allowed as component types for such children components.

@subsection System class names

A system class name will be looked up
in the same way as a Component type (see above),
except that only @code{system} and its subclasses are allowed.
Typically, one will not need to specify a system
class name, unless using a non-standard system class defined in some
ASDF extension, typically loaded through @code{DEFSYSTEM-DEPENDS-ON},
see below.  For such class names in the ASDF package, we recommend that
the @code{:class} option be specified using a keyword symbol, such as

@example
:class :MY-NEW-SYSTEM-SUBCLASS
@end example

This practice will ensure that package name conflicts are avoided.
Otherwise, the symbol @code{MY-NEW-SYSTEM-SUBCLASS} will be read into
the current package @emph{before} it has been exported from the ASDF
extension loaded by @code{:defsystem-depends-on}, causing a name
conflict in the current package.

@subsection Defsystem depends on
@cindex :defsystem-depends-on

The @code{:defsystem-depends-on} option to @code{defsystem} allows the
programmer to specify another ASDF-defined system or set of systems that
must be loaded @emph{before} the system definition is processed.
Typically this is used to load an ASDF extension that is used in the
system definition.

@subsection Weakly depends on 
@cindex :weakly-depends-on

We do @emph{NOT} recommend you use this feature.
If you are tempted to write a system @var{foo}
that weakly-depends-on a system @var{bar},
we recommend that you should instead
write system @var{foo} in a parametric way,
and offer some special variable and/or some hook to specialize its behavior;
then you should write a system @var{foo+bar}
that does the hooking of things together.

The (deprecated) @code{:weakly-depends-on} option to @code{defsystem}
allows the programmer to specify another ASDF-defined system or set of systems
that ASDF should @emph{try} to load,
but need not load in order to be successful.
Typically this is used if there are a number of systems
that, if present, could provide additional functionality,
but which are not necessary for basic function.

Currently, although it is specified to be an option only to @code{defsystem},
this option is accepted at any component, but it probably
only makes sense at the @code{defsystem} level.
Programmers are cautioned not
to use this component option except at the @code{defsystem} level, as
this anomalous behavior may be removed without warning.

Finally, you might look into the @code{asdf-system-connections} extension,
that will let you define additional code to be loaded
when two systems are simultaneously loaded.
It may or may not be considered good style, but at least it can be used
in a way that has deterministic behavior independent of load order,
unlike @code{weakly-depends-on}.


@subsection Pathname specifiers
@cindex pathname specifiers

A pathname specifier (@code{pathname-specifier})
may be a pathname, a string or a symbol.
When no pathname specifier is given for a component,
which is the usual case, the component name itself is used.

If a string is given, which is the usual case,
the string will be interpreted as a Unix-style pathname
where @code{/} characters will be interpreted as directory separators.
Usually, Unix-style relative pathnames are used
(i.e. not starting with @code{/}, as opposed to absolute pathnames);
they are relative to the path of the parent component.
Finally, depending on the @code{component-type},
the pathname may be interpreted as either a file or a directory,
and if it's a file,
a file type may be added corresponding to the @code{component-type},
or else it will be extracted from the string itself (if applicable).

For instance, the @code{component-type} @code{:module}
wants a directory pathname, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar/"}.
On the other hand, the @code{component-type} @code{:file}
wants a file of type @code{lisp}, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar.lisp"},
and a string @code{"foo/bar.quux"}
will be interpreted as the pathname @file{#p"foo/bar.quux.lisp"}.
Finally, the @code{component-type} @code{:static-file}
wants a file without specifying a type, and so a string @code{"foo/bar"}
will be interpreted as the pathname @file{#p"foo/bar"},
and a string @code{"foo/bar.quux"}
will be interpreted as the pathname @file{#p"foo/bar.quux"}.

ASDF does not interpret the string @code{".."} to designate the parent
directory.  This string will be passed through to the underlying
operating system for interpretation.  We @emph{believe} that this will
work on all platforms where ASDF is deployed, but do not guarantee this
behavior.  A pathname object with a relative directory component of
@code{:up} or @code{:back} is the only guaranteed way to specify a
parent directory.

If a symbol is given, it will be translated into a string,
and downcased in the process.
The downcasing of symbols is unconventional,
but was selected after some consideration.
Observations suggest that the type of systems we want to support
either have lowercase as customary case (Unix, Mac, windows)
or silently convert lowercase to uppercase (lpns),
so this makes more sense than attempting to use @code{:case :common}
as argument to @code{make-pathname},
which is reported not to work on some implementations.

Pathname objects may be given to override the path for a component.
Such objects are typically specified using reader macros such as @code{#p}
or @code{#.(make-pathname ...)}.
Note however, that @code{#p...} is a shorthand for @code{#.(parse-namestring ...)}
and that the behavior of @code{parse-namestring} is completely non-portable,
unless you are using Common Lisp @code{logical-pathname}s
(@pxref{The defsystem grammar,,Using logical pathnames}, below).
Pathnames made with @code{#.(make-pathname ...)}
can usually be done more easily with the string syntax above.
The only case that you really need a pathname object is to override
the component-type default file type for a given component.
Therefore, pathname objects should only rarely be used.
Unhappily, ASDF 1 didn't properly support
parsing component names as strings specifying paths with directories,
and the cumbersome @code{#.(make-pathname ...)} syntax had to be used.
An alternative to @code{#.} read-time evaluation is to use
@code{(eval `(defsystem ... ,pathname ...))}.

Note that when specifying pathname objects,
ASDF does not do any special interpretation of the pathname
influenced by the component type, unlike the procedure for
pathname-specifying strings.
On the one hand, you have to be careful to provide a pathname that correctly
fulfills whatever constraints are required from that component type
(e.g. naming a directory or a file with appropriate type);
on the other hand, you can circumvent the file type that would otherwise
be forced upon you if you were specifying a string.

@subsection Version specifiers
@cindex version specifiers
@cindex :version

Version specifiers are strings to be parsed as period-separated lists of integers.
I.e., in the example, @code{"0.2.1"} is to be interpreted,
roughly speaking, as @code{(0 2 1)}.
In particular, version @code{"0.2.1"} is interpreted the same as @code{"0.0002.1"},
though the latter is not canonical and may lead to a warning being issued.
Also, @code{"1.3"} and @code{"1.4"} are both strictly @code{uiop:version<} to @code{"1.30"},
quite unlike what would have happened
had the version strings been interpreted as decimal fractions.

System definers are encouraged to use version identifiers of the form
@var{x}.@var{y}.@var{z} for
major version, minor version and patch level,
where significant API incompatibilities are signaled by an increased major number.

@xref{Common attributes of components}.


@subsection Using logical pathnames
@cindex logical pathnames

We do not generally recommend the use of logical pathnames,
especially not so to newcomers to Common Lisp.
However, we do support the use of logical pathnames by old timers,
when such is their preference.

To use logical pathnames,
you will have to provide a pathname object as a @code{:pathname} specifier
to components that use it, using such syntax as
@code{#p"LOGICAL-HOST:absolute;path;to;component.lisp"}.

You only have to specify such logical pathname
for your system or some top-level component.
Sub-components' relative pathnames,
specified using the string syntax for names,
will be properly merged with the pathnames of their parents.
The specification of a logical pathname host however is @emph{not}
otherwise directly supported in the ASDF syntax
for pathname specifiers as strings.

The @code{asdf-output-translation} layer will
avoid trying to resolve and translate logical pathnames.
The advantage of this is that
you can define yourself what translations you want to use
with the logical pathname facility.
The disadvantage is that if you do not define such translations,
any system that uses logical pathnames will behave differently under
asdf-output-translations than other systems you use.

If you wish to use logical pathnames you will have to configure the
translations yourself before they may be used.
ASDF currently provides no specific support
for defining logical pathname translations.

Note that the reasons we do not recommend logical pathnames are that
(1) there is no portable way to set up logical pathnames before they are used,
(2) logical pathnames are limited to only portably use
a single character case, digits and hyphens.
While you can solve the first issue on your own,
describing how to do it on each of fifteen implementations supported by ASDF
is more than we can document.
As for the second issue, mind that the limitation is notably enforced on SBCL,
and that you therefore can't portably violate the limitations
but must instead define some encoding of your own and add individual mappings
to name physical pathnames that do not fit the restrictions.
This can notably be a problem when your Lisp files are part of a larger project
in which it is common to name files or directories in a way that
includes the version numbers of supported protocols,
or in which files are shared with software written
in different programming languages where conventions include the use of
underscores, dots or CamelCase in pathnames.


@subsection Serial dependencies
@cindex serial dependencies

If the @code{:serial t} option is specified for a module,
ASDF will add dependencies for each child component,
on all the children textually preceding it.
This is done as if by @code{:depends-on}.

@lisp
:serial t
:components ((:file "a") (:file "b") (:file "c"))
@end lisp

is equivalent to

@lisp
:components ((:file "a")
             (:file "b" :depends-on ("a"))
             (:file "c" :depends-on ("a" "b")))
@end lisp


@subsection Source location

The @code{:pathname} option is optional in all cases for systems
defined via @code{defsystem},
and in the usual case the user is recommended not to supply it.

Instead, ASDF follows a hairy set of rules that are designed so that
@enumerate
@item
@code{find-system}
will load a system from disk
and have its pathname default to the right place.

@item
This pathname information will not be overwritten with
@code{*default-pathname-defaults*}
(which could be somewhere else altogether)
if the user loads up the @file{.asd} file into his editor
and interactively re-evaluates that form.
@end enumerate

If a system is being loaded for the first time,
its top-level pathname will be set to:

@itemize
@item
The host/device/directory parts of @code{*load-truename*},
if it is bound.
@item
@code{*default-pathname-defaults*}, otherwise.
@end itemize

If a system is being redefined, the top-level pathname will be

@itemize
@item
changed, if explicitly supplied or obtained from @code{*load-truename*}
(so that an updated source location is reflected in the system definition)

@item
changed if it had previously been set from @code{*default-pathname-defaults*}

@item
left as before, if it had previously been set from @code{*load-truename*}
and @code{*load-truename*} is currently unbound
(so that a developer can evaluate a @code{defsystem} form
from within an editor without clobbering its source location)
@end itemize

@subsection if-feature option
This option allows you to specify a feature expression to be evaluated
as if by @code{#+} to conditionally include a component in your build.
If the expression is false, the component is dropped
as well as any dependency pointing to it.
As compared to using @code{#+} which is expanded at read-time,
this allows you to have an object in your component hierarchy
that can be used for manipulations beside building your project.
This option was added in ASDF 3.

@subsection if-component-dep-fails option
This option was removed in ASDF 3.
Its semantics was limited in purpose and dubious to explain,
and its implementation was breaking a hole into the ASDF object model.
Please use the @code{if-feature} option instead.

@node Other code in .asd files,  , The defsystem grammar, Defining systems with defsystem
@section Other code in .asd files

Files containing @code{defsystem} forms
are regular Lisp files that are executed by @code{load}.
Consequently, you can put whatever Lisp code you like into these files.
However, it is recommended to keep such forms to a minimal,
and to instead define @code{defsystem} extensions
that you use with @code{:defsystem-depends-on}.

If however, you might insist on including code in the @code{.asd} file itself,
e.g., to examine and adjust the compile-time environment,
possibly adding appropriate features to @code{*features*}.
If so, here are some conventions we recommend you follow,
so that users can control certain details of execution
of the Lisp in @file{.asd} files:

@itemize
@item
Any informative output
(other than warnings and errors,
which are the condition system's to dispose of)
should be sent to the standard CL stream @code{*standard-output*},
so that users can easily control the disposition
of output from ASDF operations.
@end itemize


@node The object model of ASDF, Controlling where ASDF searches for systems, Defining systems with defsystem, Top
@comment  node-name,  next,  previous,  up
@chapter The object model of ASDF

ASDF is designed in an object-oriented way from the ground up.
Both a system's structure and the operations that can be performed on systems
follow a extensible protocol.

This allows the addition of behaviours:
for example, @code{cffi} adds support of special FFI description files
to interface with C libraries and of wrapper files to embed C code in Lisp;
@code{abcl-jar} supports creating Java JAR archives in ABCL;
and @code{poiu} supports for compiling code in parallel using background processes.

This chapter deals with @code{component}s and @code{operation}s.

A @code{component} represents an individual source file or a group of source files,
and the things that get transformed into.
A @code{system} is a component at the top level of the component hierarchy.
A @code{source-file} is a component representing a single source-file
and the successive output files into which it is transformed.
A @code{module} is an intermediate component itself grouping several other components,
themselves source-files or further modules.

An @code{Operation} represents a transformation that can be performed on a component,
turning them from source files to intermediate results to final outputs.

A pair of an @code{operation} and a @code{component} is called an @code{action}.
An @code{action} represents a particular build step to be @code{perform}ed,
after all its dependencies have been fulfilled.
In the ASDF model, actions depend on other actions.
The term @emph{action} itself was used by Kent Pitman in his old article,
but was only used by ASDF hackers starting with the ASDF 2;
but the concept is ubiquitous since the very beginning of ASDF 1,
though previously implicit.

Then, there are many @emph{functions} available
to users, extenders and implementers of ASDF
to use, define or implement the activities
that are part of building your software.
Though they manipulate @code{action}s,
most of these functions do not take as an argument
a reified pair (a CONS cell) of an operation and a component;
instead, they usually take two separate arguments,
which allows to take advantage of the power CLOS-style multiple dispatch
for fun and profit.

There are many @emph{hooks} in which to add functionality,
by customizing the behavior of existing @emph{functions}.

Last but not least is the notion of @emph{dependency} between two actions.
The structure of dependencies between actions is
a directed @emph{dependency graph}.
ASDF is invoked by being told to @emph{operate}
with some @emph{operation} on some toplevel @emph{system};
it will then @emph{traverse} the graph and build a @emph{plan}
that follows its structure.
To be successfully buildable, this graph of actions but be acyclic.
If, as a user, extender or implementer of ASDF, you fail
to keep the dependency graph without cycles,
ASDF will fail loudly as it eventually finds one.
To clearly distinguish the direction of dependencies,
ASDF 3 uses the words @emph{requiring} and @emph{required}
as applied to an action depending on the other:
the requiring action @code{depends-on} the completion of all required actions
before it may itself be @code{perform}ed.

Using the @code{defsystem} syntax, users may easily express
direct dependencies along the graph of the object hierarchy:
between a component and its parent, its children, and its siblings.
By defining custom CLOS methods, you can express more elaborate dependencies as you wish.
Most common operations, such as @code{load-op}, @code{compile-op} or @code{load-source-op}
are automatically propagate ``downward'' the component hierarchy and are ``covariant'' with it:
to act the operation on the parent module, you must first act it on all the children components,
with the action on the parent being parent of the action on each child.
Other operations, such as @code{prepare-op} and @code{prepare-source-op}
(introduced in ASDF 3) are automatically propagated ``upward'' the component hierarchy
and are ``contravariant'' with it:
to perform the operation of preparing for compilation of a child component,
you must perform the operation of preparing for compilation of its parent component, and so on,
ensuring that all the parent's dependencies are (compiled and) loaded
before the child component may be compiled and loaded.
Yet other operations, such as @code{test-op} or @code{load-fasl-op}
remain at the system level, and are not propagated along the hierarchy,
but instead do something global on the system.

@menu
* Operations::
* Components::
* Functions::
@end menu

@node  Operations, Components, The object model of ASDF, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Operations
@cindex operation

An @dfn{operation} object of the appropriate type is instantiated
whenever the user wants to do something with a system like

@itemize
@item compile all its files
@item load the files into a running lisp environment
@item copy its source files somewhere else
@end itemize

Operations can be invoked directly, or examined
to see what their effects would be without performing them.
There are a bunch of methods specialised on operation and component type
that actually do the grunt work.

The operation object contains whatever state is relevant for this purpose
(perhaps a list of visited nodes, for example)
but primarily is a nice thing to specialise operation methods on
and easier than having them all be @code{EQL} methods.

Operations are invoked on systems via @code{operate}.
@anchor{operate}
@deffn {Generic function} @code{operate} @var{operation} @var{system} @Arest @var{initargs} @Akey @code{force} @code{force-not} @code{verbose} @AallowOtherKeys
@deffnx {Generic function} @code{oos} @var{operation} @var{system} @Arest @var{initargs} @Akey @AallowOtherKeys
@code{operate} invokes @var{operation} on @var{system}.
@code{oos} is a synonym for @code{operate}.

@var{operation} is a symbol that is passed, along with the supplied
@var{initargs}, to @code{make-instance} to create the operation object.
@var{system} is a system designator.

The @var{initargs} are passed to the @code{make-instance} call
when creating the operation object.
Note that dependencies may cause the operation
to invoke other operations on the system or its components:
the new operations will be created
with the same @var{initargs} as the original one.

If @var{force} is @code{:all}, then all systems
are forced to be recompiled even if not modified since last compilation.
If @var{force} is @code{t}, then only the system being loaded
is forced to be recompiled even if not modified since last compilation,
but other systems are not affected.
If @var{force} is a list, then it specifies a list of systems that
are forced to be recompiled even if not modified since last compilation.
If @var{force-not} is @code{:all}, then all systems
are forced not to be recompiled even if modified since last compilation.
If @var{force-not} is @code{t}, then only the system being loaded
is forced not to be recompiled even if modified since last compilation,
but other systems are not affected.
If @var{force-not} is a list, then it specifies a list of systems that
are forced not to be recompiled even if modified since last compilation.
@var{force} takes precedences over @var{force-not};
both of them apply to systems that are dependencies and were already compiled.

To see what @code{operate} would do, you can use:
@example
(asdf::traverse (make-instance operation-class initargs ...) (find-system system-name))
@end example

@end deffn

@menu
* Predefined operations of ASDF::
* Creating new operations::
@end menu

@node Predefined operations of ASDF, Creating new operations, Operations, Operations
@comment  node-name,  next,  previous,  up
@subsection Predefined operations of ASDF

All the operations described in this section are in the @code{asdf} package.
They are invoked via the @code{operate} generic function.

@lisp
(asdf:operate 'asdf:@var{operation-name} :@var{system-name} @{@var{operation-options ...}@})
@end lisp

@deffn Operation @code{compile-op} @Akey @code{proclamations}

This operation compiles the specified component.
If proclamations are supplied, they will be proclaimed.
This is a good place to specify optimization settings.

When creating a new component type,
you should provide methods for @code{compile-op}.

When @code{compile-op} is invoked,
component dependencies often cause some parts of the system
to be loaded as well as compiled.
Invoking @code{compile-op}
does not necessarily load all the parts of the system, though;
use @code{load-op} to load a system.
@end deffn

@deffn Operation @code{load-op} @Akey @code{proclamations}

This operation loads a system.

The default methods for @code{load-op} compile files before loading them.
For parity, your own methods on new component types should probably do so too.
@end deffn

@deffn Operation @code{parent-load-op} @Akey @code{proclamations}

This operation ensures that the dependencies
of a module, and its parent, and so on, are loaded (as per @code{load-op})
before the components within that module may be operated upon.

By default, all operations depend on this @code{parent-operation}
for actions on components to depend on this ``parent operation'' being acted on the parent.

The default methods for @code{load-op} compile files before loading them.
For parity, your own methods on new component types should probably do so too.
@end deffn

@deffn Operation @code{load-source-op}

This operation will load the source for the files in a module
even if the source files have been compiled.
Systems sometimes have knotty dependencies
which require that sources are loaded
before they can be compiled.
This is how you do that.

If you are creating a component type,
you need to implement this operation --- at least, where meaningful.
@end deffn

@anchor{test-op}
@deffn Operation @code{test-op}

This operation will perform some tests on the module.
The default method will do nothing.
The default dependency is to require
@code{load-op} to be performed on the module first.
The default @code{operation-done-p} is that the operation is @emph{never} done
---
we assume that if you invoke the @code{test-op},
you want to test the system, even if you have already done so.

The results of this operation are not defined by ASDF.
It has proven difficult to define how the test operation
should signal its results to the user
in a way that is compatible with all of the various test libraries
and test techniques in use in the community.

People typically define @code{test-op} methods like thus:
@example
(defmethod perform ((o asdf:test-op) (s (eql (asdf:find-system @var{:mysystem}))))
  (asdf:load-system @var{:mysystem})
  (eval (read-from-string "(some expression that runs the tests)"))
  t)
@end example
@end deffn

@deffn Operation @code{load-fasl-op}

This operation will load and create if need be
a single fasl file for all the files in each loaded system.
(Its compilation-only equivalent is @code{asdf::fasl-op}.)

Once you have created such a fasl,
you can use @code{precompiled-system} to deliver it in a way
that is compatible with clients having asdf dependencies
on your system whether it is distributed as source of as a single binary.

On your build platform, you run something like that:
@example
@code{(asdf:operate 'load-fasl-op @var{:mysystem})}
@end example

And on your delivery platform, a form like this is evaluated
in a prologue or at some point before you save your image:
@example
(defsystem :mysystem :class :precompiled-system
    :fasl (some expression that will evaluate to a pathname))
@end example

Of course, @emph{before} you define such systems,
you should not forget to @code{(asdf:clear-configuration)}.

@code{load-fasl-op} is available on all actively supported Lisp implementations,
and on those implementations only, and only since ASDF 3.
This functionality was previously available for select implementations,
as part of a separate system @code{asdf-bundle},
itself descended from @code{asdf-ecl}.
@end deffn


@node  Creating new operations,  , Predefined operations of ASDF, Operations
@comment  node-name,  next,  previous,  up
@subsection Creating new operations

ASDF was designed to be extensible in an object-oriented fashion.
To teach ASDF new tricks, a programmer can implement the behaviour he wants
by creating a subclass of @code{operation}.

ASDF's pre-defined operations are in no way ``privileged'',
but it is requested that developers never use the @code{asdf} package
for operations they develop themselves.
The rationale for this rule is that we don't want to establish a
``global asdf operation name registry'',
but also want to avoid name clashes.

An operation must provide methods for the following generic functions
when invoked with an object of type @code{source-file}:
@emph{FIXME describe this better}

@itemize

@item @code{input-files}
ASDF has a pretty clever default @code{input-files} mechanism.
You only need create a method if there are multiple ultimate input files,
and/or the bottom one doesn't depend
on the @code{component-pathname} of the component.

@item @code{output-files}
The @code{output-files} method determines where the method will put its files.
It returns two values, a list of pathnames, and a boolean.
If the boolean is @code{T} then the pathnames are marked
not be translated by enclosing @code{:around} methods.
If the boolean is @code{NIL} then enclosing @code{:around} methods
may translate these pathnames, e.g. to ensure object files
are somehow stored in some implementation-dependent cache.

@item @code{perform}
The @code{perform} method must call @code{output-files}
to find out where to put its files,
because the user is allowed to override.
@item @code{output-files}
for local policy @code{explain}

@item @code{operation-done-p}
You only need to define a method on that function
if you can detect conditions that invalidate previous runs of the operation,
even though no filesystem timestamp has changed,
in which case you return @code{nil} (the default is @code{t}).

For instance, the method for @code{test-op} always returns @code{nil},
so that tests are always run afresh.
Of course, the @code{test-op} for your system could depend
on a deterministically repeatable @code{test-report-op},
and just read the results from the report files.

@item @code{component-depends-on}
When you add new operations, you probably need to explain
how they relate to loading, compiling, testing, etc.,
in terms of dependencies between actions.

That's where you typically define methods on @code{component-depends-on}.
Your method will take as arguments
some properly specialized operation
and a component denoting a current action,
and return a list of entries,
denoting the children actions that the current action depends on.
The format of entries is described below.

It is @emph{strongly} advised that
you should always append the results of @code{(call-next-method)}
to the results of your method,
or ``interesting'' failures will likely occur,
unless you're a true specialist of ASDF internals.

Each entry returned by @code{component-depends-on} is itself a list.

The first element of an entry is the name of an operation:
a symbol that you can use with @code{make-instance}
(ASDF will instead use with @code{asdf::make-sub-operation}),
to create a related operation for use in a build plan.
For instance, @code{load-op} and @code{compile-op}
are common such names, denoting the respective operations.

The rest of an entry is a list of identifiers
each denote a component such that
the pair of the previous operation and this component
is a children action of current action.

Identifiers follow the @code{defsystem} grammar
previously documented.
The main format for identifiers is a string or symbol
(that will be downcase as per @code{coerce-name}),
and looked up against the sibling list of the parent module's children components,
as per @code{find-component}.
As a special case, @code{nil} denotes the parent itself.
Other syntaxes are allowed, for instance to specify a component with a version.

@end itemize

Operations that print output should send that output to the standard
CL stream @code{*standard-output*}, as the Lisp compiler and loader do.

@node Components, Functions, Operations, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Components
@cindex component
@cindex system
@cindex system designator
@vindex *system-definition-search-functions*

A @dfn{component} represents a source file or
(recursively) a collection of components.
A @dfn{system} is (roughly speaking) a top-level component
that can be found via @code{find-system}.

A @dfn{system designator} is a string or symbol
and behaves just like any other component name
(including with regard to the case conversion rules for component names).


@defun find-system system-designator &optional (error-p t)

Given a system designator, @code{find-system} finds and returns a system.
If no system is found, an error of type
@code{missing-component} is thrown,
or @code{nil} is returned if @code{error-p} is false.

To find and update systems, @code{find-system} funcalls each element
in the @code{*system-definition-search-functions*} list,
expecting a pathname to be returned, or a system object,
from which a pathname may be extracted, and that will be registered.
The resulting pathname (if any) is loaded
if one of the following conditions is true:

@itemize
@item
there is no system of that name in memory
@item
the pathname is different from that which was previously loaded
@item
the file's @code{last-modified} time exceeds the @code{last-modified} time
of the system in memory
@end itemize

When system definitions are loaded from @file{.asd} files,
a new scratch package is created for them to load into,
so that different systems do not overwrite each others operations.
The user may also wish to (and is recommended to)
include @code{defpackage} and @code{in-package} forms
in his system definition files, however,
so that they can be loaded manually if need be.

The default value of @code{*system-definition-search-functions*}
is a list of two functions.
The first function looks in each of the directories given
by evaluating members of @code{*central-registry*}
for a file whose name is the name of the system and whose type is @file{asd}.
The first such file is returned,
whether or not it turns out to actually define the appropriate system.
The second function does something similar,
for the directories specified in the @code{source-registry}.
Hence, it is strongly advised to define a system
@var{foo} in the corresponding file @var{foo.asd}.
@end defun


@menu
* Common attributes of components::
* Pre-defined subclasses of component::
* Creating new component types::
@end menu

@node  Common attributes of components, Pre-defined subclasses of component, Components, Components
@comment  node-name,  next,  previous,  up
@subsection Common attributes of components

All components, regardless of type, have the following attributes.
All attributes except @code{name} are optional.

@subsubsection Name

A component name is a string or a symbol.
If a symbol, its name is taken and lowercased.

Unless overridden by a @code{:pathname} attribute,
the name will be interpreted as a pathname specifier according
to a Unix-style syntax.
@xref{The defsystem grammar,,Pathname specifiers}.

@subsubsection Version identifier
@findex version-satisfies
@cindex :version

This optional attribute specifies a version for the current component.
The version should typically be a string of integers separated by dots,
for example @samp{1.0.11}.
For more information on version specifiers, see @ref{The defsystem grammar}.

A version may then be queried by the generic function @code{version-satisfies},
to see if @code{:version} dependencies are satisfied,
and when specifying dependencies, a constraint of minimal version to satisfy
can be specified using e.g. @code{(:version "mydepname" "1.0.11")}.

Note that in the wild, we typically see version numbering
only on components of type @code{system}.
Presumably it is much less useful within a given system,
wherein the library author is responsible to keep the various files in synch.

@subsubsection Required features

Traditionally defsystem users have used @code{#+} reader conditionals
to include or exclude specific per-implementation files.
This means that any single implementation cannot read the entire system,
which becomes a problem if it doesn't wish to compile it,
but instead for example to create an archive file containing all the sources,
as it will omit to process the system-dependent sources for other systems.

Each component in an asdf system may therefore specify using @code{:if-feature}
a feature expression using the same syntax as @code{#+} does,
such that any reference to the component will be ignored
during compilation, loading and/or linking if the expression evaluates to false.
Since the expression is read by the normal reader,
you must explicitly prefix your symbols with @code{:} so they be read as keywords;
this is as contrasted with the @code{#+} syntax
that implicitly reads symbols in the keyword package by default.

For instance, @code{:if-feature (:and :x86 (:or :sbcl :cmu :scl))} specifies that
the given component is only to be compiled and loaded
when the implementation is SBCL, CMUCL or Scieneer CL on an x86 machine.
You can not write it as @code{:if-feature (and x86 (or sbcl cmu scl))}
since the symbols would presumably fail to be read as keywords.

@subsubsection Dependencies

This attribute specifies dependencies of the component on its siblings.
It is optional but often necessary.

There is an excitingly complicated relationship between the initarg
and the method that you use to ask about dependencies

Dependencies are between (operation component) pairs.
In your initargs for the component, you can say

@lisp
:in-order-to ((compile-op (load-op "a" "b") (compile-op "c"))
              (load-op (load-op "foo")))
@end lisp

This means the following things:
@itemize
@item
before performing compile-op on this component, we must perform
load-op on @var{a} and @var{b}, and compile-op on @var{c},
@item
before performing @code{load-op}, we have to load @var{foo}
@end itemize

The syntax is approximately

@verbatim
(this-op @{(other-op required-components)@}+)

simple-component-name := string
                      |  symbol

required-components := simple-component-name
                     | (required-components required-components)

component-name := simple-component-name
                | (:version simple-component-name minimum-version-object)
@end verbatim

Side note:

This is on a par with what ACL defsystem does.
mk-defsystem is less general: it has an implied dependency

@verbatim
  for all source file x, (load x) depends on (compile x)
@end verbatim

and using a @code{:depends-on} argument to say that @var{b} depends on
@var{a} @emph{actually} means that

@verbatim
  (compile b) depends on (load a)
@end verbatim

This is insufficient for e.g. the McCLIM system, which requires that
all the files are loaded before any of them can be compiled ]

End side note

In ASDF, the dependency information for a given component and operation
can be queried using @code{(component-depends-on operation component)},
which returns a list

@lisp
((load-op "a") (load-op "b") (compile-op "c") ...)
@end lisp

@code{component-depends-on} can be subclassed for more specific
component/operation types: these need to @code{(call-next-method)}
and append the answer to their dependency, unless
they have a good reason for completely overriding the default dependencies.

If it weren't for CLISP, we'd be using @code{LIST} method
combination to do this transparently.
But, we need to support CLISP.
If you have the time for some CLISP hacking,
I'm sure they'd welcome your fixes.
@c Doesn't CLISP now support LIST method combination?

A minimal version can be specified for a component you depend on
(typically another system), by specifying @code{(:version "other-system" "1.2.3")}
instead of simply @code{"other-system"} as the dependency.
See the discussion of the semantics of @code{:version}
in the defsystem grammar.

@c FIXME: Should have cross-reference to "Version specifiers" in the
@c defsystem grammar, but the cross-referencing is so broken by
@c insufficient node breakdown that I have not put one in.


@subsubsection pathname

This attribute is optional and if absent (which is the usual case),
the component name will be used.

@xref{The defsystem grammar,,Pathname specifiers},
for an explanation of how this attribute is interpreted.

Note that the @code{defsystem} macro (used to create a ``top-level'' system)
does additional processing to set the filesystem location of
the top component in that system.
This is detailed elsewhere. @xref{Defining systems with defsystem}.


@subsubsection properties

This attribute is optional.

Packaging systems often require information about files or systems
in addition to that specified by ASDF's pre-defined component attributes.
Programs that create vendor packages out of ASDF systems therefore
have to create ``placeholder'' information to satisfy these systems.
Sometimes the creator of an ASDF system may know the additional
information and wish to provide it directly.

@code{(component-property component property-name)} and
associated @code{setf} method will allow
the programmatic update of this information.
Property names are compared as if by @code{EQL},
so use symbols or keywords or something.

@menu
* Pre-defined subclasses of component::
* Creating new component types::
@end menu

@node Pre-defined subclasses of component, Creating new component types, Common attributes of components, Components
@comment  node-name,  next,  previous,  up
@subsection Pre-defined subclasses of component

@deffn Component source-file

A source file is any file that the system does not know how to
generate from other components of the system.

Note that this is not necessarily the same thing as
``a file containing data that is typically fed to a compiler''.
If a file is generated by some pre-processor stage
(e.g. a @file{.h} file from @file{.h.in} by autoconf)
then it is not, by this definition, a source file.
Conversely, we might have a graphic file
that cannot be automatically regenerated,
or a proprietary shared library that we received as a binary:
these do count as source files for our purposes.

Subclasses of source-file exist for various languages.
@emph{FIXME: describe these.}
@end deffn

@deffn Component module

A module is a collection of sub-components.

A module component has the following extra initargs:

@itemize
@item
@code{:components} the components contained in this module

@item
@code{:default-component-class}
All children components which don't specify their class explicitly
are inferred to be of this type.

@item
@code{:if-component-dep-fails}
This attribute was removed in ASDF 3. Do not use it.
Use @code{:if-feature} instead.

@item
@code{:serial} When this attribute is set,
each subcomponent of this component is assumed to depend on all subcomponents
before it in the list given to @code{:components}, i.e.
all of them are loaded before a compile or load operation is performed on it.

@end itemize

The default operation knows how to traverse a module, so
most operations will not need to provide methods specialised on modules.

@code{module} may be subclassed to represent components such as
foreign-language linked libraries or archive files.
@end deffn

@deffn Component system

@code{system} is a subclass of @code{module}.

A system is a module with a few extra attributes for documentation
purposes; these are given elsewhere.
@xref{The defsystem grammar}.

Users can create new classes for their systems:
the default @code{defsystem} macro takes a @code{:class} keyword argument.
@end deffn

@node  Creating new component types,  , Pre-defined subclasses of component, Components
@comment  node-name,  next,  previous,  up
@subsection Creating new component types

New component types are defined by subclassing one of the existing
component classes and specializing methods on the new component class.

@emph{FIXME: this should perhaps be explained more throughly,
not only by example ...}

As an example, suppose we have some implementation-dependent
functionality that we want to isolate
in one subdirectory per Lisp implementation our system supports.
We create a subclass of
@code{cl-source-file}:

@lisp
(defclass unportable-cl-source-file (cl-source-file)
  ())
@end lisp

Function @code{asdf:implementation-type} (exported since 2.014.14)
gives us the name of the subdirectory.
All that's left is to define how to calculate the pathname
of an @code{unportable-cl-source-file}.

@lisp
(defmethod component-pathname ((component unportable-cl-source-file))
  (merge-pathnames*
   (coerce-pathname (format nil "~(~A~)/" (asdf:implementation-type)))
   (call-next-method)))
@end lisp

The new component type is used in a @code{defsystem} form in this way:

@lisp
(defsystem :foo
    :components
    ((:file "packages")
     ...
     (:unportable-cl-source-file "threads"
      :depends-on ("packages" ...))
     ...
    )
@end lisp

@node Functions,  , Components, The object model of ASDF
@comment  node-name,  next,  previous,  up
@section Functions
@findex version-satisfies

@deffn version-satisfies @var{version} @var{version-spec}
Does @var{version} satisfy the @var{version-spec}.  A generic function.
ASDF provides built-in methods for @var{version} being a @code{component} or @code{string}.
@var{version-spec} should be a string.
If it's a component, its version is extracted as a string before further processing.

A version string satisfies the version-spec if after parsing,
the former is no older than the latter.
Therefore @code{"1.9.1"}, @code{"1.9.2"} and @code{"1.10"} all satisfy @code{"1.9.1"},
but @code{"1.8.4"} or @code{"1.9"} do not.
For more information about how @code{version-satisfies} parses and interprets
version strings and specifications,
@pxref{The defsystem grammar} (version specifiers) and
@ref{Common attributes of components}.

Note that in versions of ASDF prior to 3.0.1,
including the entire ASDF 1 and ASDF 2 series,
@code{version-satisfies} would also require that the version and the version-spec
have the same major version number (the first integer in the list);
if the major version differed, the version would be considered as not matching the spec.
But that feature was not documented, therefore presumably not relied upon,
whereas it was a nuisance to several users.
Starting with ASDF 3.0.1,
@code{version-satisfies} does not treat the major version number specially,
and returns T simply if the first argument designates a version that isn't older
than the one specified as a second argument.
If needs be, the @code{(:version ...)} syntax for specifying dependencies
could be in the future extended to specify an exclusive upper bound for compatible versions
as well as an inclusive lower bound.
@end deffn

@node Controlling where ASDF searches for systems, Controlling where ASDF saves compiled files, The object model of ASDF, Top
@comment  node-name,  next,  previous,  up
@chapter Controlling where ASDF searches for systems

@section Configurations

Configurations specify paths where to find system files.

@enumerate

@item
The search registry may use some hardcoded wrapping registry specification.
This allows some implementations (notably SBCL) to specify where to find
some special implementation-provided systems that
need to precisely match the version of the implementation itself.

@item
An application may explicitly initialize the source-registry configuration
using the configuration API
(@pxref{Controlling where ASDF searches for systems,Configuration API,Configuration API}, below)
in which case this takes precedence.
It may itself compute this configuration from the command-line,
from a script, from its own configuration file, etc.

@item
The source registry will be configured from
the environment variable @code{CL_SOURCE_REGISTRY} if it exists.

@item
The source registry will be configured from
user configuration file
@file{$XDG_CONFIG_DIRS/common-lisp/source-registry.conf}
(which defaults to
@file{~/.config/common-lisp/source-registry.conf})
if it exists.

@item
The source registry will be configured from
user configuration directory
@file{$XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d/}
(which defaults to
@file{~/.config/common-lisp/source-registry.conf.d/})
if it exists.

@item
The source registry will be configured from
system configuration file
@file{/etc/common-lisp/source-registry.conf}
if it exists/

@item
The source registry will be configured from
system configuration directory
@file{/etc/common-lisp/source-registry.conf.d/}
if it exists.

@item
The source registry will be configured from a default configuration.
This configuration may allow for implementation-specific systems
to be found, for systems to be found the current directory
(at the time that the configuration is initialized) as well as
@code{:directory} entries for @file{$XDG_DATA_DIRS/common-lisp/systems/} and
@code{:tree} entries for @file{$XDG_DATA_DIRS/common-lisp/source/}.
For instance, SBCL will include directories for its contribs
when it can find them; it will look for them where SBCL was installed,
or at the location specified by the @code{SBCL_HOME} environment variable.

@end enumerate

Each of these configurations is specified as an s-expression
in a trivial domain-specific language (defined below).
Additionally, a more shell-friendly syntax is available
for the environment variable (defined yet below).

Each of these configurations is only used if the previous
configuration explicitly or implicitly specifies that it
includes its inherited configuration.

Additionally, some implementation-specific directories
may be automatically prepended to whatever directories are specified
in configuration files, no matter if the last one inherits or not.


@section Truenames and other dangers

One great innovation of the original ASDF was its ability to leverage
@code{CL:TRUENAME} to locate where your source code was and where to build it,
allowing for symlink farms as a simple but effective configuration mechanism
that is easy to control programmatically.
ASDF 3 still supports this configuration style, and it is enabled by default;
however we recommend you instead use
our source-registry configuration mechanism described below,
because it is easier to setup in a portable way across users and implementations.

Addtionally, some people dislike truename,
either because it is very slow on their system, or
because they are using content-addressed storage where the truename of a file
is related to a digest of its individual contents,
and not to other files in the same intended project.
For these people, ASDF 3 allows to eschew the @code{TRUENAME} mechanism,
by setting the variable @var{asdf:*resolve-symlinks*} to @code{NIL}.

PS: Yes, if you haven't read Vernor Vinge's short but great classic
``True Names... and Other Dangers'' then you're in for a treat.


@section XDG base directory

Note that we purport to respect the XDG base directory specification
as to where configuration files are located,
where data files are located,
where output file caches are located.
Mentions of XDG variables refer to that document.

@url{http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html}

This specification allows the user to specify some environment variables
to customize how applications behave to his preferences.

On Windows platforms, when not using Cygwin,
instead of the XDG base directory specification,
we try to use folder configuration from the registry regarding
@code{Common AppData} and similar directories.
Since support for querying the Windows registry
is not possible to do in reasonable amounts of portable Common Lisp code,
ASDF 3 relies on the environment variables that Windows usually exports.

@section Backward Compatibility

For backward compatibility as well as to provide a practical backdoor for hackers,
ASDF will first search for @code{.asd} files in the directories specified in
@code{asdf:*central-registry*}
before it searches in the source registry above.

@xref{Configuring ASDF,,Configuring ASDF to find your systems --- old style}.

By default, @code{asdf:*central-registry*} will be empty.

This old mechanism will therefore not affect you if you don't use it,
but will take precedence over the new mechanism if you do use it.

@section Configuration DSL

Here is the grammar of the s-expression (SEXP) DSL for source-registry
configuration:

@c FIXME: This is too wide for happy compilation into pdf.

@example
;; A configuration is a single SEXP starting with keyword :source-registry
;; followed by a list of directives.
CONFIGURATION := (:source-registry DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of either of these:
    :inherit-configuration | ; splices inherited configuration (often specified last)
    :ignore-inherited-configuration | ; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap a
    ;; the newer required ASDF from an older release that doesn't sport said features:
    :ignore-invalid-entries | ; drops subsequent invalid entries instead of erroring out

    ;; add a single directory to be scanned (no recursion)
    (:directory DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; add a directory hierarchy, recursing but excluding specified patterns
    (:tree DIRECTORY-PATHNAME-DESIGNATOR) |

    ;; override the defaults for exclusion patterns
    (:exclude EXCLUSION-PATTERN ...) |
    ;; augment the defaults for exclusion patterns
    (:also-exclude EXCLUSION-PATTERN ...) |
    ;; Note that the scope of a an exclude pattern specification is
    ;; the rest of the current configuration expression or file.

    ;; splice the parsed contents of another config file
    (:include REGULAR-FILE-PATHNAME-DESIGNATOR) |

    ;; This directive specifies that some default must be spliced.
    :default-registry

REGULAR-FILE-PATHNAME-DESIGNATOR := PATHNAME-DESIGNATOR ;; interpreted as a file
DIRECTORY-PATHNAME-DESIGNATOR := PATHNAME-DESIGNATOR ;; interpreted as a directory name

PATHNAME-DESIGNATOR :=
    NIL | ;; Special: skip this entry.
    ABSOLUTE-COMPONENT-DESIGNATOR ;; see pathname DSL

EXCLUSION-PATTERN := a string without wildcards, that will be matched exactly
        against the name of a any subdirectory in the directory component
        of a path. e.g. @code{"_darcs"} will match @file{#p"/foo/bar/_darcs/src/bar.asd"}
@end example

Pathnames are designated using another DSL,
shared with the output-translations configuration DSL below.
The DSL is resolved by the function @code{asdf::resolve-location},
to be documented and exported at some point in the future.

@example
ABSOLUTE-COMPONENT-DESIGNATOR :=
    (ABSOLUTE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING | ;; namestring (better be absolute or bust, directory assumed where applicable).
             ;; In output-translations, directory is assumed and **/*.*.* added if it's last.
             ;; On MCL, a MacOSX-style POSIX namestring (for MacOS9 style, use #p"...");
             ;; Note that none of the above applies to strings used in *central-registry*,
             ;; which doesn't use this DSL: they are processed as normal namestrings.
             ;; however, you can compute what you put in the *central-registry*
             ;; based on the results of say (asdf::resolve-location "/Users/fare/cl/cl-foo/")
    PATHNAME | ;; pathname (better be an absolute path, or bust)
               ;; In output-translations, unless followed by relative components,
               ;; it better have appropriate wildcards, as in **/*.*.*
    :HOME | ;; designates the user-homedir-pathname ~/
    :USER-CACHE | ;; designates the default location for the user cache
    :HERE | ;; designates the location of the configuration file
            ;; (or *default-pathname-defaults*, if invoked interactively)
    :ROOT ;; magic, for output-translations source only: paths that are relative
          ;; to the root of the source host and device
    ;; Not valid anymore: :SYSTEM-CACHE (was a security hazard)

RELATIVE-COMPONENT-DESIGNATOR :=
    (RELATIVE-COMPONENT-DESIGNATOR RELATIVE-COMPONENT-DESIGNATOR ...) |
    STRING | ;; relative directory pathname as interpreted by coerce-pathname.
             ;; In output translations, if last component, **/*.*.* is added
    PATHNAME | ;; pathname; unless last component, directory is assumed.
    :IMPLEMENTATION | ;; directory based on implementation, e.g. sbcl-1.0.45-linux-x64
    :IMPLEMENTATION-TYPE | ;; a directory based on lisp-implementation-type only, e.g. sbcl
    :DEFAULT-DIRECTORY | ;; a relativized version of the default directory
    :*/ | ;; any direct subdirectory (since ASDF 2.011.4)
    :**/ | ;; any recursively inferior subdirectory (since ASDF 2.011.4)
    :*.*.* | ;; any file (since ASDF 2.011.4)
    ;; Not supported (anymore): :UID and :USERNAME
@end example

For instance, as a simple case, my @file{~/.config/common-lisp/source-registry.conf},
which is the default place ASDF looks for this configuration, once contained:
@example
(:source-registry
  (:tree (:home "cl")) ;; will expand to e.g. "/home/joeluser/cl/"
  :inherit-configuration)
@end example

@section Configuration Directories

Configuration directories consist in files each containing
a list of directives without any enclosing @code{(:source-registry ...)} form.
The files will be sorted by namestring as if by @code{string<} and
the lists of directives of these files with be concatenated in order.
An implicit @code{:inherit-configuration} will be included
at the @emph{end} of the list.

This allows for packaging software that has file granularity
(e.g. Debian's @code{dpkg} or some future version of @code{clbuild})
to easily include configuration information about distributed software.

The convention is that, for sorting purposes,
the names of files in such a directory begin with two digits
that determine the order in which these entries will be read.
Also, the type of these files is conventionally @code{"conf"}
and as a limitation to some implementations (e.g. GNU clisp),
the type cannot be @code{NIL}.

Directories may be included by specifying a directory pathname
or namestring in an @code{:include} directive, e.g.:

@example
        (:include "/foo/bar/")
@end example

Hence, to achieve the same effect as
my example @file{~/.config/common-lisp/source-registry.conf} above,
I could simply create a file
@file{~/.config/common-lisp/source-registry.conf.d/33-home-fare-cl.conf}
alone in its directory with the following contents:
@example
(:tree "/home/fare/cl/")
@end example

@subsection The :here directive

The @code{:here} directive is an absolute pathname designator that
refers to the directory containing the configuration file currently
being processed.

The @code{:here} directive is intended to simplify the delivery of
complex CL systems, and for easy configuration of projects shared through
revision control systems, in accordance with our design principle that
each participant should be able to provide all and only the information
available to him or her.

Consider a person X who has set up the source code repository for a
complex project with a master directory @file{dir/}.  Ordinarily, one
might simply have the user add a directive that would look something
like this:
@example
   (:tree  "path/to/dir")
@end example
But what if X knows that there are very large subtrees
under dir that are filled with, e.g., Java source code, image files for
icons, etc.?  All of the asdf system definitions are contained in the
subdirectories @file{dir/src/lisp/} and @file{dir/extlib/lisp/}, and
these are the only directories that should be searched.

In this case, X can put into @file{dir/} a file @file{asdf.conf} that
contains the following:
@example
(:source-registry
   (:tree (:here "src/lisp/"))
   (:tree (:here "extlib/lisp"))
   (:directory (:here "outlier/")))
@end example

Then when someone else (call her Y) checks out a copy of this
repository, she need only add
@example
(:include "/path/to/my/checkout/directory/asdf.conf")
@end example
to one of her previously-existing asdf source location configuration
files, or invoke @code{initialize-source-registry} with a configuration
form containing that s-expression.  ASDF will find the .conf file that X
has provided, and then set up source locations within the working
directory according to X's (relative) instructions.

@section Shell-friendly syntax for configuration

When considering environment variable @code{CL_SOURCE_REGISTRY}
ASDF will skip to next configuration if it's an empty string.
It will @code{READ} the string as a SEXP in the DSL
if it begins with a paren @code{(}
and it will be interpreted much like @code{TEXINPUTS}
list of paths, where

  * paths are separated
   by a @code{:} (colon) on Unix platforms (including cygwin),
   by a @code{;} (semicolon) on other platforms (mainly, Windows).

  * each entry is a directory to add to the search path.

  * if the entry ends with a double slash @code{//}
    then it instead indicates a tree in the subdirectories
    of which to recurse.

  * if the entry is the empty string (which may only appear once),
    then it indicates that the inherited configuration should be
    spliced there.


@section Search Algorithm
@vindex *default-source-registry-exclusions*

In case that isn't clear, the semantics of the configuration is that
when searching for a system of a given name,
directives are processed in order.

When looking in a directory, if the system is found, the search succeeds,
otherwise it continues.

When looking in a tree, if one system is found, the search succeeds.
If multiple systems are found, the consequences are unspecified:
the search may succeed with any of the found systems,
or an error may be raised.
ASDF currently returns the first system found,
XCVB currently raised an error.
If none is found, the search continues.

Exclude statements specify patterns of subdirectories
the systems from which to ignore.
Typically you don't want to use copies of files kept by such
version control systems as Darcs.
Exclude statements are not propagated to further included or inherited
configuration files or expressions;
instead the defaults are reset around every configuration statement
to the default defaults from @code{asdf::*default-source-registry-exclusions*}.

Include statements cause the search to recurse with the path specifications
from the file specified.

An inherit-configuration statement cause the search to recurse with the path
specifications from the next configuration
(@pxref{Controlling where ASDF searches for systems,,Configurations} above).


@section Caching Results

The implementation is allowed to either eagerly compute the information
from the configurations and file system, or to lazily re-compute it
every time, or to cache any part of it as it goes.
To explicitly flush any information cached by the system, use the API below.


@section Configuration API

The specified functions are exported from your build system's package.
Thus for ASDF the corresponding functions are in package ASDF,
and for XCVB the corresponding functions are in package XCVB.

@defun initialize-source-registry @Aoptional PARAMETER
   will read the configuration and initialize all internal variables.
   You may extend or override configuration
   from the environment and configuration files
   with the given @var{PARAMETER}, which can be
   @code{NIL} (no configuration override),
   or a SEXP (in the SEXP DSL),
   a string (as in the string DSL),
   a pathname (of a file or directory with configuration),
   or a symbol (fbound to function that when called returns one of the above).
@end defun

@defun clear-source-registry
   undoes any source registry configuration
   and clears any cache for the search algorithm.
   You might want to call this function
   (or better, @code{clear-configuration})
   before you dump an image that would be resumed
   with a different configuration,
   and return an empty configuration.
   Note that this does not include clearing information about
   systems defined in the current image, only about
   where to look for systems not yet defined.
@end defun

@defun ensure-source-registry @Aoptional PARAMETER
   checks whether a source registry has been initialized.
   If not, initialize it with the given @var{PARAMETER}.
@end defun

Every time you use ASDF's @code{find-system}, or
anything that uses it (such as @code{operate}, @code{load-system}, etc.),
@code{ensure-source-registry} is called with parameter NIL,
which the first time around causes your configuration to be read.
If you change a configuration file,
you need to explicitly @code{initialize-source-registry} again,
or maybe simply to @code{clear-source-registry} (or @code{clear-configuration})
which will cause the initialization to happen next time around.


@section Status

This mechanism is vastly successful, and we have declared
that @code{asdf:*central-registry*} is not recommended anymore,
though we will continue to support it.
All hooks into implementation-specific search mechanisms
have been integrated in the @code{wrapping-source-registry}
that everyone uses implicitly.


@section Rejected ideas

Alternatives I considered and rejected included:

@enumerate
@item Keep @code{asdf:*central-registry*} as the master with its current semantics,
   and somehow the configuration parser expands the new configuration
   language into a expanded series of directories of subdirectories to
   lookup, pre-recursing through specified hierarchies. This is kludgy,
   and leaves little space of future cleanups and extensions.

@item Keep @code{asdf:*central-registry*} remains the master but extend its semantics
   in completely new ways, so that new kinds of entries may be implemented
   as a recursive search, etc. This seems somewhat backwards.

@item Completely remove @code{asdf:*central-registry*}
   and break backwards compatibility.
   Hopefully this will happen in a few years after everyone migrate to
   a better ASDF and/or to XCVB, but it would be very bad to do it now.

@item Replace @code{asdf:*central-registry*} by a symbol-macro with appropriate magic
   when you dereference it or setf it. Only the new variable with new
   semantics is handled by the new search procedure.
   Complex and still introduces subtle semantic issues.
@end enumerate


I've been suggested the below features, but have rejected them,
for the sake of keeping ASDF no more complex than strictly necessary.

@itemize
@item
  More syntactic sugar: synonyms for the configuration directives, such as
  @code{(:add-directory X)} for @code{(:directory X)}, or @code{(:add-directory-hierarchy X)}
  or @code{(:add-directory X :recurse t)} for @code{(:tree X)}.

@item
   The possibility to register individual files instead of directories.

@item
  Integrate Xach Beane's tilde expander into the parser,
  or something similar that is shell-friendly or shell-compatible.
  I'd rather keep ASDF minimal. But maybe this precisely keeps it
  minimal by removing the need for evaluated entries that ASDF has?
  i.e. uses of @code{USER-HOMEDIR-PATHNAME} and @code{$SBCL_HOME}
  Hopefully, these are already superseded by the @code{:default-registry}

@item
  Using the shell-unfriendly syntax @code{/**} instead of @code{//} to specify recursion
  down a filesystem tree in the environment variable.
  It isn't that Lisp friendly either.
@end itemize

@section TODO

@itemize
@item Add examples
@end itemize


@section Credits for the source-registry

Thanks a lot to Stelian Ionescu for the initial idea.

Thanks to Rommel Martinez for the initial implementation attempt.

All bad design ideas and implementation bugs are to mine, not theirs.
But so are good design ideas and elegant implementation tricks.

 --- Francois-Rene Rideau @email{fare@@tunes.org}, Mon, 22 Feb 2010 00:07:33 -0500



@node Controlling where ASDF saves compiled files, Error handling, Controlling where ASDF searches for systems, Top
@comment  node-name,  next,  previous,  up
@chapter Controlling where ASDF saves compiled files
@cindex asdf-output-translations
@vindex ASDF_OUTPUT_TRANSLATIONS

Each Common Lisp implementation has its own format
for compiled files (fasls for short, short for ``fast loading'').
If you use multiple implementations
(or multiple versions of the same implementation),
you'll soon find your source directories
littered with various @file{fasl}s, @file{dfsl}s, @file{cfsl}s and so on.
Worse yet, some implementations use the same file extension
while changing formats from version to version (or platform to platform)
which means that you'll have to recompile binaries
as you switch from one implementation to the next.

Since ASDF 2, ASDF includes the @code{asdf-output-translations} facility
to mitigate the problem.

@section Configurations

Configurations specify mappings from input locations to output locations.
Once again we rely on the XDG base directory specification for configuration.
@xref{Controlling where ASDF searches for systems,,XDG base directory}.

@enumerate

@item
Some hardcoded wrapping output translations configuration may be used.
This allows special output translations (or usually, invariant directories)
to be specified corresponding to the similar special entries in the source registry.

@item
An application may explicitly initialize the output-translations
configuration using the Configuration API
in which case this takes precedence.
(@pxref{Controlling where ASDF saves compiled files,,Configuration API}.)
It may itself compute this configuration from the command-line,
from a script, from its own configuration file, etc.

@item
The source registry will be configured from
the environment variable @code{ASDF_OUTPUT_TRANSLATIONS} if it exists.

@item
The source registry will be configured from
user configuration file
@file{$XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf}
(which defaults to
@file{~/.config/common-lisp/asdf-output-translations.conf})
if it exists.

@item
The source registry will be configured from
user configuration directory
@file{$XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf.d/}
(which defaults to
@file{~/.config/common-lisp/asdf-output-translations.conf.d/})
if it exists.

@item
The source registry will be configured from
system configuration file
@file{/etc/common-lisp/asdf-output-translations.conf}
if it exists.

@item
The source registry will be configured from
system configuration directory
@file{/etc/common-lisp/asdf-output-translations.conf.d/}
if it exists.

@end enumerate

Each of these configurations is specified as a SEXP
in a trival domain-specific language (defined below).
Additionally, a more shell-friendly syntax is available
for the environment variable (defined yet below).

Each of these configurations is only used if the previous
configuration explicitly or implicitly specifies that it
includes its inherited configuration.

Note that by default, a per-user cache is used for output files.
This allows the seamless use of shared installations of software
between several users, and takes files out of the way of the developers
when they browse source code,
at the expense of taking a small toll when developers have to clean up
output files and find they need to get familiar with output-translations first.


@section Backward Compatibility
@cindex ASDF-BINARY-LOCATIONS compatibility


We purposefully do NOT provide backward compatibility with earlier versions of
@code{ASDF-Binary-Locations} (8 Sept 2009),
@code{common-lisp-controller} (7.0) or
@code{cl-launch} (2.35),
each of which had similar general capabilities.
The previous APIs of these programs were not designed
for configuration by the end-user
in an easy way with configuration files.
Recent versions of same packages use
the new @code{asdf-output-translations} API as defined below:
@code{common-lisp-controller} (7.2) and @code{cl-launch} (3.000).
@code{ASDF-Binary-Locations} is fully superseded and not to be used anymore.

This incompatibility shouldn't inconvenience many people.
Indeed, few people use and customize these packages;
these few people are experts who can trivially adapt to the new configuration.
Most people are not experts, could not properly configure these features
(except inasmuch as the default configuration of
@code{common-lisp-controller} and/or @code{cl-launch}
might have been doing the right thing for some users),
and yet will experience software that ``just works'',
as configured by the system distributor, or by default.

Nevertheless, if you are a fan of @code{ASDF-Binary-Locations},
we provide a limited emulation mode:

@defun enable-asdf-binary-locations-compatibility @Akey centralize-lisp-binaries default-toplevel-directory include-per-user-information map-all-source-files source-to-target-mappings
This function will initialize the new @code{asdf-output-translations} facility in a way
that emulates the behavior of the old @code{ASDF-Binary-Locations} facility.
Where you would previously set global variables
@var{*centralize-lisp-binaries*},
@var{*default-toplevel-directory*},
@var{*include-per-user-information*},
@var{*map-all-source-files*} or @var{*source-to-target-mappings*}
you will now have to pass the same values as keyword arguments to this function.
Note however that as an extension the @code{:source-to-target-mappings} keyword argument
will accept any valid pathname designator for @code{asdf-output-translations}
instead of just strings and pathnames.
@end defun

If you insist, you can also keep using the old @code{ASDF-Binary-Locations}
(the one available as an extension to load of top of ASDF,
not the one built into a few old versions of ASDF),
but first you must disable @code{asdf-output-translations}
with @code{(asdf:disable-output-translations)},
or you might experience ``interesting'' issues.

Also, note that output translation is enabled by default.
To disable it, use @code{(asdf:disable-output-translations)}.


@section Configuration DSL

Here is the grammar of the SEXP DSL
for @code{asdf-output-translations} configuration:

@verbatim
;; A configuration is single SEXP starting with keyword :source-registry
;; followed by a list of directives.
CONFIGURATION := (:output-translations DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of either of these:
    :inherit-configuration | ; splices inherited configuration (often specified last)
    :ignore-inherited-configuration | ; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap a
    ;; the newer required ASDF from an older release that doesn't sport said features:
    :ignore-invalid-entries | ; drops subsequent invalid entries instead of erroring out

    ;; include a configuration file or directory
    (:include PATHNAME-DESIGNATOR) |

    ;; enable global cache in ~/.common-lisp/cache/sbcl-1.0.45-linux-amd64/ or something.
    :enable-user-cache |
    ;; Disable global cache. Map / to /
    :disable-cache |

    ;; add a single directory to be scanned (no recursion)
    (DIRECTORY-DESIGNATOR DIRECTORY-DESIGNATOR)

    ;; use a function to return the translation of a directory designator
    (DIRECTORY-DESIGNATOR (:function TRANSLATION-FUNCTION))

DIRECTORY-DESIGNATOR :=
    NIL | ;; As source: skip this entry. As destination: same as source
    T | ;; as source matches anything, as destination leaves pathname unmapped.
    ABSOLUTE-COMPONENT-DESIGNATOR ;; same as in the source-registry language

TRANSLATION-FUNCTION :=
    SYMBOL | ;; symbol of a function that takes two arguments,
             ;; the pathname to be translated and the matching DIRECTORY-DESIGNATOR
    LAMBDA   ;; A form which evalutates to a function taking two arguments consisting of
             ;; the pathname to be translated and the matching DIRECTORY-DESIGNATOR

@end verbatim

Relative components better be either relative
or subdirectories of the path before them, or bust.

The last component, if not a pathname, is notionally completed by @file{/**/*.*}.
You can specify more fine-grained patterns
by using a pathname object as the last component
e.g. @file{#p"some/path/**/foo*/bar-*.fasl"}

You may use @code{#+features} to customize the configuration file.

The second designator of a mapping may be @code{NIL}, indicating that files are not mapped
to anything but themselves (same as if the second designator was the same as the first).

When the first designator is @code{t},
the mapping always matches.
When the first designator starts with @code{:root},
the mapping matches any host and device.
In either of these cases, if the second designator
isn't @code{t} and doesn't start with @code{:root},
then strings indicating the host and pathname are somehow copied
in the beginning of the directory component of the source pathname
before it is translated.

When the second designator is @code{t}, the mapping is the identity.
When the second designator starts with @code{:root},
the mapping preserves the host and device of the original pathname.
Notably, this allows you to map files
to a subdirectory of the whichever directory the file is in.
Though the syntax is not quite as easy to use as we'd like,
you can have an (source destination) mapping entry such as follows
in your configuration file,
or you may use @code{enable-asdf-binary-locations-compatibility}
with @code{:centralize-lisp-binaries nil}
which will do the same thing internally for you:
@verbatim
  #.(let ((wild-subdir (make-pathname :directory '(:relative :wild-inferiors)))
          (wild-file (make-pathname :name :wild :version :wild :type :wild)))
     `((:root ,wild-subdir ,wild-file) ;; Or using the implicit wildcard, just :root
       (:root ,wild-subdir :implementation ,wild-file)))
@end verbatim
Starting with ASDF 2.011.4, you can use the simpler:
        @code{`(:root (:root :**/ :implementation :*.*.*))}



@code{:include} statements cause the search to recurse with the path specifications
from the file specified.

If the @code{translate-pathname} mechanism cannot achieve a desired
translation, the user may provide a function which provides the
required algorithim.  Such a translation function is specified by
supplying a list as the second @code{directory-designator}
the first element of which is the keyword @code{:function},
and the second element of which is
either a symbol which designates a function or a lambda expression.
The function designated by the second argument must take two arguments,
the first being the pathname of the source file,
the second being the wildcard that was matched.
The result of the function invocation should be the translated pathname.

An @code{:inherit-configuration} statement cause the search to recurse with the path
specifications from the next configuration.
@xref{Controlling where ASDF saves compiled files,,Configurations}, above.

@itemize
@item
@code{:enable-user-cache} is the same as @code{(t :user-cache)}.
@item
@code{:disable-cache} is the same as @code{(t t)}.
@item
@code{:user-cache} uses the contents of variable @code{asdf::*user-cache*}
which by default is the same as using
@code{(:home ".cache" "common-lisp" :implementation)}.
@item
@code{:system-cache} uses the contents of variable @code{asdf::*system-cache*}
which by default is the same as using
@code{("/var/cache/common-lisp" :uid :implementation-type)}
(on Unix and cygwin), or something semi-sensible on Windows.
@end itemize


@section Configuration Directories

Configuration directories consist in files each contains
a list of directives without any enclosing
@code{(:output-translations ...)} form.
The files will be sorted by namestring as if by @code{string<} and
the lists of directives of these files with be concatenated in order.
An implicit @code{:inherit-configuration} will be included
at the @emph{end} of the list.

This allows for packaging software that has file granularity
(e.g. Debian's @command{dpkg} or some future version of @command{clbuild})
to easily include configuration information about software being distributed.

The convention is that, for sorting purposes,
the names of files in such a directory begin with two digits
that determine the order in which these entries will be read.
Also, the type of these files is conventionally @code{"conf"}
and as a limitation of some implementations, the type cannot be @code{NIL}.

Directories may be included by specifying a directory pathname
or namestring in an @code{:include} directive, e.g.:
@verbatim
        (:include "/foo/bar/")
@end verbatim

@section Shell-friendly syntax for configuration

When considering environment variable @code{ASDF_OUTPUT_TRANSLATIONS}
ASDF will skip to next configuration if it's an empty string.
It will @code{READ} the string as an SEXP in the DSL
if it begins with a paren @code{(}
and it will be interpreted as a list of directories.
Directories should come by pairs, indicating a mapping directive.
Entries are separated
by a @code{:} (colon) on Unix platforms (including cygwin),
by a @code{;} (semicolon) on other platforms (mainly, Windows).

The magic empty entry,
if it comes in what would otherwise be the first entry in a pair,
indicates the splicing of inherited configuration.
If it comes as the second entry in a pair,
it indicates that the directory specified first is to be left untranslated
(which has the same effect as if the directory had been repeated).


@section Semantics of Output Translations

From the specified configuration,
a list of mappings is extracted in a straightforward way:
mappings are collected in order, recursing through
included or inherited configuration as specified.
To this list is prepended some implementation-specific mappings,
and is appended a global default.

The list is then compiled to a mapping table as follows:
for each entry, in order, resolve the first designated directory
into an actual directory pathname for source locations.
If no mapping was specified yet for that location,
resolve the second designated directory to an output location directory
add a mapping to the table mapping the source location to the output location,
and add another mapping from the output location to itself
(unless a mapping already exists for the output location).

Based on the table, a mapping function is defined,
mapping source pathnames to output pathnames:
given a source pathname, locate the longest matching prefix
in the source column of the mapping table.
Replace that prefix by the corresponding output column
in the same row of the table, and return the result.
If no match is found, return the source pathname.
(A global default mapping the filesystem root to itself
may ensure that there will always be a match,
with same fall-through semantics).

@section Caching Results

The implementation is allowed to either eagerly compute the information
from the configurations and file system, or to lazily re-compute it
every time, or to cache any part of it as it goes.
To explicitly flush any information cached by the system, use the API below.


@section Output location API

The specified functions are exported from package ASDF.

@defun initialize-output-translations @Aoptional PARAMETER
   will read the configuration and initialize all internal variables.
   You may extend or override configuration
   from the environment and configuration files
   with the given @var{PARAMETER}, which can be
   @code{NIL} (no configuration override),
   or a SEXP (in the SEXP DSL),
   a string (as in the string DSL),
   a pathname (of a file or directory with configuration),
   or a symbol (fbound to function that when called returns one of the above).
@end defun

@defun disable-output-translations
   will initialize output translations in a way
   that maps every pathname to itself,
   effectively disabling the output translation facility.
@end defun

@defun clear-output-translations
   undoes any output translation configuration
   and clears any cache for the mapping algorithm.
   You might want to call this function
   (or better, @code{clear-configuration})
   before you dump an image that would be resumed
   with a different configuration,
   and return an empty configuration.
   Note that this does not include clearing information about
   systems defined in the current image, only about
   where to look for systems not yet defined.
@end defun

@defun ensure-output-translations @Aoptional PARAMETER
   checks whether output translations have been initialized.
   If not, initialize them with the given @var{PARAMETER}.
   This function will be called before any attempt to operate on a system.
@end defun

@defun apply-output-translations PATHNAME
   Applies the configured output location translations to @var{PATHNAME}
   (calls @code{ensure-output-translations} for the translations).
@end defun

Every time you use ASDF's @code{output-files}, or
anything that uses it (that may compile, such as @code{operate}, @code{perform}, etc.),
@code{ensure-output-translations} is called with parameter NIL,
which the first time around causes your configuration to be read.
If you change a configuration file,
you need to explicitly @code{initialize-output-translations} again,
or maybe @code{clear-output-translations} (or @code{clear-configuration}),
which will cause the initialization to happen next time around.


@section Credits for output translations

Thanks a lot to Bjorn Lindberg and Gary King for @code{ASDF-Binary-Locations},
and to Peter van Eynde for @code{Common Lisp Controller}.

All bad design ideas and implementation bugs are to mine, not theirs.
But so are good design ideas and elegant implementation tricks.

 --- Francois-Rene Rideau @email{fare@@tunes.org}

@c @section Default locations
@c @findex output-files-for-system-and-operation

@c The default binary location for each Lisp implementation
@c is a subdirectory of each source directory.
@c To account for different Lisps, Operating Systems, Implementation versions,
@c and so on, ASDF borrows code from SLIME
@c to create reasonable custom directory names.
@c Here are some examples:

@c @itemize
@c @item
@c SBCL, version 1.0.45 on Mac OS X for Intel: @code{sbcl-1.0.45-darwin-x86}

@c @item
@c Franz Allegro, version 8.0, ANSI Common Lisp: @code{allegro-8.0a-macosx-x86}

@c @item
@c Franz Allegro, version 8.1, Modern (case sensitive) Common Lisp: @code{allegro-8.1m-macosx-x86}
@c @end itemize

@c By default, all output file pathnames will be relocated
@c to some thus-named subdirectory of @file{~/.cache/common-lisp/}.

@c See the document @file{README.asdf-output-translations}
@c for a full specification on how to configure @code{asdf-output-translations}.

@node  Error handling, Miscellaneous additional functionality, Controlling where ASDF saves compiled files, Top
@comment  node-name,  next,  previous,  up
@chapter Error handling
@findex SYSTEM-DEFINITION-ERROR
@findex OPERATION-ERROR

@section ASDF errors

If ASDF detects an incorrect system definition, it will signal a generalised instance of
@code{SYSTEM-DEFINITION-ERROR}.

Operations may go wrong (for example when source files contain errors).
These are signalled using generalised instances of
@code{OPERATION-ERROR}.

@section Compilation error and warning handling
@vindex *compile-file-warnings-behaviour*
@vindex *compile-file-errors-behavior*

ASDF checks for warnings and errors when a file is compiled.
The variables @var{*compile-file-warnings-behaviour*} and
@var{*compile-file-errors-behavior*}
control the handling of any such events.
The valid values for these variables are
@code{:error}, @code{:warn}, and @code{:ignore}.

@node  Miscellaneous additional functionality, Getting the latest version, Error handling, Top
@comment  node-name,  next,  previous,  up
@chapter Miscellaneous additional functionality

ASDF includes several additional features that are generally
useful for system definition and development.

@section Controlling file compilation

When declaring a component (system, module, file),
you can specify a keyword argument @code{:around-compile function}.
If left unspecified (and therefore unbound),
the value will be inherited from the parent component if any,
or with a default of @code{nil}
if no value is specified in any transitive parent.

The argument must be a either @code{nil}, a fbound symbol,
a lambda-expression (e.g. @code{(lambda (thunk) ...(funcall thunk ...) ...)})
a function object (e.g. using @code{#.#'} but that's discouraged
because it prevents the introspection done by e.g. asdf-dependency-grovel),
or a string that when @code{read} yields a symbol or a lambda-expression.
@code{nil} means the normal compile-file function will be called.
A non-nil value designates a function of one argument
that will be called with a function that will
invoke @code{compile-file*} with various arguments;
the around-compile hook may supply additional keyword arguments
to pass to that call to @code{compile-file*}.

One notable argument that is heeded by @code{compile-file*} is
@code{:compile-check},
a function called when the compilation was otherwise a success,
with the same arguments as @code{compile-file};
the function shall return true if the compilation
and its resulting compiled file respected all system-specific invariants,
and false (@code{nil}) if it broke any of those invariants;
it may issue warnings or errors before it returns @code{nil}.
(NB: The ability to pass such extra flags
is only available starting with ASDF 2.22.3.)
This feature is notably exercised by asdf-finalizers.

By using a string, you may reference
a function, symbol and/or package
that will only be created later during the build, but
isn't yet present at the time the defsystem form is evaluated.
However, if your entire system is using such a hook, you may have to
explicitly override the hook with @code{nil} for all the modules and files
that are compiled before the hook is defined.

Using this hook, you may achieve such effects as:
locally renaming packages,
binding @var{*readtables*} and other syntax-controlling variables,
handling warnings and other conditions,
proclaiming consistent optimization settings,
saving code coverage information,
maintaining meta-data about compilation timings,
setting gensym counters and PRNG seeds and other sources of non-determinism,
overriding the source-location and/or timestamping systems,
checking that some compile-time side-effects were properly balanced,
etc.

Note that there is no around-load hook. This is on purpose.
Some implementations such as ECL, GCL or MKCL link object files,
which allows for no such hook.
Other implementations allow for concatenating FASL files,
which doesn't allow for such a hook either.
We aim to discourage something that's not portable,
and has some dubious impact on performance and semantics
even when it is possible.
Things you might want to do with an around-load hook
are better done around-compile,
though it may at times require some creativity
(see e.g. the @code{package-renaming} system).


@section Controlling source file character encoding

Starting with ASDF 2.21, components accept a @code{:encoding} option
so authors may specify which character encoding should be used
to read and evaluate their source code.
When left unspecified, the encoding is inherited
from the parent module or system;
if no encoding is specified at any point,
the default @code{:autodetect} is assumed.
By default, only @code{:default}, @code{:utf-8}
and @code{:autodetect} are accepted.
@code{:autodetect}, the default, calls
@code{*encoding-detection-hook*} which by default always returns
@code{*default-encoding*} which itself defaults to @code{:default}.

In other words, there now are plenty of extension hooks, but
by default ASDF follows the backwards compatible behavior
of using whichever @code{:default} encoding your implementation uses,
which itself may or may not vary based on environment variables
and other locale settings.
In practice this means that only source code that only uses ASCII
is guaranteed to be read the same on all implementations
independently from any user setting.

Additionally, for backward-compatibility with older versions of ASDF
and/or with implementations that do not support unicode and its many encodings,
you may want to use
the reader conditionals @code{#+asdf-unicode #+asdf-unicode}
to protect any @code{:encoding @emph{encoding}} statement
as @code{:asdf-unicode} will be present in @code{*features*}
only if you're using a recent ASDF
on an implementation that supports unicode.
We recommend that you avoid using unprotected @code{:encoding} specifications
until after ASDF 2.21 or later becomes widespread, hopefully by the end of 2012.

While it offers plenty of hooks for extension,
and one such extension is being developed (see below),
ASDF itself only recognizes one encoding beside @code{:default},
and that is @code{:utf-8}, which is the @emph{de facto} standard,
already used by the vast majority of libraries that use more than ASCII.
On implementations that do not support unicode,
the feature @code{:asdf-unicode} is absent, and
the @code{:default} external-format is used
to read even source files declared as @code{:utf-8}.
On these implementations, non-ASCII characters
intended to be read as one CL character
may thus end up being read as multiple CL characters.
In most cases, this shouldn't affect the software's semantics:
comments will be skipped just the same, strings with be read and printed
with slightly different lengths, symbol names will be accordingly longer,
but none of it should matter.
But a few systems that actually depend on unicode characters
may fail to work properly, or may work in a subtly different way.
See for instance @code{lambda-reader}.

We invite you to embrace UTF-8
as the encoding for non-ASCII characters starting today,
even without any explicit specification in your @code{.asd} files.
Indeed, on some implementations and configurations,
UTF-8 is already the @code{:default},
and loading your code may cause errors if it is encoded in anything but UTF-8.
Therefore, even with the legacy behavior,
non-UTF-8 is guaranteed to break for some users,
whereas UTF-8 is pretty much guaranteed not to break anywhere
(provided you do @emph{not} use a BOM),
although it might be read incorrectly on some implementations.
In the future, we intend to make @code{:utf-8}
the default value of @code{*default-encoding*},
to be enforced everywhere, so at least the code is guaranteed
to be read correctly everywhere it can be.

If you need non-standard character encodings for your source code,
use the extension system @code{asdf-encodings}, by specifying
@code{:defsystem-depends-on (:asdf-encodings)} in your @code{defsystem}.
This extension system will register support for more encodings using the
@code{*encoding-external-format-hook*} facility,
so you can explicitly specify @code{:encoding :latin1}
in your @code{.asd} file.
Using the @code{*encoding-detection-hook*} it will also
eventually implement some autodetection of a file's encoding
from an emacs-style @code{-*- mode: lisp ; coding: latin1 -*-} declaration,
or otherwise based on an analysis of octet patterns in the file.
At this point, asdf-encoding only supports the encodings
that are supported as part of your implementation.
Since the list varies depending on implementations,
we once again recommend you use @code{:utf-8} everywhere,
which is the most portable (next is @code{:latin1}).

If you're not using a version of Quicklisp that has it,
you may get the source for @code{asdf-encodings} using git:
@kbd{git clone git://common-lisp.net/projects/asdf/asdf-encodings.git}
or
@kbd{git clone ssh://common-lisp.net/project/asdf/git/asdf-encodings.git}.
You can also browse the repository on
@url{http://common-lisp.net/gitweb?p=projects/asdf/asdf-encodings.git}.

In the future, we intend to change the default @code{*default-encoding*}
to @code{:utf-8}, which is already the de facto standard
for most libraries that use non-ASCII characters:
utf-8 works everywhere and was backhandedly enforced by
a lot of people using SBCL and utf-8 and sending reports to authors
so they make their packages compatible.
A survey showed only about a handful few libraries
are incompatible with non-UTF-8, and then, only in comments,
and we believe that authors will adopt UTF-8 when prompted.
See the April 2012 discussion on the asdf-devel mailing-list.
For backwards compatibility with users who insist on a non-UTF-8 encoding,
but cannot immediately transition to using @code{asdf-encodings}
(maybe because it isn't ready), it will still be possible to use
the @code{:encoding :default} option in your @code{defsystem} form
to restore the behavior of ASDF 2.20 and earlier.
This shouldn't be required in libraries,
because user pressure as mentioned above will already have pushed
library authors towards using UTF-8;
but authors of end-user programs might care.

When you use @code{asdf-encodings}, any further loaded @code{.asd} file
will use the autodetection algorithm to determine its encoding;
yet if you depend on this detection happening,
you may want to explicitly load @code{asdf-encodings} early in your build,
for by the time you can use @code{:defsystem-depends-on},
it is already too late to load it.
In practice, this means that the @code{*default-encoding*}
is usually used for @code{.asd} files.
Currently, this defaults to @code{:default} for backwards compatibility,
and that means that you shouldn't rely on non-ASCII characters in a .asd file.
Since component (path)names are the only real data in these files,
and non-ASCII characters are not very portable for file names,
this isn't too much of an issue.
We still encourage you to use either plain ASCII or UTF-8
in @code{.asd} files,
as we intend to make @code{:utf-8} the default encoding in the future.
This might matter, for instance, in meta-data about author's names.


@section Miscellaneous Functions

Most of these functions are not exported by ASDF anymore,
but only used for private purposes of ASDF.
Please use ASDF-UTILS for the same functions exported from a stable library.

@defun coerce-pathname name @Akey type defaults

This function (available starting with ASDF 2.012.11)
takes an argument, and portably interprets it as a pathname.
If the argument @var{name} is a pathname or @code{nil}, it is passed through;
if it's a symbol, it's interpreted as a string by downcasing it;
if it's a string, it is first separated using @code{/} into substrings;
the leading substrings denote subdirectories of a relative pathname.
If @var{type} is @code{:directory} or the string ends with @code{/},
the last substring is also a subdirectory;
if @var{type} is a string, it is used as the type of the pathname, and
the last substring is the name component of the pathname;
if @var{type} is @code{nil}, the last substring specifies both name and type components
of the pathname, with the last @code{.} separating them, or only the name component
if there's no last @code{.} or if there is only one dot and it's the first character.
The host, device and version components come from @var{defaults}, which defaults to
@var{*default-pathname-defaults*}; but that shouldn't matter if you use @code{merge-pathnames*}.

@end defun

@defun merge-pathnames* @Akey specified defaults

This function is a replacement for @code{merge-pathnames} that uses the host and device
from the @var{defaults} rather than the @var{specified} pathname when the latter
is a relative pathname. This allows ASDF and its users to create and use relative pathnames
without having to know beforehand what are the host and device
of the absolute pathnames they are relative to.

@end defun

@defun system-relative-pathname system name @Akey type

It's often handy to locate a file relative to some system.
The @code{system-relative-pathname} function meets this need.

It takes two mandatory arguments @var{system} and @var{name}
and a keyword argument @var{type}:
@var{system} is name of a system, whereas @var{name} and optionally @var{type}
specify a relative pathname, interpreted like a component pathname specifier
by @code{coerce-pathname}. @xref{The defsystem grammar,,Pathname specifiers}.

It returns a pathname built from the location of the system's
source directory and the relative pathname. For example:

@lisp
> (asdf:system-relative-pathname 'cl-ppcre "regex.data")
#P"/repository/other/cl-ppcre/regex.data"
@end lisp

@end defun

@defun system-source-directory system-designator

ASDF does not provide a turnkey solution for locating
data (or other miscellaneous) files
that are distributed together with the source code of a system.
Programmers can use @code{system-source-directory} to find such files.
Returns a pathname object.
The @var{system-designator} may be a string, symbol, or ASDF system object.
@end defun

@defun clear-system system-designator

It is sometimes useful to force recompilation of a previously loaded system.
In these cases, it may be useful to @code{(asdf:clear-system :foo)}
to remove the system from the table of currently loaded systems;
the next time the system @code{foo} or one that depends on it is re-loaded,
@code{foo} will then be loaded again.
Alternatively, you could touch @code{foo.asd} or
remove the corresponding fasls from the output file cache.
(It was once conceived that one should provide
a list of systems the recompilation of which to force
as the @code{:force} keyword argument to @code{load-system};
but this has never worked, and though the feature was fixed in ASDF 2.000,
it remains @code{cerror}'ed out as nobody ever used it.)

Note that this does not and cannot by itself undo the previous loading
of the system. Common Lisp has no provision for such an operation,
and its reliance on irreversible side-effects to global datastructures
makes such a thing impossible in the general case.
If the software being re-loaded is not conceived with hot upgrade in mind,
this re-loading may cause many errors, warnings or subtle silent problems,
as packages, generic function signatures, structures, types, macros, constants, etc.
are being redefined incompatibly.
It is up to the user to make sure that reloading is possible and has the desired effect.
In some cases, extreme measures such as recursively deleting packages,
unregistering symbols, defining methods on @code{update-instance-for-redefined-class}
and much more are necessary for reloading to happen smoothly.
ASDF itself goes through notable pains to make such a hot upgrade possible
with respect to its own code, and what it does is ridiculously complex;
look at the beginning of @file{asdf.lisp} to see what it does.
@end defun

@defun run-program

run-program takes a @var{COMMAND} argument that is either
a list of a program path and its arguments,
or a string to be executed by a shell.
It spawns the command, waits for it to return,
verifies that it exited cleanly (unless told not too below),
and optionally captures and processes its output.
It accepts many keyword arguments to configure its behavior.

@code{output} is its most important argument;
it specifies how the output is captured and processed.
If it is @code{nil}, then the output is not captured.
If it is @code{:interactive}, then
input and output are inherited from the current process,
which the subprocess can control until it exits.
Otherwise, the output is captured and redirected to a stream,
and processed by @code{slurp-input-stream} with the object as first argument.
See below.

@code{element-type} and @code{external-format} are passed on
to your Lisp implementation, when available, for creation of the output stream.

@code{force-shell} forces evaluation of the command through a shell,
even if it was passed as a list rather than a string.

@code{ignore-error-status} causes @code{run-program}
to not raise an error if the spawned program exits in error.
Following POSIX convention, an error is anything but
a normal exit with status code zero.

run-program works on all platforms supported by ASDF, except Genera.

@end defun

@defun slurp-input-stream

It's a generic function of two arguments, a target object and an input stream,
and accepting keyword arguments.
Predefined methods based on the target object are as follow:

If the object is a function, the function is called with the stream as argument.

If the object is a cons, its first element is applied to its rest appended by
a list of the input stream.

If the object is an output stream, the contents of the input stream are copied to it.
If the linewise argument is provided, copying happens line by line,
and an optional prefix is printed before each line.
Otherwise, copying happen based on a buffer of size buffer-size,
using the element-type.

If the object is 'string or :string, the content is captured into a string
of the given element-type.

If the object is :lines, the content is captured as a list of strings,
one per line, without line ending. If the count argument is provided,
it is a maximum count of lines to be read.

If the object is :line, the content is capture as with :lines above,
and then its sub-object is extracted with the path argument,
which defaults to 0, extracting the first line.
A number will extract the corresponding line.
See the documentation for asdf-driver:sub-object.

If the object is :forms, the content is captured as a list of S-expressions,
as read by the Lisp reader.
If the count argument is provided,
it is a maximum count of lines to be read.
We recommend you control the syntax with such macro as
asdf-driver:with-safe-io-syntax.

If the object is :form, the content is capture as with :forms above,
and then its sub-object is extracted with the path argument,
which defaults to 0, extracting the first form.
A number will extract the corresponding form.
See the documentation for asdf-driver:sub-object.
We recommend you control the syntax with such macro as
asdf-driver:with-safe-io-syntax.

@end defun

@defun run-shell-command

This function is obsolete and present only for the sake of backwards-compatibility:
``If it's not backwards, it's not compatible''. We strongly discourage its use.
Its current behavior is only well-defined on Unix platforms
(which include MacOS X and cygwin). On Windows, anything goes.
The following documentation is only for the purpose of your migrating away from it
in a way that preserves semantics.

Instead we recommend the use @code{run-program} above
available as part of ASDF since ASDF 3.

@code{run-shell-command} takes as arguments a @code{format} control-string
and arguments to be passed to @code{format} after this control-string
to produce a string.
This string is a command that will be evaluated with a POSIX shell if possible;
yet, on Windows, some implementations will use CMD.EXE,
while others (like SBCL) will make an attempt at invoking a POSIX shell
(and fail if it is not present).
@end defun


@node Getting the latest version, FAQ, Miscellaneous additional functionality, Top
@comment  node-name,  next,  previous,  up
@chapter Getting the latest version

Decide which version you want.
The @code{master} branch is where development happens;
its @code{HEAD} is usually OK, including the latest fixes and portability tweaks,
but an occasional regression may happen despite our (limited) test suite.

The @code{release} branch is what cautious people should be using;
it has usually been tested more, and releases are cut at a point
where there isn't any known unresolved issue.

You may get the ASDF source repository using git:
@kbd{git clone git://common-lisp.net/projects/asdf/asdf.git}

You will find the above referenced tags in this repository.
You can also browse the repository on
@url{http://common-lisp.net/gitweb?p=projects/asdf/asdf.git}.

Discussion of ASDF development is conducted on the
mailing list
@kbd{asdf-devel@@common-lisp.net}.
@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel}


@node FAQ, TODO list, Getting the latest version, Top
@comment  node-name,  next,  previous,  up
@chapter FAQ

@section  ``Where do I report a bug?''

ASDF bugs are tracked on launchpad: @url{https://launchpad.net/asdf}.

If you're unsure about whether something is a bug, or for general discussion,
use the @url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel,asdf-devel mailing list}


@section ``What has changed between ASDF 1 and ASDF 2?''

@subsection What are ASDF 1 and ASDF 2?

On May 31st 2010, we have released ASDF 2.
ASDF 2 refers to release 2.000 and later.
(Releases between 1.656 and 1.728 were development releases for ASDF 2.)
ASDF 1 to any release earlier than 1.369 or so.
If your ASDF doesn't sport a version, it's an old ASDF 1.

ASDF 2 and its release candidates push
@code{:asdf2} onto @code{*features*} so that if you are writing
ASDF-dependent code you may check for this feature
to see if the new API is present.
@emph{All} versions of ASDF should have the @code{:asdf} feature.

Additionally, all versions of ASDF 2
define a function @code{(asdf:asdf-version)} you may use to query the version;
and the source code of recent versions of ASDF 2 features the version number
prominently on the second line of its source code.

If you are experiencing problems or limitations of any sort with ASDF 1,
we recommend that you should upgrade to ASDF 2,
or whatever is the latest release.


@subsection ASDF can portably name files in subdirectories

Common Lisp namestrings are not portable,
except maybe for logical pathnamestrings,
that themselves have various limitations and require a lot of setup
that is itself ultimately non-portable.

In ASDF 1, the only portable ways to refer to pathnames inside systems and components
were very awkward, using @code{#.(make-pathname ...)} and
@code{#.(merge-pathnames ...)}.
Even the above were themselves were inadequate in the general case
due to host and device issues, unless horribly complex patterns were used.
Plenty of simple cases that looked portable actually weren't,
leading to much confusion and greavance.

ASDF 2 implements its own portable syntax for strings as pathname specifiers.
Naming files within a system definition becomes easy and portable again.
@xref{Miscellaneous additional functionality,asdf:system-relative-pathname},
@code{merge-pathnames*},
@code{coerce-pathname}.

On the other hand, there are places where systems used to accept namestrings
where you must now use an explicit pathname object:
@code{(defsystem ... :pathname "LOGICAL-HOST:PATH;TO;SYSTEM;" ...)}
must now be written with the @code{#p} syntax:
@code{(defsystem ... :pathname #p"LOGICAL-HOST:PATH;TO;SYSTEM;" ...)}

@xref{The defsystem grammar,,Pathname specifiers}.


@subsection Output translations

A popular feature added to ASDF was output pathname translation:
@code{asdf-binary-locations}, @code{common-lisp-controller},
@code{cl-launch} and other hacks were all implementing it in ways
both mutually incompatible and difficult to configure.

Output pathname translation is essential to share
source directories of portable systems across multiple implementations
or variants thereof,
or source directories of shared installations of systems across multiple users,
or combinations of the above.

In ASDF 2, a standard mechanism is provided for that,
@code{asdf-output-translations},
with sensible defaults, adequate configuration languages,
a coherent set of configuration files and hooks,
and support for non-Unix platforms.

@xref{Controlling where ASDF saves compiled files}.

@subsection Source Registry Configuration

Configuring ASDF used to require special magic
to be applied just at the right moment,
between the moment ASDF is loaded and the moment it is used,
in a way that is specific to the user,
the implementation he is using and the application he is building.

This made for awkward configuration files and startup scripts
that could not be shared between users, managed by administrators
or packaged by distributions.

ASDF 2 provides a well-documented way to configure ASDF,
with sensible defaults, adequate configuration languages,
and a coherent set of configuration files and hooks.

We believe it's a vast improvement because it decouples
application distribution from library distribution.
The application writer can avoid thinking where the libraries are,
and the library distributor (dpkg, clbuild, advanced user, etc.)
can configure them once and for every application.
Yet settings can be easily overridden where needed,
so whoever needs control has exactly as much as required.

At the same time, ASDF 2 remains compatible
with the old magic you may have in your build scripts
(using @code{*central-registry*} and
@code{*system-definition-search-functions*})
to tailor the ASDF configuration to your build automation needs,
and also allows for new magic, simpler and more powerful magic.

@xref{Controlling where ASDF searches for systems}.


@subsection Usual operations are made easier to the user

In ASDF 1, you had to use the awkward syntax
@code{(asdf:oos 'asdf:load-op :foo)}
to load a system,
and similarly for @code{compile-op}, @code{test-op}.

In ASDF 2, you can use shortcuts for the usual operations:
@code{(asdf:load-system :foo)}, and
similarly for @code{compile-system}, @code{test-system}.


@subsection Many bugs have been fixed

The following issues and many others have been fixed:

@itemize
@item
The infamous TRAVERSE function has been revamped completely
between ASDF 1 and ASDF 2, with many bugs squashed.
In particular, dependencies were not correctly propagated
across modules but now are.
It has been completely rewritten many times over
between ASDF 2.000 and ASDF 3,
with fundamental issues in the original model being fixed.
Timestamps were not propagated at all, and now are.
The internal model of how actions depend on each other
is now both consistent and complete.
The :version and
the :force (system1 .. systemN) feature have been fixed.

@item
Performance has been notably improved for large systems
(say with thousands of components) by using
hash-tables instead of linear search,
and linear-time list accumulation
instead of quadratic-time recursive appends.

@item
Many features used to not be portable,
especially where pathnames were involved.
Windows support was notably quirky because of such non-portability.

@item
The internal test suite used to massively fail on many implementations.
While still incomplete, it now fully passes
on all implementations supported by the test suite,
except for GCL (due to GCL bugs).

@item
Support was lacking for some implementations.
ABCL and GCL were notably wholly broken.
ECL extensions were not integrated with ASDF release.

@item
The documentation was grossly out of date.

@end itemize


@subsection ASDF itself is versioned

Between new features, old bugs fixed, and new bugs introduced,
there were various releases of ASDF in the wild,
and no simple way to check which release had which feature set.
People using or writing systems had to either make worst-case assumptions
as to what features were available and worked,
or take great pains to have the correct version of ASDF installed.

With ASDF 2, we provide a new stable set of working features
that everyone can rely on from now on.
Use @code{#+asdf2} to detect presence of ASDF 2,
@code{(asdf:version-satisfies (asdf:asdf-version) "2.345.67")}
to check the availability of a version no earlier than required.


@subsection ASDF can be upgraded

When an old version of ASDF was loaded,
it was very hard to upgrade ASDF in your current image
without breaking everything.
Instead you had to exit the Lisp process and
somehow arrange to start a new one from a simpler image.
Something that can't be done from within Lisp,
making automation of it difficult,
which compounded with difficulty in configuration,
made the task quite hard.
Yet as we saw before, the task would have been required
to not have to live with the worst case or non-portable
subset of ASDF features.

With ASDF 2, it is easy to upgrade
from ASDF 2 to later versions from within Lisp,
and not too hard to upgrade from ASDF 1 to ASDF 2 from within Lisp.
We support hot upgrade of ASDF and any breakage is a bug
that we will do our best to fix.
There are still limitations on upgrade, though,
most notably the fact that after you upgrade ASDF,
you must also reload or upgrade all ASDF extensions.

@subsection Decoupled release cycle

When vendors were releasing their Lisp implementations with ASDF,
they had to basically never change version
because neither upgrade nor downgrade was possible
without breaking something for someone,
and no obvious upgrade path was visible and recommendable.

With ASDF 2, upgrade is possible, easy and can be recommended.
This means that vendors can safely ship a recent version of ASDF,
confident that if a user isn't fully satisfied,
he can easily upgrade ASDF and deal
with a supported recent version of it.
This means that release cycles will be causally decoupled,
the practical consequence of which will mean faster convergence
towards the latest version for everyone.


@subsection Pitfalls of the transition to ASDF 2

The main pitfalls in upgrading to ASDF 2 seem to be related
to the output translation mechanism.

@itemize

@item
Output translations is enabled by default. This may surprise some users,
most of them in pleasant way (we hope), a few of them in an unpleasant way.
It is trivial to disable output translations.
@xref{FAQ,,``How can I wholly disable the compiler output cache?''}.

@item
Some systems in the large have been known
not to play well with output translations.
They were relatively easy to fix.
Once again, it is also easy to disable output translations,
or to override its configuration.

@item
The new ASDF output translations are incompatible with ASDF-Binary-Locations.
They replace A-B-L, and there is compatibility mode to emulate
your previous A-B-L configuration.
See @code{enable-asdf-binary-locations-compatibility} in
@pxref{Controlling where ASDF saves compiled files,,Backward Compatibility}.
But thou shalt not load ABL on top of ASDF 2.

@end itemize

Other issues include the following:

@itemize

@item
ASDF pathname designators are now specified
in places where they were unspecified,
and a few small adjustments have to be made to some non-portable defsystems.
Notably, in the @code{:pathname} argument
to a @code{defsystem} and its components,
a logical pathname (or implementation-dependent hierarchical pathname)
must now be specified with @code{#p} syntax
where the namestring might have previously sufficed;
moreover when evaluation is desired @code{#.} must be used,
where it wasn't necessary in the toplevel @code{:pathname} argument
(but necessary in other @code{:pathname} arguments).

@item
There is a slight performance bug, notably on SBCL,
when initially searching for @file{asd} files,
the implicit @code{(directory "/configured/path/**/*.asd")}
for every configured path @code{(:tree "/configured/path/")}
in your @code{source-registry} configuration can cause a slight pause.
Try to @code{(time (asdf:initialize-source-registry))}
to see how bad it is or isn't on your system.
If you insist on not having this pause,
you can avoid the pause by overriding the default source-registry configuration
and not use any deep @code{:tree} entry but only @code{:directory} entries
or shallow @code{:tree} entries.
Or you can fix your implementation to not be quite that slow
when recursing through directories.
@emph{Update}: This performance bug fixed the hard way in 2.010.

@item
On Windows, only LispWorks supports proper default configuration pathnames
based on the Windows registry.
Other implementations make do with environment variables,
that you may have to define yourself
if you're using an older version of Windows.
Windows support is somewhat less tested than Unix support.
Please help report and fix bugs.
@emph{Update}: As of ASDF 2.21, all implementations
should now use the same proper default configuration pathnames
and they should actually work, though they haven't all been tested.

@item
The mechanism by which one customizes a system so that Lisp files
may use a different extension from the default @file{.lisp} has changed.
Previously, the pathname for a component
was lazily computed when operating on a system,
and you would
@code{(defmethod source-file-type ((component cl-source-file) (system (eql (find-system 'foo))))
  (declare (ignorable component system)) "lis")}.
Now, the pathname for a component is eagerly computed when defining the system,
and instead you will @code{(defclass cl-source-file.lis (cl-source-file) ((type :initform "lis")))}
and use @code{:default-component-class cl-source-file.lis}
as argument to @code{defsystem},
as detailed in a @pxref{FAQ,How do I create a system definition where all the source files have a .cl extension?} below.

@findex source-file-type


@end itemize


@section Issues with installing the proper version of ASDF

@subsection ``My Common Lisp implementation comes with an outdated version of ASDF. What to do?''

We recommend you upgrade ASDF.
@xref{Loading ASDF,,Upgrading ASDF}.

If this does not work, it is a bug, and you should report it.
@xref{FAQ, report-bugs, Where do I report a bug}.
In the meantime, you can load @file{asdf.lisp} directly.
@xref{Loading ASDF,Loading an otherwise installed ASDF}.


@subsection ``I'm a Common Lisp implementation vendor. When and how should I upgrade ASDF?''

Since ASDF 2,
it should always be a good time to upgrade to a recent version of ASDF.
You may consult with the maintainer for which specific version they recommend,
but the latest @code{release} should be correct.
We trust you to thoroughly test it with your implementation
before you release it.
If there are any issues with the current release,
it's a bug that you should report upstream and that we will fix ASAP.

As to how to include ASDF, we recommend the following:

@itemize
@item
If ASDF isn't loaded yet, then @code{(require "asdf")}
should load the version of ASDF that is bundled with your system.
If possible so should @code{(require "ASDF")}.
You may have it load some other version configured by the user,
if you allow such configuration.

@item
If your system provides a mechanism to hook into @code{CL:REQUIRE},
then it would be nice to add ASDF to this hook the same way that
ABCL, CCL, CLISP, CMUCL, ECL, SBCL and SCL do it.
Please send us appropriate code to this end.

@item
You may, like SBCL, have ASDF be implicitly used to require systems
that are bundled with your Lisp distribution.
If you do have a few magic systems that come with your implementation
in a precompiled way such that one should only use the binary version
that goes with your distribution, like SBCL does,
then you should add them in the beginning of @code{wrapping-source-registry}.

@item
If you have magic systems as above, like SBCL does,
then we explicitly ask you to @emph{NOT} distribute
@file{asdf.asd} as part of those magic systems.
You should still include the file @file{asdf.lisp} in your source distribution
and precompile it in your binary distribution,
but @file{asdf.asd} if included at all,
should be secluded from the magic systems,
in a separate file hierarchy.
Alternatively, you may provide the system
after renaming it and its @file{.asd} file to e.g.
@code{asdf-ecl} and @file{asdf-ecl.asd}, or
@code{sb-asdf} and @file{sb-asdf.asd}.
Indeed, if you made @file{asdf.asd} a magic system,
then users would no longer be able to upgrade ASDF using ASDF itself
to some version of their preference that
they maintain independently from your Lisp distribution.

@item
If you do not have any such magic systems, or have other non-magic systems
that you want to bundle with your implementation,
then you may add them to the @code{wrapping-source-registry},
and you are welcome to include @file{asdf.asd} amongst them.
Non-magic systems should be at the back of the @code{wrapping-source-registry}
while magic systems are at the front.

@item
Please send us upstream any patches you make to ASDF itself,
so we can merge them back in for the benefit of your users
when they upgrade to the upstream version.

@end itemize



@section Issues with configuring ASDF

@subsection ``How can I customize where fasl files are stored?''

@xref{Controlling where ASDF saves compiled files}.

Note that in the past there was an add-on to ASDF called
@code{ASDF-binary-locations}, developed by Gary King.
That add-on has been merged into ASDF proper,
then superseded by the @code{asdf-output-translations} facility.

Note that use of @code{asdf-output-translations}
can interfere with one aspect of your systems
--- if your system uses @code{*load-truename*} to find files
(e.g., if you have some data files stored with your program),
then the relocation that this ASDF customization performs
is likely to interfere.
Use @code{asdf:system-relative-pathname} to locate a file
in the source directory of some system, and
use @code{asdf:apply-output-translations} to locate a file
whose pathname has been translated by the facility.

@subsection ``How can I wholly disable the compiler output cache?''

To permanently disable the compiler output cache
for all future runs of ASDF, you can:

@example
mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
echo ':disable-cache' > ~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf
@end example

This assumes that you didn't otherwise configure the ASDF files
(if you did, edit them again),
and don't somehow override the configuration at runtime
with a shell variable (see below) or some other runtime command
(e.g. some call to @code{asdf:initialize-output-translations}).

To disable the compiler output cache in Lisp processes
run by your current shell, try (assuming @code{bash} or @code{zsh})
(on Unix and cygwin only):

@example
export ASDF_OUTPUT_TRANSLATIONS=/:
@end example

To disable the compiler output cache just in the current Lisp process,
use (after loading ASDF but before using it):

@example
(asdf:disable-output-translations)
@end example

@section Issues with using and extending ASDF to define systems

@subsection ``How can I cater for unit-testing in my system?''

ASDF provides a predefined test operation, @code{test-op}.
@xref{Predefined operations of ASDF, test-op}.
The test operation, however, is largely left to the system definer to specify.
@code{test-op} has been
a topic of considerable discussion on the
@url{http://common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel,asdf-devel mailing list},
and on the
@url{https://launchpad.net/asdf,launchpad bug-tracker}.

Here are some guidelines:

@itemize
@item
For a given system, @var{foo}, you will want to define a corresponding
test system, such as @var{foo-test}.  The reason that you will want this
separate system is that ASDF does not out of the box supply components
that are conditionally loaded.  So if you want to have source files
(with the test definitions) that will not be loaded except when testing,
they should be put elsewhere.

@item
The @var{foo-test} system can be defined in an asd file of its own or
together with @var{foo}.  An aesthetic preference against cluttering up
the filesystem with extra asd files should be balanced against the
question of whether one might want to directly load @var{foo-test}.
Typically one would not want to do this except in early stages of
debugging.

@item
Record that testing is implemented by @var{foo-test}.  For example:
@example
(defsystem @var{foo}
   :in-order-to ((test-op (test-op @var{foo-test})))
   ....)

(defsystem @var{foo-test}
   :depends-on (@var{foo} @var{my-test-library} ...)
   ....)
@end example
@end itemize

This procedure will allow you to support users who do not wish to
install your test framework.

One oddity of ASDF is that @code{operate} (@pxref{Operations,operate})
does not return a value.  So in current versions of ASDF there is no
reliable programmatic means of determining whether or not a set of tests
has passed, or which tests have failed.  The user must simply read the
console output.  This limitation has been the subject of much
discussion.

@subsection ``How can I cater for documentation generation in my system?''

The ASDF developers are currently working to add a @code{doc-op}
to the set of predefined ASDF operations.
@xref{Predefined operations of ASDF}.
See also @url{https://bugs.launchpad.net/asdf/+bug/479470}.



@subsection ``How can I maintain non-Lisp (e.g. C) source files?''

See @code{cffi}'s @code{cffi-grovel}.

@anchor{report-bugs}


@subsection ``I want to put my module's files at the top level.  How do I do this?''

By default, the files contained in an asdf module go
in a subdirectory with the same name as the module.
However, this can be overridden by adding a @code{:pathname ""} argument
to the module description.
For example, here is how it could be done
in the spatial-trees ASDF system definition for ASDF 2:

@example
(asdf:defsystem :spatial-trees
  :components
  ((:module base
            :pathname ""
            :components
            ((:file "package")
             (:file "basedefs" :depends-on ("package"))
             (:file "rectangles" :depends-on ("package"))))
   (:module tree-impls
            :depends-on (base)
            :pathname ""
            :components
            ((:file "r-trees")
             (:file "greene-trees" :depends-on ("r-trees"))
             (:file "rstar-trees" :depends-on ("r-trees"))
             (:file "rplus-trees" :depends-on ("r-trees"))
             (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
   (:module viz
            :depends-on (base)
            :pathname ""
            :components
            ((:static-file "spatial-tree-viz.lisp")))
   (:module tests
            :depends-on (base)
            :pathname ""
            :components
            ((:static-file "spatial-tree-test.lisp")))
   (:static-file "LICENCE")
   (:static-file "TODO")))
@end example

All of the files in the @code{tree-impls} module are at the top level,
instead of in a @file{tree-impls/} subdirectory.

Note that the argument to @code{:pathname} can be either a pathname object or a string.
A pathname object can be constructed with the @file{#p"foo/bar/"} syntax,
but this is discouraged because the results of parsing a namestring are not portable.
A pathname can only be portably constructed with such syntax as
@code{#.(make-pathname :directory '(:relative "foo" "bar"))},
and similarly the current directory can only be portably specified as
@code{#.(make-pathname :directory '(:relative))}.
However, as of ASDF 2, you can portably use a string to denote a pathname.
The string will be parsed as a @code{/}-separated path from the current directory,
such that the empty string @code{""} denotes the current directory, and
@code{"foo/bar"} (no trailing @code{/} required in the case of modules)
portably denotes the same subdirectory as above.
When files are specified, the last @code{/}-separated component is interpreted
either as the name component of a pathname
(if the component class specifies a pathname type),
or as a name component plus optional dot-separated type component
(if the component class doesn't specifies a pathname type).

@subsection How do I create a system definition where all the source files have a .cl extension?

Starting with ASDF 2.014.14, you may just pass
the builtin class @code{cl-source-file.cl} as
the @code{:default-component-class} argument to @code{defsystem}:

@lisp
(defsystem my-cl-system
  :default-component-class cl-source-file.cl
  ...)
@end lisp

Another builtin class @code{cl-source-file.lsp} is offered
for files ending in @file{.lsp}.

If you want to use a different extension
for which ASDF doesn't provide builtin support,
or want to support versions of ASDF
earlier than 2.014.14 (but later than 2.000),
you can define a class as follows:

@lisp
;; Prologue: make sure we're using a sane package.
(defpackage :my-asdf-extension
   (:use :asdf :common-lisp)
   (:export #:cl-source-file.lis))
(in-package :my-asdf-extension)

(defclass cl-source-file.lis (cl-source-file)
  ((type :initform "lis")))
@end lisp

Then you can use it as follows:
@lisp
(defsystem my-cl-system
  :default-component-class my-asdf-extension:cl-source-file.lis
  ...)
@end lisp

Of course, if you're in the same package, e.g. in the same file,
you won't need to use the package qualifier before @code{cl-source-file.lis}.
Actually, if all you're doing is defining this class
and using it in the same file without other fancy definitions,
you might skip package complications:

@lisp
(in-package :asdf)
(defclass cl-source-file.lis (cl-source-file)
   ((type :initform "lis")))
(defsystem my-cl-system
  :default-component-class cl-source-file.lis
  ...)
@end lisp

It is possible to achieve the same effect
in a way that supports both ASDF 1 and ASDF 2,
but really, friends don't let friends use ASDF 1.
Please upgrade to ASDF 3.
In short, though: do same as above, but
@emph{before} you use the class in a @code{defsystem},
you also define the following method:

@lisp
(defmethod source-file-type ((f cl-source-file.lis) (s system))
  (declare (ignorable f s))
  "lis")
@end lisp

@comment FIXME: Add a FAQ about how to use a new system class...


@node  TODO list, Inspiration, FAQ, Top
@comment  node-name,  next,  previous,  up
@chapter TODO list

Here is an old list of things to do,
in addition to the bugs that are now tracked on launchpad:
@url{https://launchpad.net/asdf}.

@section Outstanding spec questions, things to add

** packaging systems

*** manual page component?

** style guide for .asd files

You should either use keywords or be careful
with the package that you evaluate defsystem forms in.
Otherwise @code{(defsystem partition ...)}
being read in the @code{cl-user} package
will intern a @code{cl-user:partition} symbol,
which will then collide with the @code{partition:partition} symbol.

Actually there's a hairier packages problem to think about too.
@code{in-order-to} is not a keyword:
if you read @code{defsystem} forms in a package that doesn't use ASDF,
odd things might happen.


** extending defsystem with new options

You might not want to write a whole parser,
but just to add options to the existing syntax.
Reinstate @code{parse-option} or something akin.


** Diagnostics

A ``dry run'' of an operation can be made with the following form:

@lisp
(let ((asdf::*verbose-out* *standard-output*))
  (loop :for (op . comp) :in
    (asdf::traverse (make-instance '<operation-name> :force t)
                    (asdf:find-system <system-name>))
    :do (asdf:explain op comp)))
@end lisp

This uses unexported symbols.
What would be a nice interface for this functionality?

@section Missing bits in implementation

** reuse the same scratch package whenever a system is reloaded from disk

Have a package ASDF-USER instead of all these temporary packages?

** proclamations probably aren't

** A revert function

Other possible interface: have a ``revert'' function akin to @code{make clean}.

@lisp
(asdf:revert 'asdf:compile-op 'araneida)
@end lisp

would delete any files produced by @code{(compile-system :araneida)}.
Of course, it wouldn't be able to do much about stuff in the image itself.

How would this work?

@code{traverse}

There's a difference between a module's dependencies (peers)
and its components (children).
Perhaps there's a similar difference in operations?
For example, @code{(load "use") depends-on (load "macros")} is a peer,
whereas @code{(load "use") depends-on (compile "use")}
is more of a ``subservient'' relationship.

@node  Inspiration, Concept Index, TODO list, Top
@comment  node-name,  next,  previous,  up
@chapter Inspiration

@section mk-defsystem (defsystem-3.x)

We aim to solve basically the same problems as @code{mk-defsystem} does.
However, our architecture for extensibility
better exploits CL language features (and is documented),
and we intend to be portable rather than just widely-ported.
No slight on the @code{mk-defsystem} authors and maintainers is intended here;
that implementation has the unenviable task
of supporting pre-ANSI implementations, which is no longer necessary.

The surface defsystem syntax of asdf is more-or-less compatible with
@code{mk-defsystem}, except that we do not support
the @code{source-foo} and @code{binary-foo} prefixes
for separating source and binary files, and
we advise the removal of all options to specify pathnames.

The @code{mk-defsystem} code for topologically sorting
a module's dependency list was very useful.

@section defsystem-4 proposal

Marco and Peter's proposal for defsystem 4 served as the driver for
many of the features in here.  Notable differences are:

@itemize
@item
We don't specify output files or output file extensions
as part of the system.

If you want to find out what files an operation would create,
ask the operation.

@item
We don't deal with CL packages

If you want to compile in a particular package, use an @code{in-package} form
in that file (ilisp / SLIME will like you more if you do this anyway)

@item
There is no proposal here that @code{defsystem} does version control.

A system has a given version which can be used to check dependencies,
but that's all.
@end itemize

The defsystem 4 proposal tends to look more at the external features,
whereas this one centres on a protocol for system introspection.

@section kmp's ``The Description of Large Systems'', MIT AI Memo 801

Available in updated-for-CL form on the web at
@url{http://nhplace.com/kent/Papers/Large-Systems.html}

In our implementation we borrow kmp's overall @code{PROCESS-OPTIONS}
and concept to deal with creating component trees
from @code{defsystem} surface syntax.
[ this is not true right now, though it used to be and
probably will be again soon ]


@c -------------------


@node Concept Index, Function and Class Index, Inspiration, Top
@unnumbered Concept Index

@printindex cp

@node Function and Class Index, Variable Index, Concept Index, Top
@unnumbered Function and Class Index

@printindex fn

@node Variable Index,  , Function and Class Index, Top
@unnumbered Variable Index

@printindex vr

@bye