README,v 1.39 2006/08/21 10:52:32 crhodes Exp -*- Text -*- The canonical documentation for asdf is in the file asdf.texinfo. The significant overlap between this file and that will one day be resolved by deleting text from this file; in the meantime, please look there before here. asdf: another system definition facility ======================================== * Getting the latest version 0) Decide which version you want. HEAD is the newest version and usually OK, whereas RELEASE is for cautious people (e.g. who already have systems using asdf that they don't want broken), a slightly older version about which none of the HEAD users have complained. 1) Check it out from sourceforge cCLan CVS: 1a) cvs -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan login (no password: just press Enter) 1a.1) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan co -r RELEASE asdf or for the bleeding edge, instead 1a.2) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan co -A asdf If you are tracking the bleeding edge, you may want to subscribe to the cclan-commits mailing list (see ) to receive commit messages and diffs whenever changes are made. For more CVS information, look at http://sourceforge.net/cvs/?group_id=28536 * Getting started - The single file asdf.lisp is all you need to use asdf normally. For maximum convenience you want to have it loaded whenever you start your Lisp implementation, by loading it from the startup script, or dumping a custom core, or something. - The variable asdf:*central-registry* is a list of system directory designators. A system directory designator is a form which will be evaluated whenever a system is to be found, and must evaluate to a directory to look in. For example, you might have (*default-pathname-defaults* "/home/me/cl/systems/" "/usr/share/common-lisp/systems/") (When we say "directory" here, we mean "designator for a pathname with a supplied DIRECTORY component") It is possible to customize the system definition file search. That's considered advanced use, and covered later: search forward for *system-definition-search-functions* - To compile and load a system 'foo', you need to (1) ensure that foo.asd is in one of the directories in *central-registry* (a symlink to the real location of foo.asd is preferred), (2) execute ``(asdf:operate 'asdf:load-op 'foo)'' $ cd /home/me/cl/systems/ $ ln -s ~/src/foo/foo.asd . $ lisp * (asdf:operate 'asdf:load-op 'foo) - To write your own system definitions, look at the test systems in test/ , and read the rest of this. Ignore systems/ which is old and may go away when next I clean up - Syntax is similar to mk-defsystem 3 for straightforward systems, you may only need to remove the :source-pathname option (and replace it with :pathname if the asd file is not in the same place as the system sources) - Join cclan-list@lists.sf.net for discussion, bug reports, questions, etc - cclan.asd and the source files listed therein contain useful extensions for maintainers of systems in the cCLan. If this isn't you, you don't need them - although you may want to look at them anyway - For systems that do complicated things (e.g. compiling C files to load as foreign code), the packages in vn-cclan may provide some guidance. db-sockets, for example, is known to do outlandish things with preprocessors http://ww.telent.net/cliki/vn-cclan * Concepts This system definition utility talks in terms of 'components' and 'operations'. Components form systems: a component represents a source file, or a collection of components. A system is therefore a component, recursively formed of a tree of subcomponents. Operations are instantiated then performed on the nodes of a tree to do things like - compile all its files - load the files into a running lisp environment - copy its source files somewhere else Operations can be invoked directly, or examined to see what their effects would be without performing them. There are a bunch of methods specialised on operation and component type which actually do the grunt work. asdf is extensible to new operations and to new component types. This allows the addition of behaviours: for example, a new component could be added for Java JAR archives, and methods specialised on compile-op added for it that would accomplish the relevant actions. Users defining their own operations and component types should inherit from the asdf base classes asdf:operation and asdf:component respectively. * Inspiration ** mk-defsystem (defsystem-3.x) We aim to solve basically the same problems as mk-defsystem does. However, our architecture for extensibility better exploits CL language features (and is documented), and we intend to be portable rather than just widely-ported. No slight on the mk-defsystem authors and maintainers is intended here; that implementation has the unenviable task of supporting non-ANSI implementations, which I propose to ignore. The surface defsystem syntax of asdf is more-or-less compatible with mk-defsystem The mk-defsystem code for topologically sorting a module's dependency list was very useful. ** defsystem-4 proposal Marco and Peter's proposal for defsystem 4 served as the driver for many of the features in here. Notable differences are - we don't specify output files or output file extensions as part of the system If you want to find out what files an operation would create, ask the operation - we don't deal with CL packages If you want to compile in a particular package, use an in-package form in that file (ilisp will like you more if you do this anyway) - there is no proposal here that defsystem does version control. A system has a given version which can be used to check dependencies, but that's all. The defsystem 4 proposal tends to look more at the external features, whereas this one centres on a protocol for system introspection. ** kmp's "The Description of Large Systems", MIT AI Memu 801 Available in updated-for-CL form on the web at http://world.std.com/~pitman/Papers/Large-Systems.html In our implementation we borrow kmp's overall PROCESS-OPTIONS and concept to deal with creating component trees from defsystem surface syntax. [ this is not true right now, though it used to be and probably will be again soon ] * The Objects ** component *** Component Attributes **** A name (required) This is a string or a symbol. If a symbol, its name is taken and lowercased. The name must be a suitable value for the :name initarg to make-pathname in whatever filesystem the system is to be found. The lower-casing-symbols behaviour is unconventional, but was selected after some consideration. Observations suggest that the type of systems we want to support either have lowercase as customary case (Unix, Mac, windows) or silently convert lowercase to uppercase (lpns), so this makes more sense than attempting to use :case :common, which is reported not to work on some implementations **** a version identifier (optional) This is used by the test-system-version operation (see later). **** *features* required Traditionally defsystem users have used reader conditionals to include or exclude specific per-implementation files. This means that any single implementation cannot read the entire system, which becomes a problem if it doesn't wish to compile it, but instead for example to create an archive file containing all the sources, as it will omit to process the system-dependent sources for other systems. Each component in an asdf system may therefore specify features using the same syntax as #+ does, and it will (somehow) be ignored for certain operations unless the feature conditional matches **** dependencies on its siblings (optional but often necessary) There is an excitingly complicated relationship between the initarg and the method that you use to ask about dependencies Dependencies are between (operation component) pairs. In your initargs, you can say :in-order-to ((compile-op (load-op "a" "b") (compile-op "c")) (load-op (load-op "foo"))) - before performing compile-op on this component, we must perform load-op on "a" and "b", and compile-op on c, - before performing load-op, we have to load "foo" The syntax is approximately (this-op {(other-op required-components)}+) required-components := component-name | (required-components required-components) component-name := string | (:version string minimum-version-object) [ This is on a par with what ACL defsystem does. mk-defsystem is less general: it has an implied dependency for all x, (load x) depends on (compile x) and using a :depends-on argument to say that b depends on a _actually_ means that (compile b) depends on (load a) This is insufficient for e.g. the McCLIM system, which requires that all the files are loaded before any of them can be compiled ] In asdf, the dependency information for a given component and operation can be queried using (component-depends-on operation component), which returns a list ((load-op "a") (load-op "b") (compile-op "c") ...) component-depends-on can be subclassed for more specific component/operation types: these need to (call-next-method) and append the answer to their dependency, unless they have a good reason for completely overriding the default dependencies (If it weren't for CLISP, we'd be using a LIST method combination to do this transparently. But, we need to support CLISP. If you have the time for some CLISP hacking, I'm sure they'd welcome your fixes) **** a pathname This is optional and if absent will be inferred from name, type (the subclass of source-file), and the location of parent. The rules for this inference are: (for source-files) - the host is taken from the parent - pathname type is (source-file-type component system) - the pathname case option is :local - the pathname is merged against the parent (for modules) - the host is taken from the parent - the name and type are NIL - the directory is (:relative component-name) - the pathname case option is :local - the pathname is merged against the parent Note that the DEFSYSTEM operator (used to create a "top-level" system) does additional processing to set the filesystem location of the top component in that system. This is detailed elsewhere The answer to the frequently asked question "how do I create a system definition where all the source files have a .cl extension" is thus (defmethod source-file-type ((c cl-source-file) (s (eql (find-system 'my-sys)))) "cl") **** properties (optional) Packaging systems often require information about files or systems additional to that specified here. Programs that create vendor packages out of asdf systems therefore have to create "placeholder" information to satisfy these systems. Sometimes the creator of an asdf system may know the additional information and wish to provide it directly. (component-property component property-name) and associated setf method will allow the programmatic update of this information. Property names are compared as if by EQL, so use symbols or keywords or something ** Subclasses of component *** 'source-file' A source file is any file that the system does not know how to generate from other components of the system. (Note that this is not necessarily the same thing as "a file containing data that is typically fed to a compiler". If a file is generated by some pre-processor stage (e.g. a ".h" file from ".h.in" by autoconf) then it is not, by this definition, a source file. Conversely, we might have a graphic file that cannot be automatically regenerated, or a proprietary shared library that we received as a binary: these do count as source files for our purposes. All suggestions for better terminology gratefully received) Subclasses of source-file exist for various languages. *** 'module', a collection of sub-components This has extra slots for :components - the components contained in this module :default-component-class - for child components which don't specify their class explicitly :if-component-dep-fails takes one of the values :fail, :try-next, :ignore (default value is :fail). The other values can be used for implementing conditional compilation based on implementation *features*, where it is not necessary for all files in a module to be compiled The default operation knows how to traverse a module, so most operations will not need to provide methods specialised on modules. The module may be subclassed to represent components such as foreign-language linked libraries or archive files. *** system, subclasses module A system is a module with a few extra attributes for documentation purposes. In behaviour, it's usually identical. Users can create new classes for their systems: the default defsystem macro takes a :classs keyword argument. ** operation An operation is instantiated whenever the user asks that an operation be performed, inspected, or etc. The operation object contains whatever state is relevant to this purpose (perhaps a list of visited nodes, for example) but primarily is a nice thing to specialise operation methods on and easier than having them all be EQL methods. There are no differences between standard operations and user-defined operations, except that the user is respectfully requested to keep his (or more importantly, our) package namespace clean *** invoking operations (operate operation system &rest keywords-args) keyword-args are passed to the make-instance call when creating the operation: valid keywords depend on the initargs that the operation is defined to accept. Note that dependencies may cause the operation to invoke other operations on the system or its components: the new operation will be created with the same initargs as the original one. oos is accepted as a synonym for operate *** standard operations **** feature-dependent-op This is not intended to be instantiated directly, but other operations may inherit from it. An instance of feature-dependent-op will ignore any components which have a `features' attribute, unless the feature combination it designates is satisfied by *features* See the earlier explanation about the component features attribute for more information **** compile-op &key proclamations If proclamations are supplied, they will be proclaimed. This is a good place to specify optimization settings When creating a new component, you should provide methods for this. If you invoke compile-op as a user, component dependencies often mean you may get some parts of the system loaded. This may not necessarily be the whole thing, though; for your own sanity it is recommended that you use load-op if you want to load a system. **** load-op &key proclamations The default methods for load-op compile files before loading them. For parity, your own methods on new component types should probably do so too **** load-source-op This method will load the source for the files in a module even if the source files have been compiled. Systems sometimes have knotty dependencies which require that sources are loaded before they can be compiled. This is how you do that. If you are creating a component type, you need to implement this operation - at least, where meaningful. **** test-system-version &key minimum Asks the system whether it satisfies a version requirement. The default method accepts a string, which is expected to contain of a number of integers separated by #\. characters. The method is not recursive. The component satisfies the version dependency if it has the same major number as required and each of its sub-versions is greater than or equal to the sub-version number required. (defun version-satisfies (x y) (labels ((bigger (x y) (cond ((not y) t) ((not x) nil) ((> (car x) (car y)) t) ((= (car x) (car y)) (bigger (cdr x) (cdr y)))))) (and (= (car x) (car y)) (or (not (cdr y)) (bigger (cdr x) (cdr y)))))) If that doesn't work for your system, you can override it. I hope yoyu have as much fun writing the new method as #lisp did reimplementing this one. *** Creating new operations subclass operation, provide methods for source-file for - output-files - perform The perform method must call output-files to find out where to put its files, because the user is allowed to override output-files for local policy - explain - operation-done-p, if you don't like the default one * Writing system definitions ** System designators System designators are strings or symbols and behave just like any other component names (including case conversion) ** find-system Given a system designator, find-system finds an actual system - either in memory, or in a file on the disk. It funcalls each element in the *system-definition-search-functions* list, expecting a pathname to be returned. If a suitable file exists, it is loaded if - there is no system of that name in memory, - the file's last-modified time exceeds the last-modified time of the system in memory When system definitions are loaded from .asd files, a new scratch package is created for them to load into, so that different systems do not overwrite each others operations. The user may also wish to (and is recommended to) include defpackage and in-package forms in his system definition files, however, so that they can be loaded manually if need be. It is not recommended to use the CL-USER package for this purpose, as definitions made in this package will affect the parsing of asdf systems. For convenience in the normal case, and for backward compatibility with the spirit of mk-defsystem, the default contents of *system-definition-search-functions* is a function called sysdef-central-registry-search. This looks in each of the directories given by evaluating members of *central-registry*, for a file whose name is the name of the system and whose type is "asd". The first such file is returned, whether or not it turns out to actually define the appropriate system ** Syntax Systems can always be constructed programmatically by instantiating components using make-instance. For most purposes, however, it is likely that people will want a static defystem form. asdf is based around the principle that components should not have to know defsystem syntax. That is, the initargs that a component accepts are not necessarily related to the defsystem form which creates it. A defsystem parser must implement a `defsystem' macro, which can be named for compatibility with whatever other system definition utility is being emulated. It should instantiate components in accordance with whatever language it accepts, and register the topmost component using REGISTER-SYSTEM *** Native syntax The native syntax is inspired by mk-defsystem, to the extent that it should be possible to take most straightforward mk- system definitions and run them with only light editing. For my convenience, this turns out to be basically the same as the initargs to the various components, with a few extensions for convenience system-definition := ( defsystem system-designator {option}* ) option := :components component-list | :pathname pathname | :default-component-class | :perform method-form | :explain method-form | :output-files method-form | :operation-done-p method-form | :depends-on ( {simple-component-name}* ) | :serial [ t | nil ] | :in-order-to ( {dependency}+ ) component-list := ( {component-def}* ) component-def := simple-component-name | ( component-type name {option}* ) component-type := :module | :file | :system | other-component-type dependency := (dependent-op {requirement}+) requirement := (required-op {required-component}+) | (feature feature-name) dependent-op := operation-name required-op := operation-name | feature For example (defsystem "foo" :version "1.0" :components ((:module "foo" :components ((:file "bar") (:file"baz") (:file "quux")) :perform (compile-op :after (op c) (do-something c)) :explain (compile-op :after (op c) (explain-something c))) (:file "blah"))) The method-form tokens need explaining: esentially, :perform (compile-op :after (op c) (do-something c)) :explain (compile-op :after (op c) (explain-something c))) has the effect of (defmethod perform :after ((op compile-op) (c (eql ...))) (do-something c)) (defmethod explain :after ((op compile-op) (c (eql ...))) (explain-something c)) where ... is the component in question; note that although this also supports :before methods, they may not do what you want them to - a :before method on perform ((op compile-op) (c (eql ...))) will run after all the dependencies and sub-components have been processed, but before the component in question has been compiled. **** Serial dependencies If the `:serial t' option is specified for a module, asdf will add dependencies for each each child component, on all the children textually preceding it. This is done as if by :depends-on :components ((:file "a") (:file "b") (:file "c")) :serial t is equivalent to :components ((:file "a") (:file "b" :depends-on ("a")) (:file "c" :depends-on ("a" "b"))) have all the **** Source location The :pathname option is optional in all cases for native-syntax systems, and in the usual case the user is recommended not to supply it. If it is not supplied for the top-level form, defsystem will set it from - The host/device/directory parts of *load-truename*, if it is bound - *default-pathname-defaults*, otherwise If a system is being redefined, the top-level pathname will be - changed, if explicitly supplied or obtained from *load-truename* - changed if it had previously been set from *default-pathname-defaults* - left as before, if it had previously been set from *load-truename* and *load-truename* is not now bound These rules are designed so that (i) find-system will load a system from disk and have its pathname default to the right place, (ii) this pathname information will not be overwritten with *default-pathname-defaults* (which could be somewhere else altogether) if the user loads up the .asd file into his editor and interactively re-evaluates that form * Error handling It is an error to define a system incorrectly: an implementation may detect this and signal a generalised instance of SYSTEM-DEFINITION-ERROR. Operations may go wrong (for example when source files contain errors). These are signalled using generalised instances of OPERATION-ERROR, with condition readers ERROR-COMPONENT and ERROR-OPERATION for the component and operation which erred. * Compilation error and warning handling ASDF checks for warnings and errors when a file is compiled. The variables *compile-file-warnings-behaviour* and *compile-file-errors-behavior* controls the handling of any such events. The valid values for these variables are :error, :warn, and :ignore. ---------------------------------------------------------- TODO List ---------------------------------------------------------- * Outstanding spec questions, things to add ** packaging systems *** manual page component? ** style guide for .asd files You should either use keywords or be careful with the package that you evaluate defsystem forms in. Otherwise (defsystem partition ...) being read in the cl-user package will intern a cl-user:partition symbol, which will then collide with the partition:partition symbol. Actually there's a hairier packages problem to think about too. in-order-to is not a keyword: if you read defsystem forms in a package that doesn't use ASDF, odd things might happen ** extending defsystem with new options You might not want to write a whole parser, but just to add options to the existing syntax. Reinstate parse-option or something akin ** document all the error classes ** what to do with compile-file failure Should check the primary return value from compile-file and see if that gets us any closer to a sensible error handling strategy ** foreign files lift unix-dso stuff from db-sockets ** Diagnostics A "dry run" of an operation can be made with the following form: (traverse (make-instance ') (find-system ) 'explain) This uses unexported symbols. What would be a nice interface for this functionality? ** patches Sometimes one wants to * missing bits in implementation ** all of the above ** reuse the same scratch package whenever a system is reloaded from disk ** rules for system pathname defaulting are not yet implemented properly ** proclamations probably aren't ** when a system is reloaded with fewer components than it previously had, odd things happen we should do something inventive when processing a defsystem form, like take the list of kids and setf the slot to nil, then transfer children from old to new list as they're found ** traverse may become a normal function If you're defining methods on traverse, speak up. ** a lot of load-op methods can be rewritten to use input-files so should be. ** (stuff that might happen later) *** david lichteblau's patch for symlink resolution? *** Propagation of the :force option. ``I notice that (oos 'compile-op :araneida :force t) also forces compilation of every other system the :araneida system depends on. This is rarely useful to me; usually, when I want to force recompilation of something more than a single source file, I want to recompile only one system. So it would be more useful to have make-sub-operation refuse to propagate ":force t" to other systems, and propagate only something like ":force :recursively". '' Ideally what we actually want is some kind of criterion that says to which systems (and which operations) a :force switch will propagate. The problem is perhaps that 'force' is a pretty meaningless concept. How obvious is it that "load :force t" should force _compilation_? But we don't really have the right dependency setup for the user to compile :force t and expect it to work (files will not be loaded after compilation, so the compile environment for subsequent files will be emptier than it needs to be) What does the user actually want to do when he forces? Usually, for me, update for use with a new version of the lisp compiler. Perhaps for recovery when he suspects that something has gone wrong. Or else when he's changed compilation options or configuration in some way that's not reflected in the dependency graph. Other possible interface: have a 'revert' function akin to 'make clean' (asdf:revert 'asdf:compile-op 'araneida) would delete any files produced by 'compile-op 'araneida. Of course, it wouldn't be able to do much about stuff in the image itself. How would this work? traverse There's a difference between a module's dependencies (peers) and its components (children). Perhaps there's a similar difference in operations? For example, (load "use") depends-on (load "macros") is a peer, whereas (load "use") depends-on (compile "use") is more of a `subservient' relationship.