1 README,v 1.35 2003/08/05 23:00:32 kevinrosenberg Exp -*- Text -*-
4 asdf: another system definition facility
5 ========================================
7 * Getting the latest version
9 0) Decide which version you want. HEAD is the newest version and
10 usually OK, whereas RELEASE is for cautious people (e.g. who already
11 have systems using asdf that they don't want broken), a slightly older
12 version about which none of the HEAD users have complained.
14 1) Check it out from sourceforge cCLan CVS:
16 1a) cvs -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan login
17 (no password: just press Enter)
19 1a.1) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan
22 or for the bleeding edge, instead
24 1a.2) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan
27 If you are tracking the bleeding edge, you may want to subscribe to
28 the cclan-commits mailing list (see
29 <URL:http://sourceforge.net/mail/?group_id=28536>) to receive commit
30 messages and diffs whenever changes are made.
32 For more CVS information, look at http://sourceforge.net/cvs/?group_id=28536
37 - The single file asdf.lisp is all you need to use asdf normally. For
38 maximum convenience you want to have it loaded whenever you start your
39 Lisp implementation, by loading it from the startup script, or dumping
40 a custom core, or something.
42 - The variable asdf:*central-registry* is a list of system directory
43 designators. A system directory designator is a form which will be
44 evaluated whenever a system is to be found, and must evaluate to a
45 directory to look in. For example, you might have
47 (*default-pathname-defaults* "/home/me/cl/systems/"
48 "/usr/share/common-lisp/systems/")
50 (When we say "directory" here, we mean "designator for a pathname
51 with a supplied DIRECTORY component")
53 It is possible to customize the system definition file search.
54 That's considered advanced use, and covered later: search forward
55 for *system-definition-search-functions*
57 - To compile and load a system 'foo', you need to (1) ensure that
58 foo.asd is in one of the directories in *central-registry* (a
59 symlink to the real location of foo.asd is preferred), (2) execute
60 ``(asdf:operate 'asdf:load-op 'foo)''
62 $ cd /home/me/cl/systems/
63 $ ln -s ~/src/foo/foo.asd .
65 * (asdf:operate 'asdf:load-op 'foo)
67 - To write your own system definitions, look at the test systems in
68 test/ , and read the rest of this. Ignore systems/ which is old
69 and may go away when next I clean up
71 - Syntax is similar to mk-defsystem 3 for straightforward systems, you
72 may only need to remove the :source-pathname option (and replace it
73 with :pathname if the asd file is not in the same place as the
76 - Join cclan-list@lists.sf.net for discussion, bug reports, questions, etc
78 - cclan.asd and the source files listed therein contain useful extensions
79 for maintainers of systems in the cCLan. If this isn't you, you
80 don't need them - although you may want to look at them anyway
82 - For systems that do complicated things (e.g. compiling C files to
83 load as foreign code), the packages in vn-cclan may provide some
84 guidance. db-sockets, for example, is known to do outlandish things
87 http://ww.telent.net/cliki/vn-cclan
93 This system definition utility talks in terms of 'components' and
96 Components form systems: a component represents a source file, or a
97 collection of components. A system is therefore a component,
98 recursively formed of a tree of subcomponents.
100 Operations are instantiated then performed on the nodes of a tree to
103 - compile all its files
104 - load the files into a running lisp environment
105 - copy its source files somewhere else
107 Operations can be invoked directly, or examined to see what their
108 effects would be without performing them. There are a bunch of
109 methods specialised on operation and component type which actually do
112 asdf is extensible to new operations and to new component types. This
113 allows the addition of behaviours: for example, a new component could
114 be added for Java JAR archives, and methods specialised on
115 compile-op added for it that would accomplish the relevant
120 ** mk-defsystem (defsystem-3.x)
122 We aim to solve basically the same problems as mk-defsystem does.
123 However, our architecture for extensibility better exploits CL
124 language features (and is documented), and we intend to be portable
125 rather than just widely-ported. No slight on the mk-defsystem authors
126 and maintainers is intended here; that implementation has the
127 unenviable task of supporting non-ANSI implementations, which I
130 The surface defsystem syntax of asdf is more-or-less compatible with
133 The mk-defsystem code for topologically sorting a module's dependency
134 list was very useful.
136 ** defsystem-4 proposal
138 Marco and Peter's proposal for defsystem 4 served as the driver for
139 many of the features in here. Notable differences are
141 - we don't specify output files or output file extensions as part of
144 If you want to find out what files an operation would create, ask
147 - we don't deal with CL packages
149 If you want to compile in a particular package, use an in-package
150 form in that file (ilisp will like you more if you do this anyway)
152 - there is no proposal here that defsystem does version control.
154 A system has a given version which can be used to check
155 dependencies, but that's all.
157 The defsystem 4 proposal tends to look more at the external features,
158 whereas this one centres on a protocol for system introspection.
160 ** kmp's "The Description of Large Systems", MIT AI Memu 801
162 Available in updated-for-CL form on the web at
163 http://world.std.com/~pitman/Papers/Large-Systems.html
165 In our implementation we borrow kmp's overall PROCESS-OPTIONS and
166 concept to deal with creating component trees from defsystem surface
167 syntax. [ this is not true right now, though it used to be and
168 probably will be again soon ]
175 *** Component Attributes
177 **** A name (required)
179 This is a string or a symbol. If a symbol, its name is taken and
180 lowercased. The name must be a suitable value for the :name initarg
181 to make-pathname in whatever filesystem the system is to be found.
183 The lower-casing-symbols behaviour is unconventional, but was selected
184 after some consideration. Observations suggest that the type of
185 systems we want to support either have lowercase as customary case
186 (Unix, Mac, windows) or silently convert lowercase to uppercase
187 (lpns), so this makes more sense than attempting to use :case :common,
188 which is reported not to work on some implementations
190 **** a version identifier (optional)
192 This is used by the test-system-version operation (see later).
194 **** *features* required
196 Traditionally defsystem users have used reader conditionals to include
197 or exclude specific per-implementation files. This means that any
198 single implementation cannot read the entire system, which becomes a
199 problem if it doesn't wish to compile it, but instead for example to
200 create an archive file containing all the sources, as it will omit to
201 process the system-dependent sources for other systems.
203 Each component in an asdf system may therefore specify features using
204 the same syntax as #+ does, and it will (somehow) be ignored for
205 certain operations unless the feature conditional matches
207 **** dependencies on its siblings (optional but often necessary)
209 There is an excitingly complicated relationship between the initarg
210 and the method that you use to ask about dependencies
212 Dependencies are between (operation component) pairs. In your
213 initargs, you can say
215 :in-order-to ((compile-op (load-op "a" "b") (compile-op "c"))
216 (load-op (load-op "foo")))
218 - before performing compile-op on this component, we must perform
219 load-op on "a" and "b", and compile-op on c, - before performing
220 load-op, we have to load "foo"
222 The syntax is approximately
224 (this-op {(other-op required-components)}+)
226 required-components := component-name
227 | (required-components required-components)
229 component-name := string
230 | (:version string minimum-version-object)
232 [ This is on a par with what ACL defsystem does. mk-defsystem is less
233 general: it has an implied dependency
235 for all x, (load x) depends on (compile x)
237 and using a :depends-on argument to say that b depends on a _actually_
240 (compile b) depends on (load a)
242 This is insufficient for e.g. the McCLIM system, which requires that
243 all the files are loaded before any of them can be compiled ]
245 In asdf, the dependency information for a given component and
246 operation can be queried using (component-depends-on operation
247 component), which returns a list
249 ((load-op "a") (load-op "b") (compile-op "c") ...)
251 component-depends-on can be subclassed for more specific
252 component/operation types: these need to (call-next-method) and append
253 the answer to their dependency, unless they have a good reason for
254 completely overriding the default dependencies
256 (If it weren't for CLISP, we'd be using a LIST method combination to
257 do this transparently. But, we need to support CLISP. If you have
258 the time for some CLISP hacking, I'm sure they'd welcome your fixes)
262 This is optional and if absent will be inferred from name, type (the
263 subclass of source-file), and the location of parent.
265 The rules for this inference are:
268 - the host is taken from the parent
269 - pathname type is (source-file-type component system)
270 - the pathname case option is :local
271 - the pathname is merged against the parent
274 - the host is taken from the parent
275 - the name and type are NIL
276 - the directory is (:relative component-name)
277 - the pathname case option is :local
278 - the pathname is merged against the parent
280 Note that the DEFSYSTEM operator (used to create a "top-level" system)
281 does additional processing to set the filesystem location of the
282 top component in that system. This is detailed elsewhere
284 The answer to the frequently asked question "how do I create a system
285 definition where all the source files have a .cl extension" is thus
287 (defmethod source-file-type ((c cl-source-file) (s (eql (find-system 'my-sys))))
290 **** properties (optional)
292 Packaging systems often require information about files or systems
293 additional to that specified here. Programs that create vendor
294 packages out of asdf systems therefore have to create "placeholder"
295 information to satisfy these systems. Sometimes the creator of an
296 asdf system may know the additional information and wish to provide it
299 (component-property component property-name) and associated setf method
300 will allow the programmatic update of this information. Property
301 names are compared as if by EQL, so use symbols or keywords or something
303 ** Subclasses of component
307 A source file is any file that the system does not know how to
308 generate from other components of the system.
310 (Note that this is not necessarily the same thing as "a file
311 containing data that is typically fed to a compiler". If a file is
312 generated by some pre-processor stage (e.g. a ".h" file from ".h.in"
313 by autoconf) then it is not, by this definition, a source file.
314 Conversely, we might have a graphic file that cannot be automatically
315 regenerated, or a proprietary shared library that we received as a
316 binary: these do count as source files for our purposes. All
317 suggestions for better terminology gratefully received)
319 Subclasses of source-file exist for various languages.
321 *** 'module', a collection of sub-components
323 This has extra slots for
325 :components - the components contained in this module
327 :default-component-class - for child components which don't specify
328 their class explicitly
330 :if-component-dep-fails takes one of the values :fail, :try-next, :ignore
331 (default value is :fail). The other values can be used for implementing
332 conditional compilation based on implementation *features*, where
333 it is not necessary for all files in a module to be compiled
335 The default operation knows how to traverse a module, so most
336 operations will not need to provide methods specialised on modules.
338 The module may be subclassed to represent components such as
339 foreign-language linked libraries or archive files.
341 *** system, subclasses module
343 A system is a module with a few extra attributes for documentation
344 purposes. In behaviour, it's usually identical.
346 Users can create new classes for their systems: the default defsystem
347 macro takes a :classs keyword argument.
352 An operation is instantiated whenever the user asks that an operation
353 be performed, inspected, or etc. The operation object contains
354 whatever state is relevant to this purpose (perhaps a list of visited
355 nodes, for example) but primarily is a nice thing to specialise
356 operation methods on and easier than having them all be EQL methods.
358 There are no differences between standard operations and user-defined
359 operations, except that the user is respectfully requested to keep his
360 (or more importantly, our) package namespace clean
362 *** invoking operations
364 (operate operation system &rest keywords-args)
366 keyword-args are passed to the make-instance call when creating the
367 operation: valid keywords depend on the initargs that the operation is
368 defined to accept. Note that dependencies may cause the operation to
369 invoke other operations on the system or its components: the new
370 operation will be created with the same initargs as the original one.
372 oos is accepted as a synonym for operate
374 *** standard operations
376 **** feature-dependent-op
378 This is not intended to be instantiated directly, but other operations
379 may inherit from it. An instance of feature-dependent-op will ignore
380 any components which have a `features' attribute, unless the feature
381 combination it designates is satisfied by *features*
383 See the earlier explanation about the component features attribute for
386 **** compile-op &key proclamations
388 If proclamations are supplied, they will be proclaimed. This is a
389 good place to specify optimization settings
391 When creating a new component, you should provide methods for this.
393 If you invoke compile-op as a user, component dependencies often mean
394 you may get some parts of the system loaded. This may not necessarily
395 be the whole thing, though; for your own sanity it is recommended that
396 you use load-op if you want to load a system.
398 **** load-op &key proclamations
400 The default methods for load-op compile files before loading them.
401 For parity, your own methods on new component types should probably do
406 This method will load the source for the files in a module even if the
407 source files have been compiled. Systems sometimes have knotty
408 dependencies which require that sources are loaded before they can be
409 compiled. This is how you do that.
411 If you are creating a component type, you need to implement this
412 operation - at least, where meaningful.
414 **** test-system-version &key minimum
416 Asks the system whether it satisfies a version requirement.
418 The default method accepts a string, which is expected to contain of a
419 number of integers separated by #\. characters. The method is not
420 recursive. The component satisfies the version dependency if it has
421 the same major number as required and each of its sub-versions is
422 greater than or equal to the sub-version number required.
424 (defun version-satisfies (x y)
425 (labels ((bigger (x y)
428 ((> (car x) (car y)) t)
430 (bigger (cdr x) (cdr y))))))
431 (and (= (car x) (car y))
432 (or (not (cdr y)) (bigger (cdr x) (cdr y))))))
434 If that doesn't work for your system, you can override it. I hope
435 yoyu have as much fun writing the new method as #lisp did
436 reimplementing this one.
438 *** Creating new operations
440 subclass operation, provide methods for source-file for
444 The perform method must call output-files to find out where to
445 put its files, because the user is allowed to override output-files
448 - operation-done-p, if you don't like the default one
450 * Writing system definitions
452 ** System designators
454 System designators are strings or symbols and behave just like
455 any other component names (including case conversion)
459 Given a system designator, find-system finds an actual system - either
460 in memory, or in a file on the disk. It funcalls each element in the
461 *system-definition-search-functions* list, expecting a pathname to be
464 If a suitable file exists, it is loaded if
466 - there is no system of that name in memory,
467 - the file's last-modified time exceeds the last-modified time of the
470 When system definitions are loaded from .asd files, a new scratch
471 package is created for them to load into, so that different systems do
472 not overwrite each others operations. The user may also wish to (and
473 is recommended to) include defpackage and in-package forms in his
474 system definition files, however, so that they can be loaded manually
477 For convenience in the normal case, and for backward compatibility
478 with the spirit of mk-defsystem, the default contents of
479 *system-definition-search-functions* is a function called
480 sysdef-central-registry-search. This looks in each of the directories
481 given by evaluating members of *central-registry*, for a file whose
482 name is the name of the system and whose type is "asd". The first
483 such file is returned, whether or not it turns out to actually define
484 the appropriate system
490 Systems can always be constructed programmatically by instantiating
491 components using make-instance. For most purposes, however, it is
492 likely that people will want a static defystem form.
494 asdf is based around the principle that components should not have to
495 know defsystem syntax. That is, the initargs that a component accepts
496 are not necessarily related to the defsystem form which creates it.
498 A defsystem parser must implement a `defsystem' macro, which can
499 be named for compatibility with whatever other system definition
500 utility is being emulated. It should instantiate components in
501 accordance with whatever language it accepts, and register the topmost
502 component using REGISTER-SYSTEM
506 The native syntax is inspired by mk-defsystem, to the extent that it
507 should be possible to take most straightforward mk- system definitions
508 and run them with only light editing. For my convenience, this turns
509 out to be basically the same as the initargs to the various
510 components, with a few extensions for convenience
512 system-definition := ( defsystem system-designator {option}* )
514 option := :components component-list
516 | :default-component-class
517 | :perform method-form
518 | :explain method-form
519 | :output-files method-form
520 | :operation-done-p method-form
521 | :depends-on ( {simple-component-name}* )
522 | :serial [ t | nil ]
523 | :in-order-to ( {dependency}+ )
525 component-list := ( {component-def}* )
527 component-def := simple-component-name
528 | ( component-type name {option}* )
530 component-type := :module | :file | :system | other-component-type
532 dependency := (dependent-op {requirement}+)
533 requirement := (required-op {required-component}+)
534 | (feature feature-name)
535 dependent-op := operation-name
536 required-op := operation-name | feature
542 :components ((:module "foo" :components ((:file "bar") (:file"baz")
544 :perform (compile-op :after (op c)
546 :explain (compile-op :after (op c)
547 (explain-something c)))
551 The method-form tokens need explaining: esentially,
553 :perform (compile-op :after (op c)
555 :explain (compile-op :after (op c)
556 (explain-something c)))
559 (defmethod perform :after ((op compile-op) (c (eql ...)))
561 (defmethod explain :after ((op compile-op) (c (eql ...)))
562 (explain-something c))
564 where ... is the component in question; note that although this also
565 supports :before methods, they may not do what you want them to - a
566 :before method on perform ((op compile-op) (c (eql ...))) will run
567 after all the dependencies and sub-components have been processed, but
568 before the component in question has been compiled.
570 **** Serial dependencies
572 If the `:serial t' option is specified for a module, asdf will add
573 dependencies for each each child component, on all the children
574 textually preceding it. This is done as if by :depends-on
576 :components ((:file "a") (:file "b") (:file "c"))
580 :components ((:file "a")
581 (:file "b" :depends-on ("a"))
582 (:file "c" :depends-on ("a" "b")))
590 The :pathname option is optional in all cases for native-syntax
591 systems, and in the usual case the user is recommended not to supply
592 it. If it is not supplied for the top-level form, defsystem will set
595 - The host/device/directory parts of *load-truename*, if it is bound
596 - *default-pathname-defaults*, otherwise
598 If a system is being redefined, the top-level pathname will be
600 - changed, if explicitly supplied or obtained from *load-truename*
601 - changed if it had previously been set from *default-pathname-defaults*
602 - left as before, if it had previously been set from *load-truename*
603 and *load-truename* is not now bound
605 These rules are designed so that (i) find-system will load a system
606 from disk and have its pathname default to the right place, (ii)
607 this pathname information will not be overwritten with
608 *default-pathname-defaults* (which could be somewhere else altogether)
609 if the user loads up the .asd file into his editor and
610 interactively re-evaluates that form
614 It is an error to define a system incorrectly: an implementation may
615 detect this and signal a generalised instance of
616 SYSTEM-DEFINITION-ERROR
618 Operations may go wrong (for example when source files contain
619 errors). These are signalled using generalised instances of
622 * Compilation error and warning handling
624 ASDF checks for warnings and errors when a file is compiled. The
625 variables *compile-file-warnings-behaviour* and
626 *compile-file-errors-behavior* controls the handling of any such
627 events. The valid values for these variables are :error, :warn, and
630 ----------------------------------------------------------
632 ----------------------------------------------------------
634 * Outstanding spec questions, things to add
638 *** manual page component?
640 ** style guide for .asd files
642 You should either use keywords or be careful with the package that you
643 evaluate defsystem forms in. Otherwise (defsystem partition ...)
644 being read in the cl-user package will intern a cl-user:partition
645 symbol, which will then collide with the partition:partition symbol.
647 Actually there's a hairier packages problem to think about too.
648 in-order-to is not a keyword: if you read defsystem forms in a package
649 that doesn't use ASDF, odd things might happen
651 ** extending defsystem with new options
653 You might not want to write a whole parser, but just to add options to
654 the existing syntax. Reinstate parse-option or something akin
656 ** document all the error classes
658 ** what to do with compile-file failure
660 Should check the primary return value from compile-file and see if
661 that gets us any closer to a sensible error handling strategy
665 lift unix-dso stuff from db-sockets
669 A "dry run" of an operation can be made with the following form:
671 (traverse (make-instance '<operation-name>)
672 (find-system <system-name>)
675 This uses unexported symbols. What would be a nice interface for this
678 * missing bits in implementation
681 ** reuse the same scratch package whenever a system is reloaded from disk
682 ** rules for system pathname defaulting are not yet implemented properly
683 ** proclamations probably aren't
684 ** when a system is reloaded with fewer components than it previously
685 had, odd things happen
687 we should do something inventive when processing a defsystem form,
688 like take the list of kids and setf the slot to nil, then transfer
689 children from old to new list as they're found
691 ** traverse may become a normal function
693 If you're defining methods on traverse, speak up.
696 ** a lot of load-op methods can be rewritten to use input-files
701 ** (stuff that might happen later)
703 *** david lichteblau's patch for symlink resolution?
705 *** Propagation of the :force option. ``I notice that
707 (oos 'compile-op :araneida :force t)
709 also forces compilation of every other system the :araneida system
710 depends on. This is rarely useful to me; usually, when I want to force
711 recompilation of something more than a single source file, I want to
712 recompile only one system. So it would be more useful to have
713 make-sub-operation refuse to propagate ":force t" to other systems, and
714 propagate only something like ":force :recursively". ''
716 Ideally what we actually want is some kind of criterion that says
717 to which systems (and which operations) a :force switch will propagate.
719 The problem is perhaps that 'force' is a pretty meaningless concept.
720 How obvious is it that "load :force t" should force _compilation_?
721 But we don't really have the right dependency setup for the user to
722 compile :force t and expect it to work (files will not be loaded after
723 compilation, so the compile environment for subsequent files will be
724 emptier than it needs to be)
726 What does the user actually want to do when he forces? Usually, for
727 me, update for use with a new version of the lisp compiler. Perhaps
728 for recovery when he suspects that something has gone wrong. Or else
729 when he's changed compilation options or configuration in some way
730 that's not reflected in the dependency graph.
732 Other possible interface: have a 'revert' function akin to 'make clean'
734 (asdf:revert 'asdf:compile-op 'araneida)
736 would delete any files produced by 'compile-op 'araneida. Of course, it
737 wouldn't be able to do much about stuff in the image itself.
743 There's a difference between a module's dependencies (peers) and its
744 components (children). Perhaps there's a similar difference in
745 operations? For example, (load "use") depends-on (load "macros") is a
746 peer, whereas (load "use") depends-on (compile "use") is more of a
747 `subservient' relationship.