The Compiler
************

This chapter will discuss most compiler issues other than efficiency, including compiler error messages, the SBCL compiler's unusual approach to type safety in the presence of type declarations, the effects of various compiler optimization policies, and the way that inlining and open coding may cause optimized code to differ from a naive translation. Efficiency issues are sufficiently varied and separate that they have their own chapter, Efficiency.

Error Messages
==============

The compiler supplies a large amount of source location information in error messages. The error messages contain a lot of detail in a terse format, so they may be confusing at first. Error messages will be illustrated using this example program:

     (defmacro zoq (x)
       `(roq (ploq (+ ,x 3))))

     (defun foo (y)
       (declare (symbol y))
       (zoq y))

The main problem with this program is that it is trying to add 3 to a symbol. Note also that the functions roq and ploq aren't defined anywhere.

The Parts of the Error Message
------------------------------

When processing this program, the compiler will produce this warning:

     file: /tmp/foo.lisp

     in: DEFUN FOO
     (ZOQ Y)
     --> ROQ PLOQ +
     ==>
     Y
     caught WARNING:
     Result is a SYMBOL, not a NUMBER.

In this example we see each of the six possible parts of a compiler error message:

 1. File: /tmp/foo.lisp This is the name of the file that the compiler read the relevant code from.  The file name is displayed because it may not be immediately obvious when there is an error during compilation of a large system, especially when with-compilation-unit is used to delay undefined warnings.

 2. in: DEFUN FOO This is the definition top level form responsible for the error. It is obtained by taking the first two elements of the enclosing form whose first element is a symbol beginning with ``def''.  If there is no such enclosing ``def'' form, then the outermost form is used.  If there are multiple def forms, then they are all printed from the outside in, separated by =>'s.  In this example, the problem was in the defun for foo.

 3. (ZOQ Y) This is the original source form responsible for the error.  Original source means that the form directly appeared in the original input to the compiler, i.e. in the lambda passed to compile or in the top level form read from the source file. In this example, the expansion of the zoq macro was responsible for the error.

 4. --> ROQ PLOQ + This is the processing path that the compiler used to produce the errorful code.  The processing path is a representation of the evaluated forms enclosing the actual source that the compiler encountered when processing the original source.  The path is the first element of each form, or the form itself if the form is not a list.  These forms result from the expansion of macros or source-to-source transformation done by the compiler.  In this example, the enclosing evaluated forms are the calls to roq, ploq and +.  These calls resulted from the expansion of the zoq macro.

 5. ==> Y This is the actual source responsible for the error. If the actual source appears in the explanation, then we print the next enclosing evaluated form, instead of printing the actual source twice.  (This is the form that would otherwise have been the last form of the processing path.) In this example, the problem is with the evaluation of the reference to the variable y.

 6. caught WARNING: Result is a SYMBOL, not a NUMBER.  This is the explanation of the problem. In this example, the problem is that y evaluates to a symbol, but is in a context where a number is required (the argument to +).

Note that each part of the error message is distinctively marked:

  * file: and in: mark the file and definition, respectively.

  * The original source is an indented form with no prefix.

  * Each line of the processing path is prefixed with -->

  * The actual source form is indented like the original source, but is marked by a preceding ==> line.

  * The explanation is prefixed with the error severity, which can be caught ERROR:, caught WARNING:, caught STYLE-WARNING:, or note:.

Each part of the error message is more specific than the preceding one.  If consecutive error messages are for nearby locations, then the front part of the error messages would be the same.  In this case, the compiler omits as much of the second message as in common with the first.  For example:

     file: /tmp/foo.lisp

     in: DEFUN FOO
     (ZOQ Y)
     --> ROQ
     ==>
     (PLOQ (+ Y 3))
     caught STYLE-WARNING:
     undefined function: PLOQ

     ==>
     (ROQ (PLOQ (+ Y 3)))
     caught STYLE-WARNING:
     undefined function: ROQ

In this example, the file, definition and original source are identical for the two messages, so the compiler omits them in the second message.  If consecutive messages are entirely identical, then the compiler prints only the first message, followed by: [Last message occurs REPEATS times] where REPEATS is the number of times the message was given.

If the source was not from a file, then no file line is printed.  If the actual source is the same as the original source, then the processing path and actual source will be omitted. If no forms intervene between the original source and the actual source, then the processing path will also be omitted.

The Original and Actual Source
------------------------------

The original source displayed will almost always be a list. If the actual source for an error message is a symbol, the original source will be the immediately enclosing evaluated list form.  So even if the offending symbol does appear in the original source, the compiler will print the enclosing list and then print the symbol as the actual source (as though the symbol were introduced by a macro.)

When the actual source is displayed (and is not a symbol), it will always be code that resulted from the expansion of a macro or a source-to-source compiler optimization.  This is code that did not appear in the original source program; it was introduced by the compiler.

Keep in mind that when the compiler displays a source form in an error message, it always displays the most specific (innermost) responsible form.  For example, compiling this function

     (defun bar (x)
       (let (a)
         (declare (fixnum a))
         (setq a (foo x))
         a))

gives this error message

     in: DEFUN BAR
     (LET (A) (DECLARE (FIXNUM A)) (SETQ A (FOO X)) A)
     caught WARNING: The binding of A is not a FIXNUM:
     NIL

This error message is not saying ``there is a problem somewhere in this let'' -- it is saying that there is a problem with the let itself. In this example, the problem is that a's nil initial value is not a fixnum.

The Processing Path
-------------------

The processing path is mainly useful for debugging macros, so if you don't write macros, you can probably ignore it. Consider this example:

     (defun foo (n)
       (dotimes (i n *undefined*)))

Compiling results in this error message:

     in: DEFUN FOO
     (DOTIMES (I N *UNDEFINED*))
     --> DO BLOCK LET TAGBODY RETURN-FROM
     ==>
     (PROGN *UNDEFINED*)
     caught STYLE-WARNING:
     undefined variable: *UNDEFINED*

Note that do appears in the processing path. This is because dotimes expands into:

     (do ((i 0 (1+ i)) (#:g1 n))
         ((>= i #:g1) *undefined*)
       (declare (type unsigned-byte i)))

The rest of the processing path results from the expansion of do:

     (block nil
       (let ((i 0) (#:g1 n))
         (declare (type unsigned-byte i))
         (tagbody (go #:g3)
           #:g2    (psetq i (1+ i))
           #:g3    (unless (>= i #:g1) (go #:g2))
           (return-from nil (progn *undefined*)))))

In this example, the compiler descended into the block, let, tagbody and return-from to reach the progn printed as the actual source. This is a place where the ``actual source appears in explanation'' rule was applied. The innermost actual source form was the symbol *undefined* itself, but that also appeared in the explanation, so the compiler backed out one level.

Error Severity
--------------

There are four levels of compiler error severity: error, warning, style warning, and note. The first three levels correspond to condition classes which are defined in the ANSI standard for Common Lisp and which have special significance to the compile and compile-file functions. These levels of compiler error severity occur when the compiler handles conditions of these classes. The fourth level of compiler error severity, note, is used for problems which are too mild for the standard condition classes, typically hints about how efficiency might be improved.

Errors During Macroexpansion
----------------------------

The compiler handles errors that happen during macroexpansion, turning them into compiler errors. If you want to debug the error (to debug a macro), you can set *break-on-signals* to error. For example, this definition:

     (defun foo (e l)
       (do ((current l (cdr current))
            ((atom current) nil))
           (when (eq (car current) e) (return current))))

gives this error:

     in: DEFUN FOO
     (DO ((CURRENT L #) (# NIL)) (WHEN (EQ # E) (RETURN CURRENT)) )
     caught ERROR:
     (in macroexpansion of (DO # #))
     (hint: For more precise location, try *BREAK-ON-SIGNALS*.)
     DO step variable is not a symbol: (ATOM CURRENT)

Read Errors
-----------

SBCL's compiler (unlike CMUCL's) does not attempt to recover from read errors when reading a source file, but instead just reports the offending character position and gives up on the entire source file.

The Compiler's Handling of Types
================================

The most unusual features of the SBCL compiler (which is very similar to the original CMUCL compiler, also known as Python) is its unusually sophisticated understanding of the Common Lisp type system and its unusually conservative approach to the implementation of type declarations. These two features reward the use of type declarations throughout development, even when high performance is not a concern. Also, as discussed in the chapter on performance (see Efficiency), the use of appropriate type declarations can be very important for performance as well.

The SBCL compiler, like the related compiler in CMUCL, treats type declarations much differently than other Lisp compilers.  By default (i.e., at ordinary levels of the safety compiler optimization parameter), the compiler doesn't blindly believe most type declarations; it considers them assertions about the program that should be checked.

The SBCL compiler also has a greater knowledge of the Common Lisp type system than other compilers.  Support is incomplete only for the not, and and satisfies types.

Type Errors at Compile Time
---------------------------

If the compiler can prove at compile time that some portion of the program cannot be executed without a type error, then it will give a warning at compile time. It is possible that the offending code would never actually be executed at run-time due to some higher level consistency constraint unknown to the compiler, so a type warning doesn't always indicate an incorrect program. For example, consider this code fragment:

     (defun raz (foo)
       (let ((x (case foo
                   (:this 13)
                   (:that 9)
                   (:the-other 42))))
         (declare (fixnum x))
         (foo x)))

Compilation produces this warning:

     in: DEFUN RAZ
     (CASE FOO (:THIS 13) (:THAT 9) (:THE-OTHER 42))
     --> LET COND IF COND IF COND IF
     ==>
     (COND)
     caught WARNING: This is not a FIXNUM:
     NIL

In this case, the warning means that if foo isn't any of :this, :that or :the-other, then x will be initialized to nil, which the fixnum declaration makes illegal. The warning will go away if ecase is used instead of case, or if :the-other is changed to t.

This sort of spurious type warning happens moderately often in the expansion of complex macros and in inline functions. In such cases, there may be dead code that is impossible to correctly execute.  The compiler can't always prove this code is dead (could never be executed), so it compiles the erroneous code (which will always signal an error if it is executed) and gives a warning.

Type warnings are inhibited when the sb-ext:inhibit-warnings optimization quality is 3. See Compiler Policy.  This can be used in a local declaration to inhibit type warnings in a code fragment that has spurious warnings.

Precise Type Checking
---------------------

With the default compilation policy, all type declarations are precisely checked, except in a few situations where they are simply ignored instead. Precise checking means that the check is done as though typep had been called with the exact type specifier that appeared in the declaration. In SBCL, adding type declarations makes code safer.  (Except that remaining bugs in the compiler's handling of types unfortunately provide some exceptions to this rule, see Implementation Limitations).

If a variable is declared to be (integer 3 17) then its value must always be an integer between 3 and 17. If multiple type declarations apply to a single variable, then all the declarations must be correct; it is as though all the types were intersected producing a single and type specifier.

Argument and result type declarations are automatically enforced. If you declare the type of a function argument, a type check will be done when that function is called. In a function call, the called function does the argument type checking.

The types of structure slots are also checked. The value of a structure slot must always be of the type indicated in any :type slot option.

In traditional Common Lisp compilers, not all type assertions are checked, and type checks are not precise. Traditional compilers blindly trust explicit type declarations, but may check the argument type assertions for built-in functions. Type checking is not precise, since the argument type checks will be for the most general type legal for that argument. In many systems, type declarations suppress what little type checking is being done, so adding type declarations makes code unsafe. This is a problem since it discourages writing type declarations during initial coding. In addition to being more error prone, adding type declarations during tuning also loses all the benefits of debugging with checked type assertions.

To gain maximum benefit from the compiler's type checking, you should always declare the types of function arguments and structure slots as precisely as possible. This often involves the use of or, member, and other list-style type specifiers.

Weakened Type Checking
----------------------

At one time, CMUCL supported another level of type checking, ``weakened type checking'', when the value for the speed optimization quality is greater than safety, and safety is not 0.  The CMUCL manual still has a description of it, but even the CMU CL code no longer corresponds to the manual. Some of this partial safety checking lingers on in SBCL, but it's not a supported feature, and should not be relied on. If you ask the compiler to optimize speed to a higher level than safety, your program is performing without a safety net, because SBCL may at its option believe any or all type declarations with either partial or nonexistent runtime checking.

Getting Existing Programs to Run
--------------------------------

Since SBCL's compiler, like CMUCL's compiler, does much more comprehensive type checking than most Lisp compilers, SBCL may detect type errors in programs that have been debugged using other compilers. These errors are mostly incorrect declarations, although compile-time type errors can find actual bugs if parts of the program have never been tested.

Some incorrect declarations can only be detected by run-time type checking. It is very important to initially compile a program with full type checks (high safety optimization) and then test this safe version. After the checking version has been tested, then you can consider weakening or eliminating type checks.  This applies even to previously debugged programs, because the SBCL compiler does much more type inference than other Common Lisp compilers, so an incorrect declaration can do more damage.

The most common problem is with variables whose constant initial value doesn't match the type declaration. Incorrect constant initial values will always be flagged by a compile-time type error, and they are simple to fix once located. Consider this code fragment:

     (prog (foo)
       (declare (fixnum foo))
       (setq foo ...)
       ...)

Here foo is given an initial value of nil, but is declared to be a fixnum.  Even if it is never read, the initial value of a variable must match the declared type.  There are two ways to fix this problem. Change the declaration

     (prog (foo)
       (declare (type (or fixnum null) foo))
       (setq foo ...)
       ...)

or change the initial value

     (prog ((foo 0))
       (declare (fixnum foo))
       (setq foo ...)
       ...)

It is generally preferable to change to a legal initial value rather than to weaken the declaration, but sometimes it is simpler to weaken the declaration than to try to make an initial value of the appropriate type.

Another declaration problem occasionally encountered is incorrect declarations on defmacro arguments. This can happen when a function is converted into a macro. Consider this macro:

     (defmacro my-1+ (x)
       (declare (fixnum x))
       `(the fixnum (1+ ,x)))

Although legal and well-defined Common Lisp code, this meaning of this definition is almost certainly not what the writer intended. For example, this call is illegal:

     (my-1+ (+ 4 5))

This call is illegal because the argument to the macro is (+ 4 5), which is a list, not a fixnum.  Because of macro semantics, it is hardly ever useful to declare the types of macro arguments.  If you really want to assert something about the type of the result of evaluating a macro argument, then put a the in the expansion:

     (defmacro my-1+ (x)
       `(the fixnum (1+ (the fixnum ,x))))

In this case, it would be stylistically preferable to change this macro back to a function and declare it inline.

Some more subtle problems are caused by incorrect declarations that can't be detected at compile time.  Consider this code:

     (do ((pos 0 (position #\a string :start (1+ pos))))
       ((null pos))
       (declare (fixnum pos))
       ...)

Although pos is almost always a fixnum, it is nil at the end of the loop. If this example is compiled with full type checks (the default), then running it will signal a type error at the end of the loop. If compiled without type checks, the program will go into an infinite loop (or perhaps position will complain because (1+ nil) isn't a sensible start.) Why? Because if you compile without type checks, the compiler just quietly believes the type declaration. Since the compiler believes that pos is always a fixnum, it believes that pos is never nil, so (null pos) is never true, and the loop exit test is optimized away. Such errors are sometimes flagged by unreachable code notes, but it is still important to initially compile and test any system with full type checks, even if the system works fine when compiled using other compilers.

In this case, the fix is to weaken the type declaration to (or fixnum null)  (Actually, this declaration is unnecessary in SBCL, since it already knows that position returns a non-negative fixnum or nil.).

Note that there is usually little performance penalty for weakening a declaration in this way. Any numeric operations in the body can still assume that the variable is a fixnum, since nil is not a legal numeric argument. Another possible fix would be to say:

     (do ((pos 0 (position #\a string :start (1+ pos))))
         ((null pos))
       (let ((pos pos))
         (declare (fixnum pos))
         ...))

This would be preferable in some circumstances, since it would allow a non-standard representation to be used for the local pos variable in the loop body.

Implementation Limitations
--------------------------

Ideally, the compiler would consider all type declarations to be assertions, so that adding type declarations to a program, no matter how incorrect they might be, would never cause undefined behavior. As of SBCL version 0.8.1, the compiler is known to fall short of this goal in two areas:

  * Proclaimed constraints on argument and result types of a function are supposed to be checked by the function. If the function type is proclaimed before function definition, type checks are inserted by the compiler, but the standard allows the reversed order, in which case the compiler will trust the declaration.

  * The compiler cannot check types of an unknown number of values; if the number of generated values is unknown, but the number of consumed is known, only consumed values are checked.

  * There are a few poorly characterized but apparently very uncommon situations where a type declaration in an unexpected location will be trusted and never checked by the compiler.

These are important bugs, but are not necessarily easy to fix, so they may, alas, remain in the system for a while.

Compiler Policy
===============

As of version 0.6.4, SBCL still uses most of the CMUCL code for compiler policy. The CMUCL code has many features and high-quality documentation, but the two unfortunately do not match. So this area of the compiler and its interface needs to be cleaned up. Meanwhile, here is some rudimentary documentation on the current behavior of the system.

Compiler policy is controlled by the optimize declaration. The compiler supports the ANSI optimization qualities, and also an extension sb-ext:inhibit-warnings.

Ordinarily, when the speed quality is high, the compiler emits notes to notify the programmer about its inability to apply various optimizations. Setting sb-ext:inhibit-warnings to a value at least as large as the speed quality inhibits this notification. This can be useful to suppress notes about code which is known to be unavoidably inefficient. (For example, the compiler issues notes about having to use generic arithmetic instead of fixnum arithmetic, which is not helpful for code which by design supports arbitrary-sized integers instead of being limited to fixnums.)

     Note: The basic functionality of the optimize inhibit-warnings extension will probably be supported in all future versions of the system, but it will probably be renamed when the compiler and its interface are cleaned up. The current name is misleading, because it mostly inhibits optimization notes, not warnings. And making it an optimization quality is misleading, because it shouldn't affect the resulting code at all. It may become a declaration identifier with a name like sb-ext:inhibit-notes, so that what's currently written.

          (declaim (optimize (sb-ext:inhibit-warnings 2)))

     would become something like

          (declaim (sb-ext:inhibit-notes 2))

In early versions of SBCL, a speed value of zero was used to enable byte compilation, but since version 0.7.0, SBCL only supports native compilation.

When safety is zero, almost all runtime checking of types, array bounds, and so forth is suppressed.

When safety is less than speed, any and all type checks may be suppressed. At some point in the past, CMUCL had a more nuanced interpretation of this (see Weakened Type Checking). However, SBCL doesn't support that interpretation, and setting safety less than speed may have roughly the same effect as setting safety to zero.

The value of space mostly influences the compiler's decision whether to inline operations, which tend to increase the size of programs. Use the value 0 with caution, since it can cause the compiler to inline operations so indiscriminately that the net effect is to slow the program by causing cache misses or swapping.

Open Coding and Inline Expansion
================================

Since Common Lisp forbids the redefinition of standard functions, the compiler can have special knowledge of these standard functions embedded in it. This special knowledge is used in various ways (open coding, inline expansion, source transformation), but the implications to the user are basically the same:

  * Attempts to redefine standard functions may be frustrated, since the function may never be called. Although it is technically illegal to redefine standard functions, users sometimes want to implicitly redefine these functions when they are debugging using the trace macro.  Special-casing of standard functions can be inhibited using the notinline declaration.

  * The compiler can have multiple alternate implementations of standard functions that implement different trade-offs of speed, space and safety.  This selection is based on the compiler policy, Compiler Policy.

When a function call is open coded, inline code whose effect is equivalent to the function call is substituted for that function call. When a function call is closed coded, it is usually left as is, although it might be turned into a call to a different function with different arguments. As an example, if nthcdr were to be open coded, then

     (nthcdr 4 foobar)

might turn into

     (cdr (cdr (cdr (cdr foobar))))

or even

     (do ((i 0 (1+ i))
       (list foobar (cdr foobar)))
       ((= i 4) list))

If nth is closed coded, then

     (nth x l)

might stay the same, or turn into something like

     (car (nthcdr x l))

In general, open coding sacrifices space for speed, but some functions (such as car) are so simple that they are always open-coded. Even when not open-coded, a call to a standard function may be transformed into a different function call (as in the last example) or compiled as static call. Static function call uses a more efficient calling convention that forbids redefinition.