The Debugger
************

The SBCL debugger (as the CMUCL debugger it was derived from) has very good support for source-level debugging of compiled code.  Although some other debuggers allow access of variables by name, this seems to be the first Lisp debugger that:

  * Tells you when a variable doesn't have a value because it hasn't been initialized yet or has already been deallocated, or

  * Can display the precise source location corresponding to a code location in the debugged program.

These features allow the debugging of compiled code to be made almost indistinguishable from interpreted code debugging.

Starting the Debugger
=====================

The debugger is an interactive command loop that allows a user to examine the function call stack.  The debugger is invoked when:

  * A serious-condition is signaled, and it is not handled, or

  * error is called, and the condition it signals is not handled, or

  * the debugger is explicitly entered with the Lisp break or debug functions.

When you enter the TTY debugger, it looks something like this:

     debugger invoked on a TYPE-ERROR in thread 11184:
       The value 3 is not of type LIST.
     restarts (invokable by number or by possibly-abbreviated name):
       0: [ABORT   ] Reduce debugger level (leaving debugger, returning to toplevel).
       1: [TOPLEVEL] Restart at toplevel READ/EVAL/PRINT loop.
     (CAR 1 3)[:EXTERNAL]
     0]

The first group of lines describe what the error was that put us in the debugger.  In this case car was called on 3.  After restarts is a list of all the ways that we can restart execution after this error.  In this case, both options return to top-level.  After printing its banner, the debugger prints the current frame and the debugger prompt.

The Debugger Command Loop
=========================

The debugger is an interactive read-eval-print loop much like the normal top-level, but some symbols are interpreted as debugger commands instead of being evaluated.  A debugger command starts with the symbol name of the command, possibly followed by some arguments on the same line.  Some commands prompt for additional input.  Debugger commands can be abbreviated by any unambiguous prefix: help can be typed as h, he, etc.  For convenience, some commands have ambiguous one-letter abbreviations: f for frame.

The package is not significant in debugger commands; any symbol with the name of a debugger command will work.  If you want to show the value of a variable that happens also to be the name of a debugger command, you can use the list-locals command or the sb-debug:var function, or you can wrap the variable in a progn to hide it from the command loop.

The debugger prompt is ``FRAME]'', where FRAME is the number of the current frame.  Frames are numbered starting from zero at the top (most recent call), increasing down to the bottom. The current frame is the frame that commands refer to.  The current frame also provides the lexical environment for evaluation of non-command forms.

The debugger evaluates forms in the lexical environment of the functions being debugged.  The debugger can only access variables. You can't go or return-from into a function, and you can't call local functions.  Special variable references are evaluated with their current value (the innermost binding around the debugger invocation) -- you don't get the value that the special had in the current frame.  For more information on debugger variable access, see Variable Access.

Controlling Printing in the Debugger
====================================

In the debugger, it is possible to override the printing behaviour of the REPL.

 -- Variable: *debug-print-variable-alist*

     An association list describing new bindings for special variables (typically *PRINT-FOO* variables) to be used within the debugger, e.g.

          ((*PRINT-LENGTH* . 10) (*PRINT-LEVEL* . 6) (*PRINT-PRETTY* . NIL))

     The variables in the car position are bound to the values in the cdr during the execution of some debug commands.  When evaluating arbitrary expressions in the debugger, the normal values of the printer control variables are in effect. *debug-print-variable-alist* does not contain any bindings initially.

Stack Frames
============

A stack frame is the run-time representation of a call to a function; the frame stores the state that a function needs to remember what it is doing.  Frames have:

  * Variables (see Variable Access), which are the values being operated on, and

  * Arguments to the call (which are really just particularly interesting variables), and

  * A current location (see Source Location Printing), which is the place in the program where the function was running when it stopped to call another function, or because of an interrupt or error.

Stack Motion
------------

These commands move to a new stack frame and print the name of the function and the values of its arguments in the style of a Lisp function call:

 -- Debugger Command: up
     Move up to the next higher frame.  More recent function calls are considered to be higher on the stack.

 -- Debugger Command: down
     Move down to the next lower frame.

 -- Debugger Command: top
     Move to the highest frame, that is, the frame where the debugger was entered.

 -- Debugger Command: bottom
     Move to the lowest frame.

 -- Debugger Command: frame [N]
     Move to the frame with the specified number.  Prompts for the number if not supplied.  The frame with number 0 is the frame where the debugger was entered.

How Arguments are Printed
-------------------------

A frame is printed to look like a function call, but with the actual argument values in the argument positions.  So the frame for this call in the source:

     (myfun (+ 3 4) 'a)

would look like this:

     (MYFUN 7 A)

All keyword and optional arguments are displayed with their actual values; if the corresponding argument was not supplied, the value will be the default.  So this call:

     (subseq "foo" 1)

would look like this:

     (SUBSEQ "foo" 1 3)

And this call:

     (string-upcase "test case")

would look like this:

     (STRING-UPCASE "test case" :START 0 :END NIL)

The arguments to a function call are displayed by accessing the argument variables.  Although those variables are initialized to the actual argument values, they can be set inside the function; in this case the new value will be displayed.

&rest arguments are handled somewhat differently.  The value of the rest argument variable is displayed as the spread-out arguments to the call, so:

     (format t "~A is a ~A." "This" 'test)

would look like this:

     (FORMAT T "~A is a ~A." "This" 'TEST)

Rest arguments cause an exception to the normal display of keyword arguments in functions that have both &rest and &key arguments.  In this case, the keyword argument variables are not displayed at all; the rest arg is displayed instead.  So for these functions, only the keywords actually supplied will be shown, and the values displayed will be the argument values, not values of the (possibly modified) variables.

If the variable for an argument is never referenced by the function, it will be deleted.  The variable value is then unavailable, so the debugger prints #<unused-arg> instead of the value.  Similarly, if for any of a number of reasons the value of the variable is unavailable or not known to be available (see Variable Access), then #<unavailable-arg> will be printed instead of the argument value.

Printing of argument values is controlled by *debug-print-variable-alist*.  See Controlling Printing in the Debugger.

Function Names
--------------

If a function is defined by defun, labels, or flet, then the debugger will print the actual function name after the open parenthesis, like:

     (STRING-UPCASE "test case" :START 0 :END NIL)
     ((SETF AREF) #\a "for" 1)

Otherwise, the function name is a string, and will be printed in quotes:

     ("DEFUN MYFUN" BAR)
     ("DEFMACRO DO" (DO ((I 0 (1+ I))) ((= I 13))) NIL)
     ("SETQ *GC-NOTIFY-BEFORE*")

This string name is derived from the defMUMBLE form that encloses or expanded into the lambda, or the outermost enclosing form if there is no defMUMBLE.

Funny Frames
------------

Sometimes the evaluator introduces new functions that are used to implement a user function, but are not directly specified in the source.  The main place this is done is for checking argument type and syntax.  Usually these functions do their thing and then go away, and thus are not seen on the stack in the debugger.  But when you get some sort of error during lambda-list processing, you end up in the debugger on one of these funny frames.

These funny frames are flagged by printing ``[KEYWORD]'' after the parentheses.  For example, this call:

     (car 'a 'b)

will look like this:

     (CAR 2 A)[:EXTERNAL]

And this call:

     (string-upcase "test case" :end)

would look like this:

     ("SB!INT:&MORE processing" "test case" 1053984 1)[:OPTIONAL]

As you can see, these frames have only a vague resemblance to the original call.  Fortunately, the error message displayed when you enter the debugger will usually tell you what problem is (in these cases, too many arguments and odd keyword arguments.)  Also, if you go down the stack to the frame for the calling function, you can display the original source.  See Source Location Printing.

With recursive or block compiled functions, an :EXTERNAL frame may appear before the frame representing the first call to the recursive function or entry to the compiled block. This is a consequence of the way the compiler does block compilation: there is nothing odd with your program. You will also see :CLEANUP frames during the execution of unwind-protect cleanup code. Note that inline expansion and open-coding affect what frames are present in the debugger, see Debugger Policy Control.

Debug Tail Recursion
--------------------

Both the compiler and the interpreter are ``properly tail recursive.'' If a function call is in a tail-recursive position, the stack frame will be deallocated at the time of the call, rather than after the call returns.  Consider this backtrace:

     (BAR ...)
     (FOO ...)

Because of tail recursion, it is not necessarily the case that FOO directly called BAR.  It may be that FOO called some other function FOO2 which then called BAR tail-recursively, as in this example:

     (defun foo ()
       ...
       (foo2 ...)
       ...)

     (defun foo2 (...)
       ...
       (bar ...))

     (defun bar (...)
       ...)

Usually the elimination of tail-recursive frames makes debugging more pleasant, since theses frames are mostly uninformative.  If there is any doubt about how one function called another, it can usually be eliminated by finding the source location in the calling frame. See Source Location Printing.

The elimination of tail-recursive frames can be prevented by disabling tail-recursion optimization, which happens when the debug optimization quality is greater than 2. See Debugger Policy Control.

Unknown Locations and Interrupts
--------------------------------

The debugger operates using special debugging information attached to the compiled code.  This debug information tells the debugger what it needs to know about the locations in the code where the debugger can be invoked.  If the debugger somehow encounters a location not described in the debug information, then it is said to be unknown.  If the code location for a frame is unknown, then some variables may be inaccessible, and the source location cannot be precisely displayed.

There are three reasons why a code location could be unknown:

  * There is inadequate debug information due to the value of the debug optimization quality.  See Debugger Policy Control.

  * The debugger was entered because of an interrupt such as C-c.

  * A hardware error such as ``bus error'' occurred in code that was compiled unsafely due to the value of the safety optimization quality.

In the last two cases, the values of argument variables are accessible, but may be incorrect.  For more details on when variable values are accessible, Variable Value Availability.

It is possible for an interrupt to happen when a function call or return is in progress.  The debugger may then flame out with some obscure error or insist that the bottom of the stack has been reached, when the real problem is that the current stack frame can't be located.  If this happens, return from the interrupt and try again.

Variable Access
===============

There are two ways to access the current frame's local variables in the debugger: list-locals and sb-debug:var.

The debugger doesn't really understand lexical scoping; it has just one namespace for all the variables in the current stack frame.  If a symbol is the name of multiple variables in the same function, then the reference appears ambiguous, even though lexical scoping specifies which value is visible at any given source location.  If the scopes of the two variables are not nested, then the debugger can resolve the ambiguity by observing that only one variable is accessible.

When there are ambiguous variables, the evaluator assigns each one a small integer identifier.  The sb-debug:var function uses this identifier to distinguish between ambiguous variables.  The list-locals command prints the identifier.  In the following example, there are two variables named X.  The first one has identifier 0 (which is not printed), the second one has identifier 1.

     X  =  1
     X#1  =  2

 -- Debugger Command: list-locals [PREFIX]
     This command prints the name and value of all variables in the current frame whose name has the specified PREFIX.  PREFIX may be a string or a symbol.  If no PREFIX is given, then all available variables are printed.  If a variable has a potentially ambiguous name, then the name is printed with a ``#IDENTIFIER'' suffix, where IDENTIFIER is the small integer used to make the name unique.

 -- Function: sb-debug:var NAME &optional IDENTIFIER
     This function returns the value of the variable in the current frame with the specified NAME.  If supplied, IDENTIFIER determines which value to return when there are ambiguous variables.

     When NAME is a symbol, it is interpreted as the symbol name of the variable, i.e. the package is significant.  If NAME is an uninterned symbol (gensym), then return the value of the uninterned variable with the same name.  If NAME is a string, sb-debug:var interprets it as the prefix of a variable name that must unambiguously complete to the name of a valid variable.

     IDENTIFIER is used to disambiguate the variable name; use list-locals to find out the identifiers.

Variable Value Availability
---------------------------

The value of a variable may be unavailable to the debugger in portions of the program where Lisp says that the variable is defined.  If a variable value is not available, the debugger will not let you read or write that variable.  With one exception, the debugger will never display an incorrect value for a variable.  Rather than displaying incorrect values, the debugger tells you the value is unavailable.

The one exception is this: if you interrupt (e.g., with C-c) or if there is an unexpected hardware error such as ``bus error'' (which should only happen in unsafe code), then the values displayed for arguments to the interrupted frame might be incorrect. (Since the location of an interrupt or hardware error will always be an unknown location, non-argument variable values will never be available in the interrupted frame.  See Unknown Locations and Interrupts.)  This exception applies only to the interrupted frame: any frame farther down the stack will be fine.

The value of a variable may be unavailable for these reasons:

  * The value of the debug optimization quality may have omitted debug information needed to determine whether the variable is available. Unless a variable is an argument, its value will only be available when debug is at least 2.

  * The compiler did lifetime analysis and determined that the value was no longer needed, even though its scope had not been exited.  Lifetime analysis is inhibited when the debug optimization quality is 3.

  * The variable's name is an uninterned symbol (gensym).  To save space, the compiler only dumps debug information about uninterned variables when the debug optimization quality is 3.

  * The frame's location is unknown (see Unknown Locations and Interrupts) because the debugger was entered due to an interrupt or unexpected hardware error.  Under these conditions the values of arguments will be available, but might be incorrect.  This is the exception mentioned above.

  * The variable (or the code referencing it) was optimized out of existence.  Variables with no reads are always optimized away.  The degree to which the compiler deletes variables will depend on the value of the compilation-speed optimization quality, but most source-level optimizations are done under all compilation policies.

  * The variable is never set and its definition looks like

          (LET ((var1 var2))
             ...)

     In this case, var1 is substituted with var2.

  * The variable is never set and is referenced exactly once.  In this case, the reference is substituted with the variable initial value.

Since it is especially useful to be able to get the arguments to a function, argument variables are treated specially when the speed optimization quality is less than 3 and the debug quality is at least 1.  With this compilation policy, the values of argument variables are almost always available everywhere in the function, even at unknown locations.  For non-argument variables, debug must be at least 2 for values to be available, and even then, values are only available at known locations.

Note On Lexical Variable Access
-------------------------------

When the debugger command loop establishes variable bindings for available variables, these variable bindings have lexical scope and dynamic extent. (The variable bindings are actually created using the Lisp symbol-macrolet special form.)  You can close over them, but such closures can't be used as upward funargs.

You can also set local variables using setq, but if the variable was closed over in the original source and never set, then setting the variable in the debugger may not change the value in all the functions the variable is defined in.  Another risk of setting variables is that you may assign a value of a type that the compiler proved the variable could never take on.  This may result in bad things happening.

Source Location Printing
========================

One of the debugger's capabilities is source level debugging of compiled code.  These commands display the source location for the current frame:

 -- Debugger Command: source [CONTEXT]
     This command displays the file that the current frame's function was defined from (if it was defined from a file), and then the source form responsible for generating the code that the current frame was executing.  If CONTEXT is specified, then it is an integer specifying the number of enclosing levels of list structure to print.

The source form for a location in the code is the innermost list present in the original source that encloses the form responsible for generating that code.  If the actual source form is not a list, then some enclosing list will be printed.  For example, if the source form was a reference to the variable *some-random-special*, then the innermost enclosing evaluated form will be printed.  Here are some possible enclosing forms:

     (let ((a *some-random-special*))
       ...)

     (+ *some-random-special* ...)

If the code at a location was generated from the expansion of a macro or a source-level compiler optimization, then the form in the original source that expanded into that code will be printed.  Suppose the file /usr/me/mystuff.lisp looked like this:

     (defmacro mymac ()
       '(myfun))

     (defun foo ()
       (mymac)
       ...)

If foo has called myfun, and is waiting for it to return, then the source command would print:

     ; File: /usr/me/mystuff.lisp

     (MYMAC)

Note that the macro use was printed, not the actual function call form, (myfun).

If enclosing source is printed by giving an argument to source or vsource, then the actual source form is marked by wrapping it in a list whose first element is #:***HERE***.  In the previous example, source 1 would print:

     ; File: /usr/me/mystuff.lisp

     (DEFUN FOO ()
       (#:***HERE***
        (MYMAC))
       ...)

How the Source is Found
-----------------------

If the code was defined from Lisp by compile or eval, then the source can always be reliably located.  If the code was defined from a fasl file created by compile-file, then the debugger gets the source forms it prints by reading them from the original source file.  This is a potential problem, since the source file might have moved or changed since the time it was compiled.

The source file is opened using the truename of the source file pathname originally given to the compiler.  This is an absolute pathname with all logical names and symbolic links expanded.  If the file can't be located using this name, then the debugger gives up and signals an error.

If the source file can be found, but has been modified since the time it was compiled, the debugger prints this warning:

     ; File has been modified since compilation:
     ;   FILENAME
     ; Using form offset instead of character position.

where FILENAME is the name of the source file.  It then proceeds using a robust but not foolproof heuristic for locating the source. This heuristic works if:

  * No top-level forms before the top-level form containing the source have been added or deleted, and

  * The top-level form containing the source has not been modified much. (More precisely, none of the list forms beginning before the source form have been added or deleted.)

If the heuristic doesn't work, the displayed source will be wrong, but will probably be near the actual source.  If the ``shape'' of the top-level form in the source file is too different from the original form, then an error will be signaled.  When the heuristic is used, the the source location commands are noticeably slowed.

Source location printing can also be confused if (after the source was compiled) a read-macro you used in the code was redefined to expand into something different, or if a read-macro ever returns the same eq list twice.  If you don't define read macros and don't use ## in perverted ways, you don't need to worry about this.

Source Location Availability
----------------------------

Source location information is only available when the debug optimization quality is at least 2.  If source location information is unavailable, the source commands will give an error message.

If source location information is available, but the source location is unknown because of an interrupt or unexpected hardware error (see Unknown Locations and Interrupts), then the command will print:

     Unknown location: using block start.

and then proceed to print the source location for the start of the basic block enclosing the code location.  It's a bit complicated to explain exactly what a basic block is, but here are some properties of the block start location:

  * The block start location may be the same as the true location.

  * The block start location will never be later in the the program's flow of control than the true location.

  * No conditional control structures (such as if, cond, or) will intervene between the block start and the true location (but note that some conditionals present in the original source could be optimized away.)  Function calls do not end basic blocks.

  * The head of a loop will be the start of a block.

  * The programming language concept of ``block structure'' and the Lisp block special form are totally unrelated to the compiler's basic block.

In other words, the true location lies between the printed location and the next conditional (but watch out because the compiler may have changed the program on you.)

Debugger Policy Control
=======================

The compilation policy specified by optimize declarations affects the behavior seen in the debugger.  The debug quality directly affects the debugger by controlling the amount of debugger information dumped.  Other optimization qualities have indirect but observable effects due to changes in the way compilation is done.

Unlike the other optimization qualities (which are compared in relative value to evaluate tradeoffs), the debug optimization quality is directly translated to a level of debug information.  This absolute interpretation allows the user to count on a particular amount of debug information being available even when the values of the other qualities are changed during compilation.  These are the levels of debug information that correspond to the values of the debug quality:

0
     Only the function name and enough information to allow the stack to be parsed.

> 0
     Any level greater than 0 gives level 0 plus all argument variables.  Values will only be accessible if the argument variable is never set and speed is not 3.  SBCL allows any real value for optimization qualities.  It may be useful to specify 0.5 to get backtrace argument display without argument documentation.

1
     Level 1 provides argument documentation (printed arglists) and derived argument/result type information.  This makes describe more informative, and allows the compiler to do compile-time argument count and type checking for any calls compiled at run-time.  This is the default.

2
     Level 1 plus all interned local variables, source location information, and lifetime information that tells the debugger when arguments are available (even when speed is 3 or the argument is set).

> 2
     Any level greater than 2 gives level 2 and in addition disables tail-call optimization, so that the backtrace will contain frames for all invoked functions, even those in tail positions.

3
     Level 2 plus all uninterned variables.  In addition, lifetime analysis is disabled (even when speed is 3), ensuring that all variable values are available at any known location within the scope of the binding.  This has a speed penalty in addition to the obvious space penalty.

> (max speed space)
     If debug is greater than both speed and space, the command return can be used to continue execution by returning a value from the current stack frame.

As you can see, if the speed quality is 3, debugger performance is degraded.  This effect comes from the elimination of argument variable special-casing (see Variable Value Availability).  Some degree of speed/debuggability tradeoff is unavoidable, but the effect is not too drastic when debug is at least 2.

In addition to inline and notinline declarations, the relative values of the speed and space qualities also change whether functions are inline expanded. If a function is inline expanded, then there will be no frame to represent the call, and the arguments will be treated like any other local variable.  Functions may also be ``semi-inline'', in which case there is a frame to represent the call, but the call is to an optimized local version of the function, not to the original function.

Exiting Commands
================

These commands get you out of the debugger.

 -- Debugger Command: toplevel
     Throw to top level.

 -- Debugger Command: restart [N]
     Invokes the Nth restart case as displayed by the error command.  If N is not specified, the available restart cases are reported.

 -- Debugger Command: continue
     Calls continue on the condition given to debug.  If there is no restart case named CONTINUE, then an error is signaled.

 -- Debugger Command: abort
     Calls abort on the condition given to debug.  This is useful for popping debug command loop levels or aborting to top level, as the case may be.

 -- Debugger Command: return VALUE
     Returns VALUE from the current stack frame.  This command is available when the debug optimization quality is greater than both speed and space.  Care must be taken that the value is of the same type as SBCL expects the stack frame to return.

Information Commands
====================

Most of these commands print information about the current frame or function, but a few show general information.

 -- Debugger Command: help
 -- Debugger Command: ?
     Displays a synopsis of debugger commands.

 -- Debugger Command: describe
     Calls describe on the current function and displays the number of local variables.

 -- Debugger Command: print
     Displays the current function call as it would be displayed by moving to this frame.

 -- Debugger Command: error
     Prints the condition given to invoke-debugger and the active proceed cases.

 -- Debugger Command: backtrace [N]
     Displays all the frames from the current to the bottom.  Only shows N frames if specified.  The printing is controlled by *debug-print-variable-alist*.

Function Tracing
================

The tracer causes selected functions to print their arguments and their results whenever they are called.  Options allow conditional printing of the trace information and conditional breakpoints on function entry or exit.

     [file macro-common-lisp-trace.texinfo not included]

     [file macro-common-lisp-untrace.texinfo not included]

     [file var-sb-debug-star-trace-indentation-step-star.texinfo not included]

     [file var-sb-debug-star-max-trace-indentation-star.texinfo not included]

     [file var-sb-debug-star-trace-encapsulate-default-star.texinfo not included]

     [file var-sb-debug-star-trace-values-star.texinfo not included]