0.8.13.70: MORE DOCUMENTATION

[sbcl.git] / doc / manual / beyond-ansi.texinfo

SBCL is mostly an implementation of the ANSI standard for
Common Lisp. However, there's some important behavior which extends
or clarifies the standard, and various behavior which outright
violates the standard.


@menu
* Non-Conformance With The ANSI Standard::  
* Idiosyncrasies::              
* Extensions::                  
@end menu

@node Non-Conformance With The ANSI Standard
@comment  node-name,  next,  previous,  up
@section Non-Conformance With The ANSI Standard

Essentially every type of non-conformance is considered a bug.  (The
exceptions involve internal inconsistencies in the standard.)  In SBCL
0.7.6, the master record of known bugs is in the @file{BUGS} file in
the distribution.  Some highlight information about bugs may also be
found in the manual page. The recommended way to report bugs is
through the sbcl-help or sbcl-devel mailing lists.  For mailing list
addresses, @xref{More SBCL Information}.


@node Idiosyncrasies
@comment  node-name,  next,  previous,  up
@section Idiosyncrasies

The information in this section describes some of the ways that SBCL
deals with choices that the ANSI standard leaves to the
implementation.

Declarations are generally treated as assertions. This general
principle, and its implications, and the bugs which still keep the
compiler from quite satisfying this principle, are discussed in
@ref{Compiler}.

SBCL is essentially a compiler-only implementation of Common
Lisp. That is, for all but a few special cases, @code{eval} creates a
lambda expression, calls @code{compile} on the lambda expression to
create a compiled function, and then calls @code{funcall} on the
resulting function object. This is explicitly allowed by the ANSI
standard, but leads to some oddities, e.g. collapsing @code{functionp}
and @code{compiled-function-p} into the same predicate.

@findex defconstant
SBCL is quite strict about ANSI's definition of
@code{defconstant}. ANSI says that doing @code{defconstant} of the
same symbol more than once is undefined unless the new value is
@code{eql} to the old value. Conforming to this specification is a
nuisance when the ``constant'' value is only constant under some
weaker test like @code{string=} or @code{equal}. It's especially
annoying because, in SBCL, @code{defconstant} takes effect not only at
load time but also at compile time, so that just compiling and loading
reasonable code like
@lisp
(defconstant +foobyte+ '(1 4))
@end lisp
runs into this undefined behavior. Many implementations of Common Lisp
try to help the programmer around this annoyance by silently accepting
the undefined code and trying to do what the programmer probably
meant. SBCL instead treats the undefined behavior as an error. Often
such code can be rewritten in portable ANSI Common Lisp which has the
desired behavior.  E.g., the code above can be given an exactly
defined meaning by replacing @code{defconstant} either with
@code{defparameter} or with a customized macro which does the right
thing, possibly along the lines of the @code{defconstant-eqx} macro
used internally in the implementation of SBCL itself.  In
circumstances where this is not appropriate, the programmer can handle
the condition type @code{sb-ext:defconstant-uneql}, and choose either
the @command{continue} or @command{abort} restart as appropriate.

SBCL gives style warnings about various kinds of perfectly legal code,
e.g.

@itemize
  
@item
@code{defmethod} without a preceding @code{defgeneric};
  
@item
multiple @code{defun}s of the same symbol in different units;
  
@item
special variables not named in the conventional @code{*foo*} style,
and lexical variables unconventionally named in the @code{*foo*} style

@end itemize

This causes friction with people who point out that other ways of
organizing code (especially avoiding the use of @code{defgeneric}) are
just as aesthetically stylish.  However, these warnings should be read
not as ``warning, bad aesthetics detected, you have no style'' but
``warning, this style keeps the compiler from understanding the code
as well as you might like.'' That is, unless the compiler warns about
such conditions, there's no way for the compiler to warn about some
programming errors which would otherwise be easy to overlook. (Related
bug: The warning about multiple @code{defun}s is pointlessly annoying
when you compile and then load a function containing @code{defun}
wrapped in @code{eval-when}, and ideally should be suppressed in that
case, but still isn't as of SBCL 0.7.6.)


@node  Extensions
@comment  node-name,  next,  previous,  up
@section Extensions

SBCL is derived from CMUCL, which implements many extensions to the
ANSI standard. SBCL doesn't support as many extensions as CMUCL, but
it still has quite a few.  @xref{Contributed Modules}.


@menu
* Things Which Might Be In The Next ANSI Standard::  
* Threading::                   
* Support For Unix::            
* Customization Hooks for Users::  
* Tools To Help Developers::    
* Interface To Low-Level SBCL Implementation::  
* Stale Extensions::            
* Efficiency Hacks::            
@end menu

@node  Things Which Might Be In The Next ANSI Standard
@comment  node-name,  next,  previous,  up
@subsection Things Which Might Be In The Next ANSI Standard

SBCL provides extensive support for calling external C code,
@ref{Foreign Function Interface}.

SBCL provides additional garbage collection functionality not
specified by ANSI. Weak pointers allow references to objects to be
maintained without keeping them from being GCed (garbage
collected). And ``finalization'' hooks are available to cause code to
be executed when an object has been GCed.
@c <!-- FIXME: Actually documenting these would be good.:-| -->

SBCL supports @dfn{Gray streams}, user-overloadable CLOS classes whose
instances can be used as Lisp streams (e.g. passed as the first
argument to @code{format}).  Additionally, the bundled contrib module
@dfn{sb-simple-streams} implements a subset of the Franz Allegro
simple-streams proposal.

SBCL supports a MetaObject Protocol which is intended to be compatible
with AMOP; present exceptions to this (as distinct from current bugs)
are:

@itemize
  
@item
the abstract @code{metaobject} class is not present in the class
hierarchy;
  
@item
the @code{standard-object} and @code{funcallable-standard-object}
classes are disjoint;
  
@item
@code{compute-effective-method} only returns one value, not two;
  
@item
the system-supplied @code{:around} method for @code{compute-slots}
specialized on @code{funcallable-standard-class} does not respect the
requested order from a user-supplied primary method.

@end itemize


@node  Threading
@comment  node-name,  next,  previous,  up
@subsection Threading (a.k.a Multiprocessing)

SBCL supports a fairly low-level threading interface that maps onto
the host operating system's concept of threads or lightweight
processes.  This means that threads may take advantage of hardware
multiprocessing on machines that have more than one CPU, but it does 
not allow Lisp control of the scheduler.  This is found in the
SB-THREAD package.

This requires x86 and Linux kernel 2.6 or systems with NPTL backports.

@subsubsection Special variables

The interaction of special variables with multiple threads is mostly
as one would expect, but users of other Lisps are warned that the
behaviour of locally bound specials differs in places from what they
may expect.

@itemize
@item 
global special values are visible across all threads;
@item
bindings (e.g. using LET) are local to the thread;
@item
initial values in a new thread are taken from the thread that created it. 
@end itemize

@subsubsection Mutex support

Mutexes are used for controlling access to a shared resource. One
thread is allowed to hold the mutex, others which attempt to take it
will be made to wait until it's free. Threads are woken in the order
that they go to sleep.

There isn't a timeout on mutex acquisition, but the usual WITH-TIMEOUT
macro (which throws a TIMEOUT condition after n seconds) can be used
if you want a bounded wait.

@lisp
(defpackage :demo (:use "CL" "SB-THREAD" "SB-EXT"))

(in-package :demo)

(defvar *a-mutex* (make-mutex :name "my lock"))

(defun thread-fn ()
  (let ((id (current-thread-id)))
    (format t "Thread ~A running ~%" id)
    (with-mutex (*a-mutex*)
      (format t "Thread ~A got the lock~%" id)
      (sleep (random 5)))
    (format t "Thread ~A dropped lock, dying now~%" id)))

(make-thread #'thread-fn)
(make-thread #'thread-fn)

@end lisp

@subsubsection Waitqueue/condition variables

These are based on the POSIX condition variable design, hence the
annoyingly CL-conflicting name. For use when you want to check a
condition and sleep until it's true. For example: you have a shared
queue, a writer process checking ``queue is empty'' and one or more
readers that need to know when ``queue is not empty''. It sounds
simple, but is astonishingly easy to deadlock if another process runs
when you weren't expecting it to.

There are three components:

@itemize
@item the condition itself (not represented in code)
@item the condition variable (a.k.a waitqueue) which proxies for it
@item a lock to hold while testing the condition 
@end itemize

Important stuff to be aware of:

@itemize
@item when calling condition-wait, you must hold the mutex. condition-wait will drop the mutex while it waits, and obtain it again before returning for whatever reason;

@item likewise, you must be holding the mutex around calls to condition-notify;

@item a process may return from condition-wait in several circumstances: it is not guaranteed that the underlying condition has become true. You must check that the resource is ready for whatever you want to do to it. 

@end itemize

@lisp
(defvar *buffer-queue* (make-waitqueue))
(defvar *buffer-lock* (make-mutex :name "buffer lock"))

(defvar *buffer* (list nil))

(defun reader ()
  (with-mutex (*buffer-lock*)
    (loop
     (condition-wait *buffer-queue* *buffer-lock*)
     (loop
      (unless *buffer* (return))
      (let ((head (car *buffer*)))
        (setf *buffer* (cdr *buffer*))
        (format t "reader ~A woke, read ~A~%" 
                (current-thread-id) head))))))

(defun writer ()
  (loop
   (sleep (random 5))
   (with-mutex (*buffer-lock*)
     (let ((el (intern
                (string (code-char 
                         (+ (char-code #\A) (random 26)))))))
       (setf *buffer* (cons el *buffer*)))
     (condition-notify *buffer-queue*))))

(make-thread #'writer)
(make-thread #'reader)
(make-thread #'reader)       

@end lisp

@subsubsection Sessions/Debugging

If the user has multiple views onto the same Lisp image (for example,
using multiple terminals, or a windowing system, or network access)
they are typically set up as multiple @dfn{sessions} such that each
view has its own collection of foreground/background/stopped threads.
A thread which wishes to create a new session can use
@code{sb-thread:with-new-session} to remove itself from the current
session (which it shares with its parent and siblings) and create a
fresh one.  
# See also @code{sb-thread:make-listener-thread}.

Within a single session, threads arbitrate between themselves for the
user's attention.  A thread may be in one of three notional states:
foreground, background, or stopped.  When a background process
attempts to print a repl prompt or to enter the debugger, it will stop
and print a message saying that it has stopped.  The user at his
leisure may switch to that thread to find out what it needs.  If a
background thread enters the debugger, selecting any restart will put
it back into the background before it resumes.  Arbitration for the
input stream is managed by calls to @code{sb-thread:get-foreground}
(which may block) and @code{sb-thread:release-foreground}.

@code{sb-ext:quit} terminates all threads in the current session, but
leaves other sessions running.


@subsubsection Implementation (Linux x86)

On Linux x86, threading is implemented using @code{clone()} and does
not involve pthreads.  This is not because there is anything wrong
with pthreads @emph{per se}, but there is plenty wrong (from our
perspective) with LinuxThreads.  SBCL threads are mapped 1:1 onto
Linux tasks which share a VM but nothing else - each has its own
process id and can be seen in e.g. @command{ps} output.

Per-thread local bindings for special variables is achieved using the
%fs segment register to point to a per-thread storage area.  This may
cause interesting results if you link to foreign code that expects
threading or creates new threads, and the thread library in question
uses %fs in an incompatible way.

Queues require the @code{sys_futex()} system call to be available:
this is the reason for the NPTL requirement.  We test at runtime that
this system call exists.

Garbage collection is done with the existing Conservative Generational
GC.  Allocation is done in small (typically 8k) regions: each thread
has its own region so this involves no stopping. However, when a
region fills, a lock must be obtained while another is allocated, and
when a collection is required, all processes are stopped.  This is
achieved by sending them signals, which may make for interesting
behaviour if they are interrupted in system calls.  The streams
interface is believed to handle the required system call restarting
correctly, but this may be a consideration when making other blocking
calls e.g. from foreign library code.

Large amounts of the SBCL library have not been inspected for
thread-safety.  Some of the obviously unsafe areas have large locks
around them, so compilation and fasl loading, for example, cannot be
parallelized.  Work is ongoing in this area.

A new thread by default is created in the same POSIX process group and
session as the thread it was created by.  This has an impact on
keyboard interrupt handling: pressing your terminal's intr key
(typically @kbd{Control-C}) will interrupt all processes in the
foreground process group, including Lisp threads that SBCL considers
to be notionally `background'.  This is undesirable, so background
threads are set to ignore the SIGINT signal.

@code{sb-thread:make-listener-thread} in addition to creating a new
Lisp session makes a new POSIX session, so that pressing
@kbd{Control-C} in one window will not interrupt another listener -
this has been found to be embarrassing.


@node  Support For Unix
@comment  node-name,  next,  previous,  up
@subsection Support For Unix

The UNIX command line can be read from the variable
@code{sb-ext:*posix-argv*}. The UNIX environment can be queried with
the @code{sb-ext:posix-getenv} function.

@include fun-sb-ext-posix-getenv.texinfo

The SBCL system can be terminated with @code{sb-ext:quit}, (but see
notes in @ref{Threading} about the interaction between this feature and
sessions) optionally returning a specified numeric value to the
calling Unix process. The normal Unix idiom of terminating on end of
file on input is also supported.

@include fun-sb-ext-quit.texinfo

@node  Customization Hooks for Users
@comment  node-name,  next,  previous,  up
@subsection Customization Hooks for Users

The toplevel repl prompt may be customized, and the function
that reads user input may be replaced completely.
@c <!-- FIXME but I don't currently remember how -->

The behaviour of @code{require} when called with only one argument is
implementation-defined.  In SBCL, @code{require} behaves in the
following way:

@include fun-common-lisp-require.texinfo
@include var-sb-ext-star-module-provider-functions-star.texinfo

Although SBCL does not provide a resident editor, the @code{ed}
function can be customized to hook into user-provided editing
mechanisms as follows:

@include fun-common-lisp-ed.texinfo
@include var-sb-ext-star-ed-functions-star.texinfo

@node  Tools To Help Developers
@comment  node-name,  next,  previous,  up
@subsection Tools To Help Developers

SBCL provides a profiler and other extensions to the ANSI @code{trace}
facility.  For more information, see @ref{macro-common-lisp-trace}.

The debugger supports a number of options. Its documentation is
accessed by typing @kbd{help} at the debugger prompt. @xref{Debugger}.

Documentation for @code{inspect} is accessed by typing @kbd{help} at
the @code{inspect} prompt.

@node  Interface To Low-Level SBCL Implementation
@comment  node-name,  next,  previous,  up
@subsection Interface To Low-Level SBCL Implementation

SBCL has the ability to save its state as a file for later
execution. This functionality is important for its bootstrapping
process, and is also provided as an extension to the user.  Note that
foreign libraries loaded via @code{load-shared-object} don't survive
this process; a core should not be saved in this case.

@emph{FIXME: what should be done for foreign libraries?}

@emph{FIXME: document load-shared-object somewhere - it's in
ffi.texinfo?}

@include fun-sb-ext-save-lisp-and-die.texinfo


@node Stale Extensions
@comment  node-name,  next,  previous,  up
@subsection Stale Extensions

SBCL has inherited from CMUCL various hooks to allow the user to
tweak and monitor the garbage collection process. These are somewhat
stale code, and their interface might need to be cleaned up. If you
have urgent need of them, look at the code in @file{src/code/gc.lisp}
and bring it up on the developers' mailing list.

SBCL has various hooks inherited from CMUCL, like
@code{sb-ext:float-denormalized-p}, to allow a program to take
advantage of IEEE floating point arithmetic properties which aren't
conveniently or efficiently expressible using the ANSI standard. These
look good, and their interface looks good, but IEEE support is
slightly broken due to a stupid decision to remove some support for
infinities (because it wasn't in the ANSI spec and it didn't occur to
me that it was in the IEEE spec). If you need this stuff, take a look
at the code and bring it up on the developers' mailing
list.


@node  Efficiency Hacks
@comment  node-name,  next,  previous,  up
@subsection Efficiency Hacks

The @code{sb-ext:purify} function causes SBCL first to collect all
garbage, then to mark all uncollected objects as permanent, never
again attempting to collect them as garbage. This can cause a large
increase in efficiency when using a primitive garbage collector, or a
more moderate increase in efficiency when using a more sophisticated
garbage collector which is well suited to the program's memory usage
pattern. It also allows permanent code to be frozen at fixed
addresses, a precondition for using copy-on-write to share code
between multiple Lisp processes. it is less important with modern
generational garbage collectors.

@include fun-sb-ext-purify.texinfo

The @code{sb-ext:truly-the} special form declares the type of the
result of the operations, producing its argument; the declaration is
not checked. In short: don't use it.

@include special-operator-sb-ext-truly-the.texinfo

The @code{sb-ext:freeze-type} declaration declares that a
type will never change, which can make type testing
(@code{typep}, etc.) more efficient for structure types.

The @code{sb-ext:constant-function} declaration specifies
that a function will always return the same value for the same
arguments, which may allow the compiler to optimize calls
to it. This is appropriate for functions like @code{sqrt}, but
is @emph{not} appropriate for functions like @code{aref},
which can change their return values when the underlying data are
changed.
@c <!-- FIXME: This declaration does not seem to be supported in the 
@c      current compiler. -->