[sbcl.git] / doc / beyond-ansi.xml

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1//EN"
"http://www.oasis-open.org/docbook/xml/4.1/docbookx.dtd" [
<!ENTITY % myents SYSTEM "entities.inc">
%myents;
]>

<chapter id="beyond-ansi"><title>Beyond The &ANSI; Standard</title>

<para>&SBCL; is mostly an implementation of the &ANSI; standard for
Common Lisp. However, there's some important behavior which extends
or clarifies the standard, and various behavior which outright
violates the standard.
</para>

<sect1 id="non-conformance"><title>Non-Conformance With The &ANSI; Standard</title>

<para>
  Essentially every type of non-conformance is considered a bug.
  (The exceptions involve internal inconsistencies in the standard.)
  In &SBCL; 0.7.6, the master record of known bugs is in
  the <filename>BUGS</filename> file in the distribution.
  Some highlight information about bugs may also be found in the
  manual page. The recommended way to report bugs is through the sbcl-help or
  sbcl-devel mailings lists.
</para>

</sect1>

<sect1 id="idiosyncrasies"><title>Idiosyncrasies</title>

<para>The information in this section describes some of the ways
that &SBCL; deals with choices that the &ANSI; standard 
leaves to the implementation.</para>

<para>Declarations are generally treated as assertions. This general
principle, and its implications, and the bugs which still keep the
compiler from quite satisfying this principle, are discussed in the
<link linkend="compiler">chapter on the compiler</link>.</para>

<para>&SBCL; is essentially a compiler-only implementation of
&CommonLisp;. That is, for all but a few special cases,
<function>eval</function> creates a
lambda expression, calls <function>compile</function> on the lambda
expression to create a compiled function, and then calls
<function>funcall</function> on the resulting function object. This 
is explicitly allowed by the &ANSI; standard, but leads to some
oddities, e.g. collapsing <function>functionp</function> and 
<function>compiled-function-p</function> into the same predicate.</para>

<para>&SBCL; is quite strict about ANSI's definition of
<function>defconstant</function>. ANSI says that doing <function>defconstant</function>
of the same symbol more than once is undefined unless the new value
is <function>eql</function> to the old value. Conforming to this specification
is a nuisance when the "constant" value is only constant under some
weaker test like <function>string=</function> or <function>equal</function>. It's
especially annoying because, in &SBCL;, <function>defconstant</function> takes effect
not only at load time but also at compile time, so that just 
compiling and loading reasonable code like 
<programlisting>(defconstant +foobyte+ '(1 4))</programlisting>
runs into this undefined behavior. Many
implementations of Common Lisp try to help the programmer around
this annoyance by silently accepting the undefined code and 
trying to do what the programmer probably meant. &SBCL; instead
treats the undefined behavior as an error. Often
such code can be rewritten
in portable &ANSI; Common Lisp which has the desired behavior.
E.g., the code above can be given an exactly defined meaning by replacing
<function>defconstant</function> either with <function>defparameter</function> or 
with a customized macro which does the right thing, possibly along the
lines of the <function>defconstant-eqx</function> macro used internally in the
implementation of &SBCL; itself.  In circumstances where this is not 
appropriate, the programmer can handle the condition type 
<errortype>sb-ext:defconstant-uneql</errortype>, and choose either the 
<action>continue</action> or <action>abort</action> restart as 
appropriate.</para>

<para>&SBCL; gives style warnings about various kinds of perfectly
legal code, e.g.
<itemizedlist>
  <listitem><para><function>defmethod</function> without
  <function>defgeneric</function></para></listitem>
  <listitem><para>multiple <function>defun</function>s of the same
  symbol</para></listitem>
  <listitem><para>special variables not named in the conventional
  <varname>*foo*</varname> style, and lexical variables unconventionally named
  in the <varname>*foo*</varname> style</para></listitem>
</itemizedlist>
This causes friction with people
who point out that other ways of organizing code (especially
avoiding the use of <function>defgeneric</function>)
are just as aesthetically stylish.
However, these warnings should be read not
as "warning, bad aesthetics detected, you have no style" but
"warning, this style keeps the compiler from understanding
the code as well as you might like." That is, 
unless the compiler warns about such conditions, there's no
way for the compiler to warn 
about some programming errors which would otherwise be
easy to overlook. (related bug: The warning about
multiple <function>defun</function>s is pointlessly annoying when you compile
and then load a function containing <function>defun</function> wrapped
in <function>eval-when</function>, and ideally should be suppressed in 
that case, but still isn't as of &SBCL; 0.7.6.)</para>

</sect1>

<sect1 id="extensions"><title>Extensions</title>

<para>&SBCL; is derived from &CMUCL;, which implements many extensions
to the &ANSI; standard. &SBCL; doesn't support as many extensions as
&CMUCL;, but it still has quite a few.</para>

<sect2><title>Things Which Might Be In The Next &ANSI; Standard</title>

<para>&SBCL; provides extensive support for 
calling external C code, described 
<link linkend="ffi">in its own chapter</link>.</para>

<para>&SBCL; provides additional garbage collection functionality not
specified by &ANSI;. Weak pointers allow references to objects to be
maintained without keeping them from being GCed. And "finalization"
hooks are available to cause code to be executed when an object has been
GCed.</para> <!-- FIXME: Actually documenting these would be good.:-| -->

<para>&SBCL; supports Gray streams, user-overloadable CLOS classes
whose instances can be used as Lisp streams (e.g. passed as the
first argument to <function>format</function>).  Additionally, the 
bundled contrib module <interface>sb-simple-streams</interface>
implements a subset of the Franz Allegro simple-streams proposal.</para>  

<para>&SBCL; supports a MetaObject Protocol which is intended to be
compatible with &AMOP;; present exceptions to this (as distinct from
current bugs) are:</para>
<itemizedlist>
  <listitem><para>the abstract <classname>metaobject</classname> class is not
  present in the class hierarchy;</para></listitem>
  <listitem><para>the <classname>standard-object</classname> and
  <classname>funcallable-standard-object</classname> classes are
  disjoint;</para></listitem>
  <listitem><para><function>compute-effective-method</function> only returns
  one value, not two;</para></listitem>
  <listitem><para>the system-supplied <property>:around</property> method for
  <function>compute-slots</function> specialized on
  <classname>funcallable-standard-class</classname> does not respect the
  requested order from a user-supplied primary method.</para>
  </listitem>
</itemizedlist>

</sect2>

<sect2><title>Threading (a.k.a Multiprocessing)</title>

<para>&SBCL; (as of version 0.8.3, on Linux x86 only) supports a
fairly low-level threading interface that maps onto the host operating
system's concept of threads or lightweight processes.  </para>

<sect3><title>Lisp-level view</title>

<para>A rudimentary interface to creating and managing multiple threads
can be found in the <literal>sb-thread</literal> package.  This is
intended for public consumption, so look at the exported symbols and
their documentation strings.  
</para>

<para>Dynamic bindings to symbols are per-thread.   Signal handlers
are per-thread.
</para>

<para>Mutexes and condition variables are available for 
managing access to shared data: see 
<itemizedlist>
  <listitem>
    <programlisting>(apropos "mutex" :sb-thread)</programlisting> 
  </listitem>
  <listitem>
    <programlisting>(apropos "condition" :sb-thread)</programlisting> 
  </listitem>
  <listitem> <para>and the <structname>waitqueue</structname> structure
  </para>
  </listitem>
</itemizedlist>
and poke around in their documentation strings.</para>

<sect4><title>Sessions</title>

<para>If the user has multiple views onto the same Lisp image (for
example, using multiple terminals, or a windowing system, or network
access) they are typically set up as multiple `sessions' such that
each view has its own collection of foreground/background/stopped
threads.  A thread which wishes to create a new session can use the
<function>sb-thread:with-new-session</function> to remove itself from
the current session (which it shares with its parent and siblings) and
create a fresh one.  See also
<function>sb-thread:make-listener-thread</function>.</para>

<para>Within a single session, threads arbitrate between themselves
for the user's attention.  A thread may be in one of three notional
states: foreground, background, or stopped.  When a background process
attempts to print a repl prompt or to enter the debugger, it will stop
and print a message saying that it has stopped.  The user at his
leisure may switch to that thread to find out what it needs.  If a
background thread enters the debugger, selecting any restart will put
it back into the background before it resumes.  Arbitration for the
input stream is managed by calls to
<function>sb-thread:get-foreground</function> (which may block) and
<function>sb-thread:release-foreground</function>.</para>

<para><function>sb-ext:quit</function> terminates all threads in the 
current session, but leaves other sessions running</para>
</sect4>

</sect3>

<sect3><title>Implementation (Linux x86)</title>

<para>On Linux x86, this is implemented using
<function>clone()</function> and does not involve pthreads.  This is
not because there is anything wrong with pthreads <emphasis>per
se</emphasis>, but there is plenty wrong (from our perspective) with
LinuxThreads.  &SBCL; threads are mapped 1:1 onto Linux tasks which
share a VM but nothing else - each has its own process id and can be
seen in e.g. <command>ps</command> output.
</para>

<para>Per-thread local bindings for special variables is achieved
using the %fs segment register to point to a per-thread storage area.
This may cause interesting results if you link to foreign code that
expects threading or creates new threads, and the thread library in
question uses %fs in an incompatible way.
</para>

<para>There are two implementation mechanisms for queueing.  If SBCL
was built on an NPTL-capable Linux system (2.6 or some vendor 2.4
ports) with the :SB-FUTEX feature, queuing will be done using the
<function>sys_futex()</function> system call if it's available at
runtime.  Otherwise it will fall back to using
<function>sigtimedwait()</function> to sleep and a signal
(SIG_DEQUEUE, one of the POSIX RT signals) to wake.</para>

<para>Garbage collection is done with the existing Conservative
Generational GC.  Allocation is done in small (typically 8k) regions :
each thread has its own region so this involves no stopping. However,
when a region fills, a lock must be obtained while another is
allocated, and when a collection is required, all processes are
stopped.  This is achieved by sending them signals, which may make for 
interesting behaviour if they are interrupted in system calls.  The
streams interface is believed to handle the required system call
restarting correctly, but this may be a consideration when making
other blocking calls e.g. from foreign library code.
</para>

<para>Large amounts of the &SBCL; library have not been inspected for
thread-safety.  Some of the obviously unsafe areas have large locks
around them, so compilation and fasl loading, for example, cannot be
parallelized.  Work is ongoing in this area.
</para>

<para>A new thread by default is created in the same POSIX process
group and session as the thread it was created by.  This has an impact
on keyboard interrupt handling: pressing your terminal's intr key
(typically Control-C) will interrupt all processes in the foreground
process group, including Lisp threads that &SBCL; considers to be
notionally `background'.  This is undesirable, so background threads
are set to ignore the SIGINT signal.</para>

<para><function>sb-thread:make-listener-thread</function> in addition
to creating a new Lisp session makes a new POSIX session, so that
pressing Control-C in one window will not interrupt another listener -
this has been found to be embarrassing.
</para>

</sect3>

</sect2>

<sect2><title>Support For Unix</title>

<para>The UNIX command line can be read from the variable
<varname>sb-ext:*posix-argv*</varname>. The UNIX environment can be queried with the
<function>sb-ext:posix-getenv</function> function.</para>

<para>The &SBCL; system can be terminated with
<function>sb-ext:quit</function>, (but see notes above about threads
and sessions) optionally returning a specified numeric value to the
calling Unix process. The normal Unix idiom of terminating on end of
file on input is also supported.</para>

</sect2>

<sect2><title>Customization Hooks for Users</title>

<para>The behaviour of <function>require</function> when called with only
one argument is implementation-defined.  In &SBCL; it calls functions
on the user-settable list <varname>sb-ext:*module-provider-functions*</varname>
- see the <function>require</function> documentation string for details.
</para>

<para>The toplevel repl prompt may be customized, and the function
that reads user input may be replaced completely.  <!-- FIXME but I 
don't currently remember how -->
</para>

</sect2>

<sect2><title>Tools To Help Developers</title>

<para>&SBCL; provides a profiler and other extensions to the &ANSI;
<function>trace</function> facility. See the online function documentation for
<function>trace</function> for more information.</para>

<para>The debugger supports a number of options. Its documentation is
accessed by typing <userinput>help</userinput> at the debugger prompt.</para>
<!-- FIXME:
     A true debugger section in the manual would be good. Start
     with CMU CL's debugger section, but remember:
     * no QUIT command (TOPLEVEL restart instead)
     * no GO command (CONTINUE restart instead)
     * Limitations of the x86 port of the debugger should be 
     documented or fixed where possible.
     * Discuss TRACE and its unification with PROFILE. -->

<para>Documentation for <function>inspect</function> is accessed by typing
<userinput>help</userinput> at the <function>inspect</function> prompt.</para>

</sect2>

<sect2><title>Interface To Low-Level &SBCL; Implementation</title>

<para>&SBCL; has the ability to save its state as a file for later
execution. This functionality is important for its bootstrapping
process, and is also provided as an extension to the user See the
documentation for <function>sb-ext:save-lisp-and-die</function> for more
information.</para>

<note><para>&SBCL; has inherited from &CMUCL; various hooks to allow
the user to tweak and monitor the garbage collection process. These
are somewhat stale code, and their interface might need to be cleaned
up. If you have urgent need of them, look at the code in
<filename>src/code/gc.lisp</filename> and bring it up on the
developers' mailing list.</para></note>

<note><para>&SBCL; has various hooks inherited from &CMUCL;, like
<function>sb-ext:float-denormalized-p</function>, to allow a program to take
advantage of &IEEE; floating point arithmetic properties which aren't
conveniently or efficiently expressible using the &ANSI; standard. These
look good, and their interface looks good, but &IEEE; support is
slightly broken due to a stupid decision to remove some support for
infinities (because it wasn't in the &ANSI; spec and it didn't occur to
me that it was in the &IEEE; spec). If you need this stuff, take a look
at the code and bring it up on the developers' mailing
list.</para></note>

</sect2>

<sect2><title>Efficiency Hacks</title>

<para>The <function>sb-ext:purify</function> function causes &SBCL;
first to collect all garbage, then to mark all uncollected objects as
permanent, never again attempting to collect them as garbage. This can
cause a large increase in efficiency when using a primitive garbage
collector, or a more moderate increase in efficiency when using a more
sophisticated garbage collector which is well suited to the program's
memory usage pattern. It also allows permanent code to be frozen at
fixed addresses, a precondition for using copy-on-write to share code
between multiple Lisp processes. is less important with modern
generational garbage collectors. </para>

<para>The <function>sb-ext:truly-the</function> declares the type of the
result of the operations, producing its argument; the declaration is
not checked. In short: don't use it.</para>

<para>The <function>sb-ext:freeze-type</function> declaration declares that a
type will never change, which can make type testing
(<function>typep</function>, etc.) more efficient for structure types.</para>

<para>The <function>sb-ext:constant-function</function> declaration specifies
that a function will always return the same value for the same
arguments, which may allow the compiler to optimize calls
to it. This is appropriate for functions like <function>sqrt</function>, but
is <emphasis>not</emphasis> appropriate for functions like <function>aref</function>,
which can change their return values when the underlying data are
changed.</para>
<!-- FIXME: This declaration does not seem to be supported in the 
     current compiler. -->

</sect2>

</sect1>

</chapter>