From 927001993f81c25f387e124cb17ebd7db4b7cb37 Mon Sep 17 00:00:00 2001 From: Nikodemus Siivola Date: Sun, 19 Jun 2005 16:16:48 +0000 Subject: [PATCH] 0.9.1.57: just how hard can "cvs add" be? ...the internals manual, really, this time... --- doc/internals/Makefile | 17 ++++ doc/internals/foreign-linkage.texinfo | 172 +++++++++++++++++++++++++++++++++ doc/internals/make-top.sh | 38 ++++++++ doc/internals/sbcl-internals.texinfo | 54 +++++++++++ version.lisp-expr | 2 +- 5 files changed, 282 insertions(+), 1 deletion(-) create mode 100644 doc/internals/Makefile create mode 100644 doc/internals/foreign-linkage.texinfo create mode 100644 doc/internals/make-top.sh create mode 100644 doc/internals/sbcl-internals.texinfo diff --git a/doc/internals/Makefile b/doc/internals/Makefile new file mode 100644 index 0000000..a620b84 --- /dev/null +++ b/doc/internals/Makefile @@ -0,0 +1,17 @@ +.PHONY: top info clean all + +all: info pdf + +top: + sh make-top.sh + +info: top + makeinfo sbcl-internals.texinfo + +pdf: top + texi2pdf sbcl-internals.texinfo + +clean: + rm -f *.include *.info *.pdf *~ *.cp *.fn *.ky *.log *.pg *.toc \ + *.tp *.vr *.aux + diff --git a/doc/internals/foreign-linkage.texinfo b/doc/internals/foreign-linkage.texinfo new file mode 100644 index 0000000..d08d77b --- /dev/null +++ b/doc/internals/foreign-linkage.texinfo @@ -0,0 +1,172 @@ +@node Foreign Linkage +@comment node-name, next, previous, up +@chapter Foreign Linkage + +@menu +* Linkage-table:: +* Lazy Alien Resolution:: +* Callbacks:: +@end menu + +@node Linkage-table +@comment node-name, next, previous, up +@section Linkage-table + +Linkage-table allows saving cores with foreign code loaded, and is +also utilized to allow references to as-of-yet unknown aliens. +@xref{Lazy Alien Resolution}. + +The SBCL implementation is somewhat simplified from the CMUCL one by +Timothy Moore, but the basic idea and mechanism remains identical: +instead of having addresses from @code{dlsym(3)} in the core, we have +addresses to an mmapped memory area (@code{LINKAGE_TABLE_SPACE}) that +is initialized at startup to contain jumps & references to the correct +addresses, based on information stored on the lisp side in +@code{*LINKAGE-INFO*}. + +@subsection Differences to CMUCL + +CMUCL does lazy linkage for code, keeps all foreign addresses in the +linkage-table, and handles the initialization from C. We do eager +linkage for everything, maintain a separate +@code{*STATIC-FOREIGN-SYMBOLS*} just like on non-linkage-table ports +(this allows more code sharing between ports, makes thread-safety +easier to achieve, and cuts one jump's worth of overhead from stuff +like closure_tramp), and do the initialization from lisp. + +@subsection Nitty Gritty Details + +Symbols in @code{*STATIC-FOREIGN-SYMBOLS*} are handled the old +fashioned way: linkage-table is only used for symbols resolved with +@code{dlsym(3)}. + +On system startup @code{FOREIGN-REINIT} iterates through the +@code{*LINKAGE-INFO*}, which is a hash-table mapping dynamic foreign +names to @code{LINKAGE-INFO} structures, and calls +@code{arch_write_linkage_table_jmp}@code{/ref} to write the +appropriate entries to the linkage-table. + +When a foreign symbol is referred to, it is first looked for in the +@code{*STATIC-FOREIGN-SYMBOLS*}. If not found, +@code{ENSURE-FOREIGN-LINKAGE} is called, which looks for the +corresponding entry in @code{*LINKAGE-INFO*}, creating one and writing +the appropriate entry in the linkage table if necessary. + +@code{FOREIGN-SYMBOL-ADDRESS} and +@code{FOREIGN-SYMBOL-ADDRESS-AS-INTEGER} take an optional datap +argument, used to indicate that the symbol refers to a variable. In +similar fashion there is a new kind of fixup and a new VOP: +@code{:FOREIGN-DATAREF} and @code{FOREIGN-SYMBOL-DATAREF-ADDRESS}. The +@code{DATAP} argument is automagically provided by the alien interface +for normal definitions, but is really needed only for dynamic foreign +variables. For those it indicates the need for the indirection either +within a conditional branch in @code{FOREIGN-SYMBOL-ADDRESS}, or via +@code{:FOREIGN-DATAREF} fixup and +@code{FOREIGN-SYMBOL-DATAREF-ADDRESS} VOP: "this address holds the +address of the foreign variable, not the variable itself". Within SBCL +itself (in the fixups manifest in various VOPs) this fixup type is +never used, as all foreign symbols used internally are static. + +One thing worth noting is that @code{FOREIGN-SYMBOL-ADDRESS} and +friends now have the potential side-effect of entering information in +@code{*LINKAGE-INFO*} and the linkage-table proper: hence it's important to +use the correct datap argument even if calling these just to "check if +it's there" (like SB-POSIX does). + +@subsection Porting + +@subsubsection Porting to new operating systems + +Find a memory area for the linkage-table, and add it for the OS in +@file{src/compiler/target/parms.lisp} by defining +@code{SB!VM:LINKAGE-TABLE-SPACE-START} and +@code{SB!VM:LINKAGE-TABLE-SPACE-END}. See existing ports and CMUCL for +examples. + +@subsubsection Porting to new architextures + +Write @code{arch_write_linkage_table_jmp} and @code{arch_write_linkage_table_ref}. + +Write @code{FOREIGN-SYMBOL-DATAREF} VOP. + +Define correct @code{SB!VM:LINKAGE-TABLE-ENTRY-SIZE} in +@file{src/compiler/target/parms.lisp}. + +@page +@node Lazy Alien Resolution +@comment node-name, next, previous, up +@section Lazy Alien Resolution + +On linkage-table ports SBCL is able to deal with forward-references to +aliens -- which is to say, compile and load code referring to aliens +before the shared object containing the alien in question has been +loaded. + +This is handled by @code{GET-DYNAMIC-FOREIGN-SYMBOL-ADDRESS}, which +first tries to resolve the address in the loaded shared objects, but +failing that records the alien as undefined and returns the address of +a read/write/execute protected guard page for variables, and address +of @code{undefined_alien_function} for routines. These are in turn +responsible for catching attempts to access the undefined alien, and +signalling the appropriate error. + +These placeholder addresses get recorded in the linkage-table. + +When new shared objects are loaded @code{UPDATE-LINKAGE-TABLE} is +called, which in turn attempts to resolve all currently undefined +aliens, and registers the correct addresses for them in the +linkage-table. + +@page +@node Callbacks +@comment node-name, next, previous, up +@section Callbacks + +SBCL is capable of providing C with linkage to Lisp -- the upshot of which is that +C-functions can call Lisp functions thru what look like function pointers to C. + +These ``function pointers'' are called Alien Callbacks. An alien +callback sequence has 4 parts / stages / bounces: + +@itemize +@item Assembler Wrapper + +saves the arguments from the C-call according to the alien-fun-type of +the callback, and calls #'ENTER-ALIEN-CALLBACK with the index +indentifying the callback, a pointer to the arguments copied on the +stack and a pointer to return value storage. When control returns to +the wrapper it returns the value to C. There is one assembler wrapper +per callback.[1] The SAP to the wrapper code vector is what is passed +to foreign code as a callback. + +The Assembler Wrapper is generated by +@code{ALIEN-CALLBACK-ASSEMBLER-WRAPPER}. + +@item #'ENTER-ALIEN-CALLBACK + +pulls the Lisp Trampoline for the given index, and calls it with the +argument and result pointers. + +@item Lisp Trampoline + +calls the Lisp Wrapper with the argument and result pointers, and the +function designator for the callback. There is one lisp trampoline per +callback. + +@item Lisp Wrapper + +parses the arguments from stack, calls the actual callback with the +arguments, and saves the return value at the result pointer. The lisp +wrapper is shared between all the callbacks having the same same +alien-fun-type. + +@end itemize + +[1] As assembler wrappers need to be allocated in static addresses and +are (in the current scheme of things) never released it might be worth +it to split it into two parts: per-callback trampoline that pushes the +index of the lisp trampoline on the stack, and jumps to the +appropriate assembler wrapper. The assembler wrapper could then be +shared between all the callbacks with the same alien-fun-type. This +would amortize most of the static allocation costs between multiple +callbacks. diff --git a/doc/internals/make-top.sh b/doc/internals/make-top.sh new file mode 100644 index 0000000..2f307b8 --- /dev/null +++ b/doc/internals/make-top.sh @@ -0,0 +1,38 @@ +#!/bin/sh + +# After the ordeal, we went back to the Factory. Greenblatt said he +# was gonna locate us in a cell. He said: "Kid, I'm gonna INTERN you in a +# cell. I want your manual and your mouse." +# I said, "Greenblatt, I can understand your wantin' my manual, so +# I don't have any documentation about the cell, but what do you want my +# mouse for?" and he said, "Kid, we don't want any window system problems". +# I said, "Greenblatt, did you think I was gonna deexpose myself for +# litterin'?" +# Greenblatt said he was makin' sure, and, friends, Greenblatt +# was, 'cause he took out the left Meta-key so I couldn't double bucky the +# rubout and cold-boot, and he took out the Inspector so I couldn't +# click-left on Modify, set the PROCESS-WARM-BOOT-ACTION on the window, +# *THROW around the UNWIND-PROTECT and have an escape. Greenblatt was +# makin' sure. +# -- from Alice's Lispm (or MIT's AI Lab) + +# This software is part of the SBCL system. See the README file for +# more information. +# +# This software is derived from the CMU CL system, which was +# written at Carnegie Mellon University and released into the +# public domain. The software is in the public domain and is +# provided with absolutely no warranty. See the COPYING and CREDITS +# files for more information. + +rm -f *.include + +echo "@menu" > top-menu.include +for texinfo in *.texinfo +do + if ! [ $texinfo = sbcl-internals.texinfo ]; then + grep @node $texinfo | head -n 1 | perl -p -e "s/\@node\ (.*)/* \$1::/" >> top-menu.include + echo "@include $texinfo" >> top-include.include + fi +done +echo "@end menu" >> top-menu.include diff --git a/doc/internals/sbcl-internals.texinfo b/doc/internals/sbcl-internals.texinfo new file mode 100644 index 0000000..676cd20 --- /dev/null +++ b/doc/internals/sbcl-internals.texinfo @@ -0,0 +1,54 @@ +\input texinfo @c -*-texinfo-*- +@c %**start of header +@setfilename sbcl-internals.info +@settitle SBCL Internals +@c %**end of header + +@settitle SBCL Internals + +@c for install-info +@dircategory Software development +@direntry +* sbcl-internals: (sbcl internals). SBCL internal documentation +@end direntry + +@copying +@quotation +This manual is part of the SBCL software system. See the +@file{README} file for more information. + +This manual is in the public domain and is provided with absolutely no +warranty. See the @file{COPYING} and @file{CREDITS} files for more +information. +@end quotation +@end copying + +@titlepage + +@title SBCL Internals +@c @author The CMUCL and SBCL teams + +@c The following two commands start the copyright page. +@page +@vskip 0pt plus 1filll +@insertcopying + +@end titlepage + +@contents + +@ifnottex + +@node Top +@comment node-name, next, previous, up +@top SBCL Internals + +@insertcopying + +@include top-menu.include + +@end ifnottex + +@include top-include.include + +@bye diff --git a/version.lisp-expr b/version.lisp-expr index 33e0104..3b49b95 100644 --- a/version.lisp-expr +++ b/version.lisp-expr @@ -17,4 +17,4 @@ ;;; checkins which aren't released. (And occasionally for internal ;;; versions, especially for internal versions off the main CVS ;;; branch, it gets hairier, e.g. "0.pre7.14.flaky4.13".) -"0.9.1.56" +"0.9.1.57" -- 1.7.10.4