Foreign Linkage
***************

Linkage-table
=============

Linkage-table allows saving cores with foreign code loaded, and is also utilized to allow references to as-of-yet unknown aliens. See Lazy Alien Resolution.

The SBCL implementation is somewhat simplified from the CMUCL one by Timothy Moore, but the basic idea and mechanism remains identical: instead of having addresses from dlsym(3) in the core, we have addresses to an mmapped memory area (LINKAGE_TABLE_SPACE) that is initialized at startup to contain jumps & references to the correct addresses, based on information stored on the lisp side in *LINKAGE-INFO*.

Differences to CMUCL
--------------------

CMUCL does lazy linkage for code, keeps all foreign addresses in the linkage-table, and handles the initialization from C. We do eager linkage for everything, maintain a separate *STATIC-FOREIGN-SYMBOLS* just like on non-linkage-table ports (this allows more code sharing between ports, makes thread-safety easier to achieve, and cuts one jump's worth of overhead from stuff like closure_tramp), and do the initialization from lisp.

Nitty Gritty Details
--------------------

Symbols in *STATIC-FOREIGN-SYMBOLS* are handled the old fashioned way: linkage-table is only used for symbols resolved with dlsym(3).

On system startup FOREIGN-REINIT iterates through the *LINKAGE-INFO*, which is a hash-table mapping dynamic foreign names to LINKAGE-INFO structures, and calls arch_write_linkage_table_jmp/ref to write the appropriate entries to the linkage-table.

When a foreign symbol is referred to, it is first looked for in the *STATIC-FOREIGN-SYMBOLS*. If not found, ENSURE-FOREIGN-LINKAGE is called, which looks for the corresponding entry in *LINKAGE-INFO*, creating one and writing the appropriate entry in the linkage table if necessary.

FOREIGN-SYMBOL-ADDRESS and FOREIGN-SYMBOL-ADDRESS-AS-INTEGER take an optional datap argument, used to indicate that the symbol refers to a variable. In similar fashion there is a new kind of fixup and a new VOP: :FOREIGN-DATAREF and FOREIGN-SYMBOL-DATAREF-ADDRESS. The DATAP argument is automagically provided by the alien interface for normal definitions, but is really needed only for dynamic foreign variables. For those it indicates the need for the indirection either within a conditional branch in FOREIGN-SYMBOL-ADDRESS, or via :FOREIGN-DATAREF fixup and FOREIGN-SYMBOL-DATAREF-ADDRESS VOP: "this address holds the address of the foreign variable, not the variable itself". Within SBCL itself (in the fixups manifest in various VOPs) this fixup type is never used, as all foreign symbols used internally are static.

One thing worth noting is that FOREIGN-SYMBOL-ADDRESS and friends now have the potential side-effect of entering information in *LINKAGE-INFO* and the linkage-table proper: hence it's important to use the correct datap argument even if calling these just to "check if it's there" (like SB-POSIX does).

Porting
-------

Porting to new operating systems
................................

Find a memory area for the linkage-table, and add it for the OS in src/compiler/target/parms.lisp by defining SB!VM:LINKAGE-TABLE-SPACE-START and SB!VM:LINKAGE-TABLE-SPACE-END. See existing ports and CMUCL for examples.

Porting to new architextures
............................

Write arch_write_linkage_table_jmp and  arch_write_linkage_table_ref.

Write FOREIGN-SYMBOL-DATAREF VOP.

Define correct SB!VM:LINKAGE-TABLE-ENTRY-SIZE in src/compiler/target/parms.lisp.

Lazy Alien Resolution
=====================

On linkage-table ports SBCL is able to deal with forward-references to aliens -- which is to say, compile and load code referring to aliens before the shared object containing the alien in question has been loaded.

This is handled by GET-DYNAMIC-FOREIGN-SYMBOL-ADDRESS, which first tries to resolve the address in the loaded shared objects, but failing that records the alien as undefined and returns the address of a read/write/execute protected guard page for variables, and address of undefined_alien_function for routines. These are in turn responsible for catching attempts to access the undefined alien, and signalling the appropriate error.

These placeholder addresses get recorded in the linkage-table.

When new shared objects are loaded UPDATE-LINKAGE-TABLE is called, which in turn attempts to resolve all currently undefined aliens, and registers the correct addresses for them in the linkage-table.

Callbacks
=========

SBCL is capable of providing C with linkage to Lisp -- the upshot of which is that C-functions can call Lisp functions thru what look like function pointers to C.

These ``function pointers'' are called Alien Callbacks. An alien callback sequence has 4 parts / stages / bounces:

  * Assembler Wrapper

     saves the arguments from the C-call according to the alien-fun-type of the callback, and calls #'ENTER-ALIEN-CALLBACK with the index indentifying the callback, a pointer to the arguments copied on the stack and a pointer to return value storage. When control returns to the wrapper it returns the value to C. There is one assembler wrapper per callback.[1] The SAP to the wrapper code vector is what is passed to foreign code as a callback.

     The Assembler Wrapper is generated by ALIEN-CALLBACK-ASSEMBLER-WRAPPER.

  * #'ENTER-ALIEN-CALLBACK

     pulls the Lisp Trampoline for the given index, and calls it with the argument and result pointers.

  * Lisp Trampoline

     calls the Lisp Wrapper with the argument and result pointers, and the function designator for the callback. There is one lisp trampoline per callback.

  * Lisp Wrapper

     parses the arguments from stack, calls the actual callback with the arguments, and saves the return value at the result pointer. The lisp wrapper is shared between all the callbacks having the same same alien-fun-type.

[1] As assembler wrappers need to be allocated in static addresses and are (in the current scheme of things) never released it might be worth it to split it into two parts: per-callback trampoline that pushes the index of the lisp trampoline on the stack, and jumps to the appropriate assembler wrapper. The assembler wrapper could then be shared between all the callbacks with the same alien-fun-type. This would amortize most of the static allocation costs between multiple callbacks.