Chibi-Scheme

Alex Shinn

Tue Apr 16 06:57:12 2013

Minimal Scheme Implementation for use as an Extension Language

http://synthcode.com/wiki/chibi-scheme/

Introduction

Chibi-Scheme is a very small library intended for use as an extension and scripting language in C programs. In addition to support for lightweight VM-based threads, each VM itself runs in an isolated heap allowing multiple VMs to run simultaneously in different OS threads.

The default language is an extended subset of the current draft R7RS Scheme, with support for all libraries. Support for additional languages such as JavaScript, Go, Lua and Bash are planned for future releases. Scheme is chosen as a substrate because its first class continuations and guaranteed tail-call optimization makes implementing other languages easy.

The system is designed in optional layers, beginning with a VM based on a small set of opcodes, a set of primitives implemented in C, a default language, a module system implementation, and a set of standard modules. You can choose whichever layer suits your needs best and customize the rest. Adding your own primitives or wrappers around existing C libraries is easy with the C FFI.

Chibi is known to build and run on 32 and 64-bit Linux, FreeBSD, DragonFly, OS X, iOS, Windows (under Cygwin) and Plan9.

Installation

To build, just run "make". This will provide a shared library "libchibi-scheme", as well as a sample "chibi-scheme" command-line repl. The "chibi-scheme-static" make target builds an equivalent static executable. If your make doesn't support GNU make conditionals, then you'll need to edit the top of the Makefile to choose the appropriate settings. On Plan9 just run "mk". You can test the build with "make test".

To install run "make install". If you want to try the executable out without installing, you will probably need to set LD_LIBRARY_PATH, depending on your platform. If you have an old version installed, run "make uninstall" first, or manually delete the directory.

You can edit the file chibi/features.h for a number of settings, mostly disabling features to make the executable smaller. You can specify standard options directly as arguments to make, for example 

make CFLAGS=-Os CPPFLAGS=-DSEXP_USE_NO_FEATURES=1

to optimize for size, or 

make LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include

to compile against a library installed in /usr/local.

By default Chibi uses a custom, precise, non-moving GC (non-moving is important so you can maintain references from C code). You can link against the Boehm conservative GC by editing the features.h file, or directly from make with: 

make SEXP_USE_BOEHM=1

To compile a static executable, use 

make chibi-scheme-static SEXP_USE_DL=0

To compile a static executable with all C libraries statically included, first you need to create a clibs.c file, which can be done with: 

make clibs.c

or edited manually. Be sure to run this with a non-static chibi-scheme. Then you can make the static executable with: 

make -B chibi-scheme-static SEXP_USE_DL=0 CPPFLAGS=-DSEXP_USE_STATIC_LIBS

Compile-Time Options

The include file "chibi/features.h" describes a number of C preprocessor values which can be enabled or disabled by setting to 1 or 0 respectively. For example, the above commands used the features SEXP_USE_BOEHM, SEXP_USE_DL and SEXP_USE_STATIC_LIBS. Many features are still experimental and may be removed from future releases, but the important features are listed below.

Installed Programs

The command-line programs chibi-scheme, chibi-doc and chibi-ffi are installed by default, along with manpages. chibi-scheme provides a REPL and way to run scripts. In the interest of size it has no --help option - see the man page for usage. chibi-doc is the command-line interface to the literate documentation system described in (chibi scribble), and used to build this manual. chibi-ffi is a tool to build wrappers for C libraries, described in the FFI section below.

Default Language

Scheme Standard

The default language is based on the latest draft of R7RS, which is mostly a superset of R5RS. Some of the more expensive bindings are not included in the interest of size and quick startup, and some extra low-level utilities are included for convenience and bootstrapping. Note the builtin equal? does not support cyclic structures (you need the R7RS (scheme base) or (chibi equiv)), nor do the default reader and writer (you need (srfi 38) or the R7RS (scheme read) and (scheme write)).

To get the exact R7RS language, you can (import (scheme base)), and likewise for the other R7RS libraries.

The reader defaults to case-sensitive, like R6RS and R7RS but unlike R5RS. The default configuration includes the full numeric tower: fixnums, flonums, bignums, exact rationals and complex numbers.

Full continuations are supported, but currently continuations don't take C code into account. This means that you can call from Scheme to C and then from C to Scheme again, but continuations passing through this chain may not do what you expect. The only higher-order C functions (thus potentially running afoul of this) in the standard environment are load and eval. The result of invoking a continuation created by a different thread is also currently unspecified.

In R7RS (and R6RS) semantics it is impossible to use two macros from different modules which both use the same auxiliary keywords (like else in cond forms) without renaming one of the keywords. By default Chibi considers all top-level bindings effectively unbound when matching auxiliary keywords, so this case will "just work". This decision was made because the chance of different modules using the same keywords seems more likely than user code unintentionally matching a top-level keyword with a different binding, however if you want to use R7RS semantics you can compile with SEXP_USE_STRICT_TOPLEVEL_BINDINGS=1.

load is extended to accept an optional environment argument, like eval. You can also load shared libraries in addition to Scheme source files - in this case the function sexp_init_library is automatically called with the following signature:

sexp_init_library(sexp context, sexp self, sexp_sint_t n, sexp environment, const char* version, sexp_abi_identifier_t abi);

The following additional procedures are available in the default environment:

Module System

Chibi uses the R7RS module system natively, which is a simple static module system in the style of the Scheme48 module system. As with most features this is optional, and can be ignored or completely disabled at compile time.

Modules names are hierarchical lists of symbols or numbers. A module definition uses the following form: 

  (define-library (foo bar baz)
    <library-declarations> ...)

where <library-declarations> can be any of 

  (export <id> ...)                    ;; specify an export list
  (import <import-spec> ...)           ;; specify one or more imports
  (begin <expr> ...)                   ;; inline Scheme code
  (include <file> ...)                 ;; load one or more files
  (include-shared <file> ...)          ;; dynamic load a library

<import-spec> can either be a module name or any of 

  (only <import-spec> <id> ...)
  (except <import-spec> <id> ...)
  (rename <import-spec> (<from-id> <to-id>) ...)
  (prefix <prefix-id> <import-spec>)

These forms perform basic selection and renaming of individual identifiers from the given module. They may be composed to perform combined selection and renaming.

Some modules can be statically included in the initial configuration, and even more may be included in image files, however in general modules are searched for in a module load path. The definition of the module (foo bar baz) is searched for in the file "foo/bar/baz.sld". The default module path includes the installed directories, "." and "./lib". Additional directories can be specified with the command-line options -I and -A (see the command-line options below) or with the add-modue-directory procedure at runtime. You can search for a module file with (find-module-file <file>), or load it with (load-module-file <file> <env>).

Within the module definition, files are loaded relative to the .sld file, and are written with their extension (so you can use whatever suffix you prefer - .scm, .ss, .sls, etc.).

Shared modules, on the other hand, should be specified without the extension - the correct suffix will be added portably (e.g. .so for Unix and .dylib for OS X).

You may also use cond-expand and arbitrary macro expansions in a module definition to generate <module-declarations>.

Macro System

syntax-rules macros are provided by default, with the extensions from SRFI-46. In addition, low-level hygienic macros are provided with a syntactic-closures interface, including sc-macro-transformer, rsc-macro-transformer, and er-macro-transformer. A good introduction to syntactic-closures can be found at http://community.schemewiki.org/?syntactic-closures.

identifier?, identifier->symbol, identifier=?, and make-syntactic-closure and strip-syntactic-closures are also available.

Types

You can define new record types with SRFI-9, or inherited record types with SRFI-99. These are just syntactic sugar for the following more primitive type constructors: 

(register-simple-type <name-string> <parent> <num-fields>)
 => <type>

(make-type-predicate <opcode-name-string> <type>)
 => <opcode>  ; takes 1 arg, returns #t iff that arg is of the type

(make-constructor <constructor-name-string> <type>)
 => <opcode>  ; takes 0 args, returns a newly allocated instance of type

(make-getter <getter-name-string> <type> <field-index>)
 => <opcode>  ; takes 1 args, retrieves the field located at the index

(make-setter <setter-name-string> <type> <field-index>)
 => <opcode>  ; takes 2 args, sets the field located at the index

Unicode

Chibi supports Unicode strings, encoding them as utf8. This provides easy interoperability with many C libraries, but means that string-ref and string-set! are O(n), so they should be avoided in performance-sensitive code.

In general you should use high-level APIs such as string-map to ensure fast string iteration. String ports also provide a simple way to efficiently iterate and construct strings, by looping over an input string or accumulating characters in an output string.

The in-string and in-string-reverse iterators in the (chibi loop) module will also iterate over strings efficiently while hiding the low-level details.

In the event that you do need a low-level interface, such as when writing your own iterator protocol, you should use the following string cursor API instead of indexes.

Embedding in C

Quick Start

To use Chibi-Scheme in a program you need to link against the "libchibi-scheme" library and include the "eval.h" header file:

#include <chibi/eval.h>

All definitions begin with a "sexp_" prefix, or "SEXP_" for constants. In addition to the prototypes and utility macros, this includes the following type definitions:

A simple program might look like: 

void dostuff(sexp ctx) {
  /* declare and preserve local variables *
  sexp_gc_var2(obj1, obj2);
  sexp_gc_preserve2(ctx, obj1, obj2);

  /* load a file containing Scheme code *
  obj1 = sexp_c_string(ctx, "/path/to/source/file.scm", -1);
  sexp_load(ctx, obj1, NULL);

  /* eval a C string as Scheme code *
  sexp_eval_string(ctx, "(some scheme expression)", -1, NULL);

  /* construct a Scheme expression to eval *
  obj1 = sexp_intern(ctx, "my-procedure", -1);
  obj2 = sexp_cons(ctx, obj1, SEXP_NULL);
  sexp_eval(ctx, obj2, NULL);

  /* release the local variables *
  sexp_gc_release2(ctx);
}

int main(int argc, char** argv) {
  sexp ctx;
  ctx = sexp_make_eval_context(NULL, NULL, NULL, 0, 0);
  sexp_load_standard_env(ctx, NULL, SEXP_SEVEN);
  sexp_load_standard_ports(ctx, NULL, stdin, stdout, stderr, 0);
  dostuff(ctx);
  sexp_destroy_context(ctx);
}

Looking at main, sexp_make_eval_context and sexp_destroy_context create and destroy a "context", which manages the heap and VM state. The meaning of the arguments is explained in detail below, but these values will give reasonable defaults, in this case constructing an environment with the core syntactic forms, opcodes, and standard C primitives.

This is still a fairly bare environment, so we call sexp_load_standard_env to find and load the default initialization file.

The resulting context can then be used to construct objects, call functions, and most importantly evaluate code, as is done in dostuff. The default garbage collector for Chibi is precise, which means we need to declare and preserve references to any temporary values we may generate, which is what the sexp_gc_var2, sexp_gc_preserve2 and sexp_gc_release2 macros do (there are similar macros for values 1-6). Precise GCs prevent a class of memory leaks (and potential attackes based thereon), but if you prefer convenience then Chibi can be compiled with a conservative GC and you can ignore these.

The interesting part is then the calls to sexp_load, eval_string and eval which evaluate code stored in files, C strings, or represented as s-expressions respectively.

Destroying a context runs any finalizers for all objects in the heap and then frees the heap memory (but has no effect on other contexts you or other users of the library may have created).

Contexts and Evaluation

Contexts represent the state needed to perform evaluation. This includes keeping track of the heap (when using precise GC), a default environment, execution stack, and any global values. A program being evaluated in one context may spawn multiple child contexts, such as when you call eval, and each child will share the same heap and globals. When using multiple interpreter threads, each thread has its own context.

You can also create independent contexts with their own separate heaps. These can run simultaneously in multiple OS threads without any need for synchronization.

Garbage Collection

Chibi uses a precise garbage collector by default, which means when performing multiple computations on the C side you must explicitly preserve any temporary values. You can declare variables to be preserved with sexp_gc_varn, for n from 1 to 6.

You can declare additional macros for larger values of n if needed.

sexp_gc_varn(obj1, obj2, ..., objn)

This is equivalent to the declaration

sexp obj1, obj2, ..., objn;

except it makes preservation possible. Because it is a declaration it must occur at the beginning of your function, and because it includes assignments (in the macro-expanded form) it should occur after all other declarations.

To preserve these variables for a given context, you can then use sexp_gc_preserven:

sexp_gc_preserven(ctx, obj1, obj2, ..., objn)

This can be delayed in your code until you know a potentially memory-allocating computation will be performed, but once you call sexp_gc_preserven it must be paired with a matching sexp_gc_releasen:

sexp_gc_releasen(ctx);

Note each of these have different signatures. sexp_gc_varn just lists the variables to be declared. sexp_gc_preserven prefixes these with the context in which they are to be preserved, and sexp_gc_releasen just needs the context.

A typical usage for these is: 

sexp foo(sexp ctx, sexp bar, sexp baz) {
  /* variable declarations *
  int i, j;
  ...
  sexp_gc_var3(tmp1, tmp2, res);

  /* asserts or other shortcut returns *
  sexp_assert_type(ctx, sexp_barp, SEXP_BAR, bar);
  sexp_assert_type(ctx, sexp_bazp, SEXP_BAZ, baz);

  /* preserve the variables in ctx *
  sexp_gc_preserve3(ctx, tmp1, tmp2, res);

  /* perform your computations *
  tmp1 = ...
  tmp2 = ...
  res = ...

  /* release before returning *
  sexp_gc_release3(ctx);

  return res;
}

If compiled with the Boehm GC, sexp_gc_varn just translates to the plain declaration, while sexp_gc_preserven and sexp_gc_releasen become noops.

When interacting with a garbage collection system from another language, or communicating between different Chibi managed heaps, you may want to manually ensure objects are preserved irrespective of any references to it from other objects in the same heap. This can be done with the sexp_preserve_object and sexp_release_object utilities.

sexp_preserve_object(ctx, obj)

Increment the absolute reference count for obj. So long as the reference count is above 0, obj will not be reclaimed even if there are no references to it from other object in the Chibi managed heap.

sexp_release_object(ctx, obj)

Decrement the absolute reference count for obj.

API Index

Type Predicates

The sexp represents different Scheme types with the use of tag bits for so-called "immediate" values, and a type tag for heap-allocated values. The following predicates can be used to distinguish these types. Note the predicates in C all end in "p". For efficiency they are implemented as macros, and so may evaluate their arguments multiple times.

Constants

The following shortcuts for various immediate values are available.

Accessors

The following macros provide access to the different components of the Scheme types. They do no type checking, essentially translating directly to pointer offsets, so you should be sure to use the above predicates to check types first. They only evaluate their arguments once.

Constructors

Constructors allocate memory and so must be passed a context argument. Any of these may fail and return the OOM exception object.

I/O

Utilities

Customizing

You can add your own types and primitives with the following functions.

See the C FFI for an easy way to automate adding bindings for C functions.

C FFI

The "chibi-ffi" script reads in the C function FFI definitions from an input file and outputs the appropriate C wrappers into a file with the same base name and the ".c" extension. You can then compile that C file into a shared library: 

chibi-ffi file.stub
cc -fPIC -shared file.c -lchibi-scheme

(or using whatever flags are appropriate to generate shared libs on your platform) and the generated .so file can be loaded directly with load, or portably using (include-shared "file") in a module definition (note that include-shared uses no suffix).

The goal of this interface is to make access to C types and functions easy, without requiring the user to write any C code. That means the stubber needs to be intelligent about various C calling conventions and idioms, such as return values passed in actual parameters. Writing C by hand is still possible, and several of the core modules provide C interfaces directly without using the stubber.

Includes and Initializations

Struct Interface

C structs can be bound as Scheme types with the define-c-struct form: 

(define-c-struct struct_name
  [predicate: predicate-name]
  [constructor: constructor-name]
  [finalizer: c_finalizer_name]
  (type c_field_name getter-name setter-name) ...)

struct_name should be the name of a C struct type. If provided, predicate-name is bound to a procedure which takes one object and returns #t iff the object is of type struct_name.

If provided, constructor-name is bound to a procedure of zero arguments which creates and returns a newly allocated instance of the type.

If a finalizer is provided, c_finalizer_name must be a C function which takes one argument, a pointer to the struct, and performs any cleanup or freeing of resources necessary.

The remaining slots are similar to the SRFI-9 syntax, except they are prefixed with a C type (described below). The c_field_name should be a field name of struct_name. getter-name will then be bound to a procedure of one argument, a

struct_name

type, which returns the given field. If provided, setter-name will be bound to a procedure of two arguments to mutate the given field.

The variants define-c-class and define-c-union take the same syntax but define types with the class and union keywords respectively. define-c-type just defines accessors to an opaque type without any specific struct-like keyword. 

;; Example: the struct addrinfo returned by getaddrinfo.

(c-system-include "netdb.h")

(define-c-struct addrinfo
  finalizer: freeaddrinfo
  predicate: address-info?
  (int              ai_family    address-info-family)
  (int              ai_socktype  address-info-socket-type)
  (int              ai_protocol  address-info-protocol)
  ((link sockaddr)  ai_addr      address-info-address)
  (size_t           ai_addrlen   address-info-address-length)
  ((link addrinfo)  ai_next      address-info-next))

Function and Constant Interface

C functions are defined with:

(define-c return-type name-spec (arg-type ...))

where name-space is either a symbol name, or a list of (scheme-name c_name). If just a symbol is used, the C name is generated automatically by replacing any dashes (-) in the Scheme name with underscores (_).

Each arg-type is a type suitable for input validation and conversion as discussed below. 

;; Example: define connect(2) in Scheme
(define-c int connect (int sockaddr int))

Constants can be defined with:

(define-c-const type name-space)

where name-space is the same form as in define-c. This defines a Scheme variable with the same value as the C constant. 

;; Example: define address family constants in Scheme
(define-c-const int (address-family/unix "AF_UNIX"))
(define-c-const int (address-family/inet "AF_INET"))

C Types

Basic Types

Integer Types

Float Types

String Types

Port Types

Struct Types

Struct types are by default just referred to by the bare struct_name from define-c-struct, and it is assumed you want a pointer to that type. To refer to the full struct, use the struct modifier, as in (struct struct-name).

Type modifiers

Any type may also be written as a list of modifiers followed by the type itself. The supported modifiers are:

Standard Modules

A number of SRFIs are provided in the default installation. Note that SRFIs 0, 6, 23, 46 and 62 are built into the default environment so there's no need to import them. SRFI 22 is available with the "-r" command-line option. This list includes popular SRFIs or SRFIs used in standard Chibi modules

Additional non-standard modules are put in the (chibi) module namespace.