SonicBoom

sonicboom/inference

The (sonicboom inference) module binds the minimal LLM C ABI (capi.h sb_llm_*) through (sonicboom ffi), which is the only path Guile uses to reach libsonicboom.so. It exposes load-model / make-session / generate — a decode inference loop over the GGUF built-in tokenizer (greedy by default, seeded sampling when temperature > 0).

Guile → (sonicboom inference) → (sonicboom ffi) → libsonicboom.so

Handles are foreign pointers. Freeing is explicit (free-model!/free-session!) so a test owns its lifetime deterministically; the session holds a C++ shared_ptr to the model so a freed model handle does not dangle a live session. Three entry points: - generate string -> string (tokenize + decode + detokenize) - generate-tokens token-ids -> token-ids (the raw token-ID loop) - generate-stream tokens -> tokens, calling a procedure per token with the text that became available with it

load-model

procedure

(load-model path)

Load a model from a GGUF file.

Arguments:

Returns: an opaque model handle.

PyTorch: none

Note: GGUF is SonicBoom’s model-file format; PyTorch core has no GGUF loader.

free-model!

procedure

(free-model! model)

Release a model handle.

A session keeps the model it was created from alive, so freeing the handle does not invalidate a live session.

Arguments:

Returns: unspecified.

PyTorch: none

model-arch

procedure

(model-arch model)

The model’s architecture name, as recorded in its GGUF metadata.

Arguments:

Returns: a string such as “gemma4” or “qwen2”.

PyTorch: none

Note: The name comes from the file rather than the model family being guessed from the path.

model-eos

procedure

(model-eos model)

The model’s end-of-sequence token ids.

Arguments:

Returns: a list of token ids; empty when the file declares none.

PyTorch: none

Note: Generation stops on these ids, per model rather than per family.

make-session

procedure

(make-session model n-ctx)

Create a session over a model.

A session holds the token history and the context window of one conversation; it keeps the model alive.

Arguments:

Returns: an opaque session handle.

PyTorch: none

free-session!

procedure

(free-session! session)

Release a session handle.

Arguments:

Returns: unspecified.

PyTorch: none

set-grammar!

procedure

(set-grammar! session gbnf)

Constrain a session’s next turn to a grammar.

The grammar restarts at every prefill!, so install it before prefill! to mask the turn’s first token as well. Every sampled token must keep the generated text acceptable to the grammar, and the model’s end-of-sequence ids end the output where the grammar can end.

Arguments:

Returns: unspecified.

PyTorch: none

Note: Constrained decoding is not part of PyTorch’s generation loop.

set-json-schema!

procedure

(set-json-schema! session schema-json)

Constrain a session’s next turn to a JSON Schema.

The schema is compiled to a grammar; an unsupported keyword is rejected here by name rather than ignored.

Arguments:

Returns: unspecified.

PyTorch: none

clear-grammar!

procedure

(clear-grammar! session)

Remove a session’s grammar, so the next turn decodes unconstrained.

Arguments:

Returns: unspecified.

PyTorch: none

constraint-state

procedure

(constraint-state session)

The state of a session’s constrained decoding.

Arguments:

Returns: two values: whether a grammar is installed, and whether the text sampled so far is already a complete value of it.

PyTorch: none

json-schema->gbnf

procedure

(json-schema->gbnf schema-json #:key (prefix "") (suffix ""))

Compile a JSON Schema to GBNF grammar text.

Arguments:

Returns: the grammar text.

Example:

(use-modules (sonicboom inference))
(string? (json-schema->gbnf "{\"type\": \"string\"}"))

=> #t

PyTorch: none

Note: The prefix and suffix are how a tool-call envelope is built around a function’s arguments schema.

prefill!

procedure

(prefill! session tokens)

Set a session’s prompt to a list of token ids.

Any tokens the session already holds that are a prefix of the new prompt are reused rather than recomputed.

Arguments:

Returns: the number of tokens reused from the previous prompt.

PyTorch: none

tokenize

procedure

(tokenize model text #:key (special? #f))

Encode text to a list of token ids.

A beginning-of-sequence token is prepended, matching the command-line tokenizer. Special-token parsing is off by default: it turns text into control structure, so use it only on text the caller generated itself, never on a user’s message.

Arguments:

Returns: a list of token ids.

PyTorch: none

tokenize-segments

procedure

(tokenize-segments model segments)

Encode a rendered chat prompt given as segments.

Segments are (kind . text) pairs with kind ’text or ’special, as produced by a chat template. A ’special segment is tokenized with special parsing, so a turn marker spelled out in it becomes the vocabulary’s control token; a ’text segment is tokenized as plain text, so a marker that came from a user’s message stays text. The beginning-of-sequence token is added once, by the first segment.

Arguments:

Returns: a list of token ids.

PyTorch: none

detokenize

procedure

(detokenize model tokens)

Decode a list of token ids back to text.

Arguments:

Returns: a UTF-8 string.

PyTorch: none

generate

procedure

(generate model session prompt max-tokens #:key (temperature 0.0) (top-k 0) (top-p 1.0) (min-p 0.0) (seed 0) (repeat-penalty 1.0) (repeat-last-n 64) (presence-penalty 0.0) (frequency-penalty 0.0))

Generate text from a prompt.

Tokenizes the prompt, prefills the session, decodes up to max-tokens tokens and detokenizes them.

Arguments:

Returns: the generated text.

PyTorch: none

generate-tokens

procedure

(generate-tokens session prompt max-tokens #:optional (prefill? #t) #:key (temperature 0.0) (top-k 0) (top-p 1.0) (min-p 0.0) (seed 0) (repeat-penalty 1.0) (repeat-last-n 64) (presence-penalty 0.0) (frequency-penalty 0.0))

Generate token ids from a prompt given as token ids.

Arguments:

Returns: the list of produced token ids.

PyTorch: none

generate-stream

procedure

(generate-stream model session prompt max-tokens proc #:optional (prefill? #t) #:key (temperature 0.0) (top-k 0) (top-p 1.0) (min-p 0.0) (seed 0) (repeat-penalty 1.0) (repeat-last-n 64) (presence-penalty 0.0) (frequency-penalty 0.0))

Generate tokens while delivering the text as it becomes available.

Calls (proc token chunk) once per produced token, in order, on the calling thread. Chunk is the text that became available with that token and may be empty; chunks never split a UTF-8 character. A final call with token #f flushes any bytes held back at the end. Concatenating every chunk in order gives exactly the string generate returns for the same prompt.

Arguments:

Returns: the list of produced token ids.

PyTorch: none

Note: An exception raised by proc stops the decode and is re-thrown by generate-stream.

make-sampling-bv

procedure

(make-sampling-bv max-tokens #:key (temperature 0.0) (top-k 0) (top-p 1.0) (min-p 0.0) (seed 0) (repeat-penalty 1.0) (repeat-last-n 64) (presence-penalty 0.0) (frequency-penalty 0.0))

Build the native sampling-parameters structure for a decode call.

Temperature 0 selects the greedy path; top-k 0, top-p 1, min-p 0, repeat-penalty 1 and the two penalties 0 are neutral (off). A positive temperature samples deterministically from seed.

Arguments:

Returns: a bytevector holding the native structure.

Example:

(use-modules (sonicboom inference))
(bytevector-length (make-sampling-bv 8))

=> 56

PyTorch: none

Note: The bytevector is an internal representation; pass it through the generation procedures rather than decoding it.

numerics-profile

procedure

(numerics-profile)

The numerics profile this process is using.

The profile is the process-wide numeric contract: strict keeps every historical reduction order, standard allows the profile’s substitutions.

Returns: ’strict or ’standard.

Example:

(use-modules (sonicboom inference))
(memq (numerics-profile) '(strict standard))

=> (standard)

PyTorch: none

Note: The profile is chosen before the first model or session exists; see set-numerics-profile!.

set-numerics-profile!

procedure

(set-numerics-profile! profile)

Request a numerics profile for this process.

The first model or session latches the profile, so call this before either. Requesting the effective profile again is a no-op success; requesting the other one after latching signals a &sonicboom-error.

Arguments:

Returns: the effective profile after the request.

Example:

(use-modules (sonicboom inference))
(let ((p (numerics-profile))) (eq? p (set-numerics-profile! p)))

=> #t

PyTorch: none