sonicboom/inference
The (sonicboom inference) module binds the minimal LLM C ABI (capi.h sb_llm_*) through (sonicboom ffi), which is the only path Guile uses to reach libsonicboom.so. It exposes load-model / make-session / generate — a decode inference loop over the GGUF built-in tokenizer (greedy by default, seeded sampling when temperature > 0).
Guile → (sonicboom inference) → (sonicboom ffi) → libsonicboom.so
Handles are foreign pointers. Freeing is explicit (free-model!/free-session!) so a test owns its lifetime deterministically; the session holds a C++ shared_ptr to the model so a freed model handle does not dangle a live session. Three entry points: - generate string -> string (tokenize + decode + detokenize) - generate-tokens token-ids -> token-ids (the raw token-ID loop) - generate-stream tokens -> tokens, calling a procedure per token with the text that became available with it
load-model
procedure
Load a model from a GGUF file.
Arguments:
path: the path of the GGUF file.
Returns: an opaque model handle.
PyTorch: none
Note: GGUF is SonicBoom’s model-file format; PyTorch core has no GGUF loader.
free-model!
procedure
Release a model handle.
A session keeps the model it was created from alive, so freeing the handle does not invalidate a live session.
Arguments:
model: a model handle from load-model.
Returns: unspecified.
PyTorch: none
model-arch
procedure
The model’s architecture name, as recorded in its GGUF metadata.
Arguments:
model: a model handle.
Returns: a string such as “gemma4” or “qwen2”.
PyTorch: none
Note: The name comes from the file rather than the model family being guessed from the path.
model-eos
procedure
The model’s end-of-sequence token ids.
Arguments:
model: a model handle.
Returns: a list of token ids; empty when the file declares none.
PyTorch: none
Note: Generation stops on these ids, per model rather than per family.
make-session
procedure
Create a session over a model.
A session holds the token history and the context window of one conversation; it keeps the model alive.
Arguments:
model: a model handle.n-ctx: the context-window size in tokens.
Returns: an opaque session handle.
PyTorch: none
free-session!
procedure
Release a session handle.
Arguments:
session: a session handle from make-session.
Returns: unspecified.
PyTorch: none
set-grammar!
procedure
Constrain a session’s next turn to a grammar.
The grammar restarts at every prefill!, so install it before prefill! to mask the turn’s first token as well. Every sampled token must keep the generated text acceptable to the grammar, and the model’s end-of-sequence ids end the output where the grammar can end.
Arguments:
session: a session handle.gbnf: a grammar in the GBNF dialect.
Returns: unspecified.
PyTorch: none
Note: Constrained decoding is not part of PyTorch’s generation loop.
set-json-schema!
procedure
Constrain a session’s next turn to a JSON Schema.
The schema is compiled to a grammar; an unsupported keyword is rejected here by name rather than ignored.
Arguments:
session: a session handle.schema-json: the schema as a JSON string.
Returns: unspecified.
PyTorch: none
clear-grammar!
procedure
Remove a session’s grammar, so the next turn decodes unconstrained.
Arguments:
session: a session handle.
Returns: unspecified.
PyTorch: none
constraint-state
procedure
The state of a session’s constrained decoding.
Arguments:
session: a session handle.
Returns: two values: whether a grammar is installed, and whether the text sampled so far is already a complete value of it.
PyTorch: none
json-schema->gbnf
procedure
Compile a JSON Schema to GBNF grammar text.
Arguments:
schema-json: the schema as a JSON string.prefix: literal text that must appear before the schema value; empty bysuffix: literal text that must appear after the schema value; empty by
Returns: the grammar text.
Example:
(use-modules (sonicboom inference))
(string? (json-schema->gbnf "{\"type\": \"string\"}"))=> #t
PyTorch: none
Note: The prefix and suffix are how a tool-call envelope is built around a function’s arguments schema.
prefill!
procedure
Set a session’s prompt to a list of token ids.
Any tokens the session already holds that are a prefix of the new prompt are reused rather than recomputed.
Arguments:
session: a session handle.tokens: the prompt as a list of token ids.
Returns: the number of tokens reused from the previous prompt.
PyTorch: none
tokenize
procedure
Encode text to a list of token ids.
A beginning-of-sequence token is prepended, matching the command-line tokenizer. Special-token parsing is off by default: it turns text into control structure, so use it only on text the caller generated itself, never on a user’s message.
Arguments:
model: a model handle.text: the UTF-8 text to encode.special?: when true, control pieces of the vocabulary that occur verbatim
Returns: a list of token ids.
PyTorch: none
tokenize-segments
procedure
Encode a rendered chat prompt given as segments.
Segments are (kind . text) pairs with kind ’text or ’special, as produced by a chat template. A ’special segment is tokenized with special parsing, so a turn marker spelled out in it becomes the vocabulary’s control token; a ’text segment is tokenized as plain text, so a marker that came from a user’s message stays text. The beginning-of-sequence token is added once, by the first segment.
Arguments:
model: a model handle.segments: the prompt segments.
Returns: a list of token ids.
PyTorch: none
detokenize
procedure
Decode a list of token ids back to text.
Arguments:
model: a model handle.tokens: a list of token ids.
Returns: a UTF-8 string.
PyTorch: none
generate
procedure
Generate text from a prompt.
Tokenizes the prompt, prefills the session, decodes up to max-tokens tokens and detokenizes them.
Arguments:
model: a model handle.session: a session handle over that model.prompt: the UTF-8 prompt text.max-tokens: the maximum number of tokens to produce.presence-penalty, frequency-penalty: sampling parameters, as in
Returns: the generated text.
PyTorch: none
generate-tokens
procedure
Generate token ids from a prompt given as token ids.
Arguments:
session: a session handle.prompt: the prompt as a list of token ids.max-tokens: the maximum number of tokens to produce.prefill?: when false, decode from the prompt the session already holds.presence-penalty, frequency-penalty: sampling parameters, as in
Returns: the list of produced token ids.
PyTorch: none
generate-stream
procedure
Generate tokens while delivering the text as it becomes available.
Calls (proc token chunk) once per produced token, in order, on the calling thread. Chunk is the text that became available with that token and may be empty; chunks never split a UTF-8 character. A final call with token #f flushes any bytes held back at the end. Concatenating every chunk in order gives exactly the string generate returns for the same prompt.
Arguments:
model: a model handle.session: a session handle.prompt: the prompt as a list of token ids.max-tokens: the maximum number of tokens to produce.proc: the per-token procedure (proc token chunk).prefill?: when false, decode from the prompt the session already holds.presence-penalty, frequency-penalty: sampling parameters, as in
Returns: the list of produced token ids.
PyTorch: none
Note: An exception raised by proc stops the decode and is re-thrown by generate-stream.
make-sampling-bv
procedure
Build the native sampling-parameters structure for a decode call.
Temperature 0 selects the greedy path; top-k 0, top-p 1, min-p 0, repeat-penalty 1 and the two penalties 0 are neutral (off). A positive temperature samples deterministically from seed.
Arguments:
max-tokens: the maximum number of tokens to produce.temperature: the sampling temperature.top-k: keep the k most likely tokens; 0 disables.top-p: nucleus threshold; 1 disables.min-p: minimum relative probability; 0 disables.seed: the sampling seed.repeat-penalty: penalty for repeated tokens; 1 disables.repeat-last-n: how many recent tokens the repeat penalty looks at.presence-penalty: additive presence penalty; 0 disables.frequency-penalty: additive frequency penalty; 0 disables.
Returns: a bytevector holding the native structure.
Example:
(use-modules (sonicboom inference))
(bytevector-length (make-sampling-bv 8))=> 56
PyTorch: none
Note: The bytevector is an internal representation; pass it through the generation procedures rather than decoding it.
numerics-profile
procedure
The numerics profile this process is using.
The profile is the process-wide numeric contract: strict keeps every historical reduction order, standard allows the profile’s substitutions.
Returns: ’strict or ’standard.
Example:
(use-modules (sonicboom inference))
(memq (numerics-profile) '(strict standard))=> (standard)
PyTorch: none
Note: The profile is chosen before the first model or session exists; see set-numerics-profile!.
set-numerics-profile!
procedure
Request a numerics profile for this process.
The first model or session latches the profile, so call this before either. Requesting the effective profile again is a no-op success; requesting the other one after latching signals a &sonicboom-error.
Arguments:
profile: ’strict or ’standard.
Returns: the effective profile after the request.
Example:
(use-modules (sonicboom inference))
(let ((p (numerics-profile))) (eq? p (set-numerics-profile! p)))=> #t
PyTorch: none