SonicBoom

Architecture

Four layers, with dependencies pointing strictly downward:

Layer 3    C ABI / Guile / other bindings
Layer 2    SonicBoom stable runtime interface
Layer 1    native-torch implementation + adapter
Layer 0    CPU / CUDA

libsonicboom.so is the published core library. native-torch and MLIR are statically linked into it. native-torch’s operator sources are generated at build time by SonicCross, a torchgen written in Guile. The stable binary-compatibility boundary is the C ABI (capi/); the C++ Layer 2 interface is source-level only. Layer 2 public headers expose no ATen/c10/native-torch type, and generated code and the JIT live behind Layer 2.

Two routes share this core:

Why another runtime?

For people who need to know what they are running. The core is hand-written C++/CUDA. It is GPL-3.0-or-later, and the boundary is a plain C ABI that you can audit, pin and link against. It has no Python dependency chain to vet, and nothing is downloaded at run time. Python is not needed to build it either: PyTorch’s Python code generator, torchgen, is replaced by SonicCross, written in Guile.

Fast because of the runtime, not only the kernels. Each decode step runs as a single CUDA Graph replay. The KV cache stays on the device. Planner passes fuse norm, activation and quantization. Prefix-cache reuse is exact: a continuation is token-identical to a full recompute.

More than an LLM runner. SonicBoom is a general tensor framework: a better PyTorch, with no Python in it. You write models in Scheme, as eagerly as in PyTorch, and they are traced and compiled into graphs. Production models are defined in the C++ core and served from any language. LLM inference is the first benchmark, not the limit of the project.

A compiler stack, not a kernel collection

llama.cpp is fast because of hand-tuned kernels, and in that design every new fusion or model-specific optimization means writing another kernel by hand. SonicBoom chooses an implementation for each operation from three tiers, in this order of preference:

Tier What it does Status
1. MLIR compiler Model graphs become Boom IR, which is rewritten and lowered to MLIR. GPU kernels are generated through linalg → gpu → nvvm and JIT-compiled by shape signature; S-Expr graphs still run on the CPU JIT, and ONNX-imported ResNet-18 matches the ONNX reference. Shipped: fused glue, attention and a bf16 dense GEMM are generated on CUDA.
2. Graph planner Rewrites the whole model before it runs: quantize-once, norm→quantize and activation→quantize fusion, dead-computation elimination, layout selection and CUDA Graph capture of the full decode step. Shipped: this tier produces the decode lead measured in the README.
3. Hand-written kernels Used where the other tiers are not yet faster, for example the ported ggml MMQ/MMVQ kernels and the boom_fattn prefill attention kernel. Shipped

What follows from this design:

MLIR migration: a staged plan

MLIR is linked into libsonicboom.so and carries the production graphs: model plans become Boom IR, the compiler’s rewrite passes run on it, and CUDA kernels are generated from it through linalg → gpu → nvvm and JIT-compiled. S-Expr documents still lower to arith/func/linalg and run on the CPU through the LLVM JIT, which is how ONNX-imported ResNet-18 runs. The migration is staged so that each step is independently verifiable, and so that a generated kernel can always be checked against the hand-written one it replaces.

stage what lands status
0. Lowering S-Expr → arith/func/linalg, executed on CPU through the LLVM JIT; ONNX ResNet-18 matches the ONNX reference shipped
1. One GPU glue kernel A single fused region — norm + quantize — lowered linalg → gpu → nvvm → cubin, JIT-compiled at load and cached by shape signature shipped
2. The glue set The rest of the between-kernel glue: activation→quantize, residual add, RoPE shipped
2b. Attention and dense GEMM Generated split-KV decode attention and a bf16 dense GEMM with an mma m16n8k16 core, selected per shape class by the runtime tuner; prefill attention is served by the hand-written boom_fattn kernel shipped; generated decode attention is token-identical to the hand-written pair and is the default
3. GPU-independent targeting The same lowering for a GPU the binary was not built for (compute capability discovered at run time), and for a second vendor through SPIR-V not started
4. Beyond LLM glue Whole-model lowering for the research route’s traced graphs not started

Ground rules that hold at every stage: generated code and the JIT live behind Layer 2 (no MLIR, LLVM, CUDA or native-torch type crosses the C ABI); the hand-written kernel remains the numerical and performance reference; and the gate is the canonical benchmark plus the existing test suite, because “it compiles” and “it is faster” are not the same claim. The pinned LLVM is 23 — a different major is a warning, not an error (see CMakeLists.txt).

Scheme: write it eagerly, run it compiled

The research route keeps PyTorch’s eager style without Python’s limits. It is built on two Scheme features that Python lacks: hygienic macros and delimited continuations. It currently ships as a pure-Guile research prototype in bindings/guile/scheme/sonicboom/trace.

Next step: connect the tracer to the core, so that traced graphs are compiled once through the C ABI and run on the same planner and backends as the production route.

How much of this is ggml?

SonicBoom ports a small set of kernels from ggml/llama.cpp under the MIT license. They live in ggml_derived/ directories, with the details in NOTICE and LICENSES/ggml-MIT.txt.

About 10% of the engine’s code is identical to llama.cpp (commit 43fe9c642), counted as lines that sit in runs of four or more lines matching upstream (tools/provenance/ggml-share.py). Almost all of it is the quantized matrix-multiplication kernels in core/src/quant/ggml_derived/, which holds only files that are substantially upstream; files elsewhere that derive from llama.cpp in part carry its MIT notice in their header.

Ported: quantized matrix multiply (MMVQ for decode, MMQ for prefill), q8_1 activation quantization, the legacy Q5_0/Q8_0 block formats, and the BPE tokenizers (SPM-style for Gemma 4, byte-level with the qwen2 pre-tokenizer, plus the Unicode classes and byte maps they split on).

SonicBoom’s own: everything else. That includes the planner and Boom IR compiler passes, operator fusion, the generated GPU kernels (glue, attention and bf16 dense GEMM), per-GPU runtime autotuning, CUDA Graph capture and replay, the device-resident KV cache, split-KV flash decoding, the boom_fattn tiled flash attention for prefill, chunked prefill, prefix-cache reuse, batch-invariant dense GEMM, and speculative decoding (long-match drafter, batched verify, KV rollback).

Decode uses the same quantized GEMV kernels as llama.cpp and is still faster on the d64 family (see the README’s numbers). That difference comes from the runtime, not from the kernels.

Where this is going

One core and one C ABI, with a growing set of front ends:

 Scheme (eager, traced)  ─┐
 ONNX / PyTorch export   ─┼─→  S-Expr graph  ─→  planner  ─→  MLIR / kernels  ─→  CPU · CUDA · …
 GGUF models (C++ core)  ─┘

A model written once, by tracing, by import, or as a native definition, gets every planner pass and every back end SonicBoom adds afterwards. That is the difference between a fast LLM runner and a framework, and it is the direction SonicBoom is built for.