SonicBoom

Boom IR

Boom IR is SonicBoom’s compiler IR. It is an MLIR dialect in the sonicboom::compiler::boom namespace; the TD definitions are core/compiler/include/sonicboom/compiler/ (BoomDialect.td, BoomTypes.td, BoomOps.td), the verifier is core/compiler/lib/BoomVerify.cpp, and the pass pipeline is core/compiler/lib/BoomPasses.cpp.

Every execution plan is emitted as Boom IR, optimized by the passes for its phase, and lowered back to the planner’s runtime graph. The planner’s boxed executor still runs planner::Graph; Boom IR is the compiler’s representation, not a second runtime.

plan site (model emitter)
      ↓ graphToBoom()                 conversion, one boom.graph per module
   Boom IR
      ↓ runBoomPasses(phase)          fusion / fold / scheduling passes
   Boom IR
      ↓ boomToGraph()                 conversion back
planner::Graph → executor → kernels

Module and graph structure

A module holds one boom.graph, which is the unit of verification and the anchor the pass pipeline runs on (the op is IsolatedFromAbove).

module {
  boom.graph @decode(tensors = [#boom.tensor_desc<...>, ...],
                     inputs = array<i64: 0, 1>,
                     outputs = array<i64: 900>,
                     dynamic_extent = 0) {
  ^bb0(%arg0: !boom.tensor<[1], "int64">, %arg1: !boom.tensor<[1], "int64">):
    %0 = boom.embedding(%arg0) {node_name = "embedding", node_id = 0 : i64,
                                result_ids = array<i64: 2>, node_attrs = [...]}
         : (!boom.tensor<[1], "int64">) -> !boom.tensor<[1, 2560], "float32">
    ...
  }
}

The conversion round-trips exactly: the same tensors, node ids, and attributes in the same order. Attribute order is part of the model identity (model_fingerprint hashes it), so node_attrs is an ordered array, never a name-keyed dictionary.

The lowering accepts a document only where the operator identities it declares are the ones this runtime implements: every node’s domain must be default, and a node’s explicit version — if it has one — must equal the version the graph declares for default. A node in another domain (the draft’s aten vocabulary, a custom op) may share a name with a default operator and mean something else, so it is refused rather than lowered by name. The Blade→planner and Blade→Boom routes apply the same check (sx::ir.h), so they accept and refuse the same documents.

Types

type meaning
!boom.tensor<[d0, d1, ...], "dtype"> ranked static tensor; dtype is one of the ten frozen sx::DType names
!boom.q8_1<[...]> activation in the q8_1 activation layout the MMQ path consumes
!boom.q3_k<...> !boom.q4_k<...> !boom.q5_k<...> !boom.q6_k<...> K-quant weights (256-element super-blocks)
!boom.q5_0<...> !boom.q8_0<...> legacy 32-element quant weights
!boom.kv<layer> layer’s KV cache handle (not a value)

Quantized types exist because a quantized format is a property of the value’s layout, while sx::DType is a frozen ten-name set. Shapes are the logical element shape; block geometry follows from the format, and the block-contiguous (fastest-moving) dimension must be a multiple of the block size — the verifier enforces it. !boom.kv<L> names the backend-owned per-layer cache; it has no shape, dtype or tensor-table entry.

Attribute definitions: #boom.attr<name, value> is one sx::Attribute, and #boom.tensor_desc<...> is one tensor-table entry.

The value slot is the attribute’s MLIR encoding: a non-empty ints/floats/ strings/bools value is an ArrayAttr whose elements carry the kind, and an empty one is the kind’s zero-length typed container (array<i64>, array<f64>, array<i1>, and a zero-element dense string tensor), because an empty ArrayAttr carries no element to infer the kind from. The kind is part of the model identity, so an empty array keeps it.

Operations

Containers and generation control

op purpose
boom.graph a whole planner graph and its tensor table
boom.program one greedy generation: boom.prefill, then an scf.while body of step/sample/emit/stop
boom.prefill call the named prefill graph over the prompt
boom.step one decode step (carries its own tensor table); operands are the loop-carried values and cache handles
boom.sample choose the next token (greedy argmax when no sampling attributes are set); stochastic marks a draw whose parameters the session supplies per step, allowed_list bounds the eligible token ids to a runtime list of that capacity
boom.stop_check EOS-set + max-token stop test
boom.emit append token(s) to the host-visible ring
boom.pos_inc advance the staged position
boom.draft speculation: propose up to max_drafts tokens
boom.verify speculation: decode graph dealt over rows, one argmax per row
boom.accept speculation: longest matching draft prefix + verifier token

Speculation replaces the loop body’s step/sample pair with draft -> verify -> accept; the plain step stays for the k == 0 case.

Node operations

There is one node op per planner::OpKind, with the mnemonic equal to op_kind_name(OpKind) — the correspondence is mechanical in both directions, not a table that can drift. All node ops share one attribute record:

attribute meaning
node_name, node_id, result_ids, node_attrs graph-node identity; enough to rebuild the GraphNodeDesc exactly
backend optional backend override (absent = the planner’s route)
slot, layer, kv_source_layer, ds, batch, dim, offset, rows, length, stride, scale the typed view of the attribute vocabulary the emitters and passes use

The verifier requires node_attrs and the typed view to agree, and enforces which of them each op may carry. Operands and results are variadic and typed; arity is checked with the shape/dtype/quant checks rather than fixed per op.

Consumers

A decode graph ends at the model’s logits. What reads them is declared on the graph rather than written as nodes by the emitter:

consumer the reader
"logits" the output, whole
"precap_logits" the output before its trailing monotone map (the logit softcap), which the reader applies itself
"argmax" the index of the largest element, reduced in the graph
"sample_select" the best candidates, selected in the graph

consumer_list = N adds that only a runtime list of at most N indices along the output’s innermost dimension is eligible (a grammar’s allowed set); it is valid with the two selecting consumers.

resolveConsumer(module) turns the declaration into the graph that reader needs, and runs before the phase’s passes on every path to the lowering. It runs the demand analysis (below) from the declared consumer, removes what the reader does not need, narrows what is read only on the index list to the list’s capacity (the weight matmul producing it is marked gather), and appends the argmax node. finishConsumer(module) runs after the passes and appends the sample_select node, which reads the finished logits buffer. A graph that still declares a consumer does not lower.

When an unresolved graph is lifted into a boom.program, the declaration moves to the program’s boom.sample (stochastic, allowed_list), which is then the reader the analysis starts from.

Demand

analyzeDemand(op) is a backward data-flow analysis over a module, a graph or a program. For every SSA value it computes how closely the value is observed — not at all, only ranked (order), or for its elements (exact) — and, per dimension, which indices are read: none, an interval, a runtime list of bounded length, or all. It runs together with MLIR’s dead-code analysis, so a reader on an unreachable path demands nothing, and it does not modify the IR.

Demand flows through a node according to what the op declares:

trait statement effect on demand
Elementwise element i of the result depends only on element i of the operands operands get the result’s region
Monotone elementwise, one operand, non-decreasing a ranking of the result is a ranking of the operand
RowIndependent slice r along the leading dimension depends only on slice r of the operands the row set is kept
NoMemoryEffect the op has no effect of its own an unread result demands nothing

A weight matmul keeps the row set of its activation and records which weight rows are read. An op that declares nothing is opaque: it is kept, and its operands are demanded whole. A trait is a statement about the operator’s semantics; declaring one on an op whose kernel does not have the property makes a rewrite unsound.

Stages

analyzeStage(op) (BoomStage.h) computes, for every SSA value under a module, a graph or a program, the inputs of a generation the value depends on:

source enters at
prompt a program’s arguments other than the cache handles
counter a loop-carried value that an iteration replaces
token the result of boom.sample, boom.draft, boom.accept
cache a cache handle

It is a sparse forward analysis on the same framework as the demand analysis, solved with the same reachability. A result depends on what the operands depend on; an op not declared free of memory effects depends on every source; boom.step is transparent. A standalone graph has no loop to read from, so its arguments are taken by name: token_id is {token}, pos is {prompt, counter}, a cache handle {cache}, anything else every source.

The set orders into a stage, the earliest moment the value can be computed:

stage sources computable
model none at load
request prompt once per generation
schedule + counter for every iteration, ahead of time
token + token once the iteration’s token is chosen
state + cache in the iteration, in order

recordStages(op) writes the result on each unit as stages: (tensor id, source bits) pairs in ascending id order, with prompt, counter, token and cache as bits 0 to 3. A stage is a fact about what a tensor means, so it stays true through any pass that keeps the tensor; recording again keeps, for each tensor, the sources both recordings name. The lift into a program carries the attribute to the step, and the lowering carries it into planner::Graph::facts (planner/facts.h), where a tensor with no entry reads as every source. The facts take no part in the model fingerprint or the plan.

Effects

The KV cache is a memory-effect resource, not a side channel: boom.attention reads the prefix and appends the current row, boom.flash_attention reads only (the batched append is deferred to boom.kv_append, which writes only), so MLIR can order them. The three cache-touching ops carry their !boom.kv<layer> handle as their last operand.

Pass pipeline

resolveConsumer(module) runs first (see Consumers), then runBoomPasses(module, phase, skip_kv_append_fusion) runs one phase’s passes over the module’s graph, verifying after every pass. A pass order is significant: later passes consume the nodes earlier ones produce. A pass failure fails the plan — the IR is never lowered as if the phase had run.

phase passes, in order
Decode QuantizeOncePass, RmsNormQuantizePass, GeluMulQuantizePass, SiluMulQuantizePass, FuseGeluMulPass, BiasIntoGemvPass, ResidualAddIntoGemvPass, FuseResidualEpiloguePass, FuseSiblingGemvPass, FuseGlueRegionsPass, FuseHeadsNormRopeCastPass, FuseKvAppendIntoPrepPass, BufferReusePass, AsyncPrefetchPass
Verify the decode list with FuseKvAppendIntoPrepPass skipped: a verify step appends one row per query position, which a single-row prep region cannot express
Prefill RmsNormMmqPass, FuseAddRmsNormMmqPass, FuseRmsNormAddPass, FoldScaleIntoGluePass, GeluMulMmqPass, SiluMulMmqPass, FuseSliceScalePass, FuseGeluMulPass, FuseSiluMulPass (standard numerics profile only), ScaleIntoDensePass, FuseHeadsNormRopeCastPass, PrefillDispatchPass, DenseLayoutPropagationPass, BufferReusePass, AsyncPrefetchPass
Training BufferReusePass only (semantic-neutral for a traced forward+backward graph)

What each pass does:

pass effect
QuantizeOncePass insert one quantize before the first quantized_matmul consumer of an activation; later matmuls share it
RmsNormQuantizePass rms_norm with one reader (quantize) becomes rms_norm_quantize
GeluMulQuantizePass / SiluMulQuantizePass gated gelu_fp16/silu × up followed by quantize becomes one fused op
FuseGeluMulPass the dense-projection form of the same gated multiply
FuseSiluMulPass the SwiGLU counterpart, prefill only and only under the standard numerics profile (a down projection with no mmq activation layout, e.g. Q6_K, leaves the chain for it)
BiasIntoGemvPass add_bias over a single-reader gemv becomes the projection’s bias_slot epilogue
ResidualAddIntoGemvPass add(proj, residual) becomes a two-input projection epilogue
FuseResidualEpiloguePass the same treatment where the residual add is produced by a norm (sandwich norm / PLE post-norm)
FuseSiblingGemvPass group projections that share an activation
FuseGlueRegionsPass fold a maximal elementwise chain into one boom.fused_region (one generated kernel)
FuseHeadsNormRopeCastPass attention prep chains (norm → rope [→ cast]) in decode and prefill
FuseKvAppendIntoPrepPass fold the K/V append into the attention prep region
RmsNormMmqPass rms_norm + batched q8_1 quantize for the MMQ prefill path
FuseAddRmsNormMmqPass add → norm → quantize as one batched region
FuseRmsNormAddPass norm + add chains the MMQ fold did not take
FoldScaleIntoGluePass / ScaleIntoDensePass a scalar scale after a single-reader chain becomes an epilogue_scale attribute (node position unchanged)
GeluMulMmqPass / SiluMulMmqPass gated activation + batched quantize for MMQ
FuseSliceScalePass strided slice + scale as one elementwise region
PrefillDispatchPass rewrite a batched q8 matmul to the generated int8 tensor-core kernel, per SONICBOOM_MMQ_TC (off default, auto, on) and the SONICBOOM_MMQ_TC_MAX_MB copy budget
DenseLayoutPropagationPass mark the projections with their dense layout (attribute only)
BufferReusePass mark single-use elementwise results the executor may write in place
AsyncPrefetchPass insert prefetch markers when model weights exceed SONICBOOM_VRAM_THRESHOLD_MB (SONICBOOM_ASYNC_PREFETCH=on)
PruneByDemandPass the optional part of resolveConsumer: bypass a monotone map whose result is only ranked, and remove effect-free nodes nothing reads

Ablation

SONICBOOM_IR_PASS_DISABLE=PassName[,PassName...] skips those passes for the run. Names are the pass class names above. A phase that does not contain a named pass ignores it; an unrecognized name is reported on stderr with the valid names and disables nothing. A disabled pass can also remove a later pass’s fusion (the passes run in sequence), so a plan delta measures that pass and the folds it feeds together.

PruneByDemandPass names the demand-driven pruning inside resolveConsumer. With it disabled an argmax consumer reads the softcapped logits; the index-list narrowing is not part of the switch, because the list is what the consumer asked for rather than an optimization of it.

Adding a pass

  1. Write the class in core/compiler/lib/BoomPasses.cpp as an MLIR pass over boom::GraphOp. Read the graph through the pass-side view (graphView), which owns the tensor-id ↔︎ SSA-value relation and the tensor table; edit both only through it.
  2. Create replacement nodes with buildNode, which writes the identity attributes, the authoritative attribute record, and the typed view the verifier checks. Do not construct an op by hand.
  3. Add the pass to the phase’s list in addPasses, in the position its preconditions require, and add a {"YourPassPass", &DisabledPasses::flag} row to kPassNames so ablation can name it.
  4. Add a TEST to the conversion/verifier suite, and a plan golden if the pass changes the node set.
  5. Ensure boomToGraph still round-trips: a pass may only produce GraphNodeDescs the planner can rebuild (same tensor table invariants, same attribute kinds).

Verification rules

The verifier (BoomVerify.cpp) rejects, with the offending node named: