Boom IR
Boom IR is SonicBoom’s compiler IR. It is an MLIR dialect in the sonicboom::compiler::boom namespace; the TD definitions are core/compiler/include/sonicboom/compiler/ (BoomDialect.td, BoomTypes.td, BoomOps.td), the verifier is core/compiler/lib/BoomVerify.cpp, and the pass pipeline is core/compiler/lib/BoomPasses.cpp.
Every execution plan is emitted as Boom IR, optimized by the passes for its phase, and lowered back to the planner’s runtime graph. The planner’s boxed executor still runs planner::Graph; Boom IR is the compiler’s representation, not a second runtime.
plan site (model emitter)
↓ graphToBoom() conversion, one boom.graph per module
Boom IR
↓ runBoomPasses(phase) fusion / fold / scheduling passes
Boom IR
↓ boomToGraph() conversion back
planner::Graph → executor → kernels
Module and graph structure
A module holds one boom.graph, which is the unit of verification and the anchor the pass pipeline runs on (the op is IsolatedFromAbove).
module {
boom.graph @decode(tensors = [#boom.tensor_desc<...>, ...],
inputs = array<i64: 0, 1>,
outputs = array<i64: 900>,
dynamic_extent = 0) {
^bb0(%arg0: !boom.tensor<[1], "int64">, %arg1: !boom.tensor<[1], "int64">):
%0 = boom.embedding(%arg0) {node_name = "embedding", node_id = 0 : i64,
result_ids = array<i64: 2>, node_attrs = [...]}
: (!boom.tensor<[1], "int64">) -> !boom.tensor<[1, 2560], "float32">
...
}
}
- Tensor table.
tensorscarries everyplanner::TensorDescverbatim (#boom.tensor_desc<id, shape, dtype, byte_size, dynamic_base_bytes, is_input, is_output, is_constant, is_external, name>). It is authoritative for the fields a type cannot carry: byte size (whose dynamic entries are sized atdynamic_extent) and the four flags. A value’s type must agree with its entry. - Body. One block, an ordered node list, no terminator. The block arguments are the graph’s free variables — every tensor the graph references but no node produces (feedable inputs, model-bound weights and constants, externals) — in ascending tensor-id order. Weights are block arguments too, so a node can read its K-quant weight as an operand.
- Results. A node’s results carry their tensor ids in
result_ids, in result order. Node results plus block arguments give a total SSA → tensor-id mapping, andinputs/outputsname the feedable and returned ids. The graph’s outputs are ids, not a terminator.
The conversion round-trips exactly: the same tensors, node ids, and attributes in the same order. Attribute order is part of the model identity (model_fingerprint hashes it), so node_attrs is an ordered array, never a name-keyed dictionary.
The lowering accepts a document only where the operator identities it declares are the ones this runtime implements: every node’s domain must be default, and a node’s explicit version — if it has one — must equal the version the graph declares for default. A node in another domain (the draft’s aten vocabulary, a custom op) may share a name with a default operator and mean something else, so it is refused rather than lowered by name. The Blade→planner and Blade→Boom routes apply the same check (sx::ir.h), so they accept and refuse the same documents.
Types
| type | meaning |
|---|---|
!boom.tensor<[d0, d1, ...], "dtype"> |
ranked static tensor; dtype is one of the ten frozen sx::DType names |
!boom.q8_1<[...]> |
activation in the q8_1 activation layout the MMQ path consumes |
!boom.q3_k<...> !boom.q4_k<...> !boom.q5_k<...> !boom.q6_k<...> |
K-quant weights (256-element super-blocks) |
!boom.q5_0<...> !boom.q8_0<...> |
legacy 32-element quant weights |
!boom.kv<layer> |
layer’s KV cache handle (not a value) |
Quantized types exist because a quantized format is a property of the value’s layout, while sx::DType is a frozen ten-name set. Shapes are the logical element shape; block geometry follows from the format, and the block-contiguous (fastest-moving) dimension must be a multiple of the block size — the verifier enforces it. !boom.kv<L> names the backend-owned per-layer cache; it has no shape, dtype or tensor-table entry.
Attribute definitions: #boom.attr<name, value> is one sx::Attribute, and #boom.tensor_desc<...> is one tensor-table entry.
The value slot is the attribute’s MLIR encoding: a non-empty ints/floats/ strings/bools value is an ArrayAttr whose elements carry the kind, and an empty one is the kind’s zero-length typed container (array<i64>, array<f64>, array<i1>, and a zero-element dense string tensor), because an empty ArrayAttr carries no element to infer the kind from. The kind is part of the model identity, so an empty array keeps it.
Operations
Containers and generation control
| op | purpose |
|---|---|
boom.graph |
a whole planner graph and its tensor table |
boom.program |
one greedy generation: boom.prefill, then an scf.while body of step/sample/emit/stop |
boom.prefill |
call the named prefill graph over the prompt |
boom.step |
one decode step (carries its own tensor table); operands are the loop-carried values and cache handles |
boom.sample |
choose the next token (greedy argmax when no sampling attributes are set); stochastic marks a draw whose parameters the session supplies per step, allowed_list bounds the eligible token ids to a runtime list of that capacity |
boom.stop_check |
EOS-set + max-token stop test |
boom.emit |
append token(s) to the host-visible ring |
boom.pos_inc |
advance the staged position |
boom.draft |
speculation: propose up to max_drafts tokens |
boom.verify |
speculation: decode graph dealt over rows, one argmax per row |
boom.accept |
speculation: longest matching draft prefix + verifier token |
Speculation replaces the loop body’s step/sample pair with draft -> verify -> accept; the plain step stays for the k == 0 case.
Node operations
There is one node op per planner::OpKind, with the mnemonic equal to op_kind_name(OpKind) — the correspondence is mechanical in both directions, not a table that can drift. All node ops share one attribute record:
| attribute | meaning |
|---|---|
node_name, node_id, result_ids, node_attrs |
graph-node identity; enough to rebuild the GraphNodeDesc exactly |
backend |
optional backend override (absent = the planner’s route) |
slot, layer, kv_source_layer, ds, batch, dim, offset, rows, length, stride, scale |
the typed view of the attribute vocabulary the emitters and passes use |
The verifier requires node_attrs and the typed view to agree, and enforces which of them each op may carry. Operands and results are variadic and typed; arity is checked with the shape/dtype/quant checks rather than fixed per op.
- Frozen S-Expr v0.1 CNN operators (tier 2, lowered to linalg):
boom.conv,boom.relu,boom.add,boom.max_pool,boom.reduce_mean,boom.reshape,boom.gemm;boom.softmaxroutes to native-torch;boom.whole_graphis a task-granularity marker. - Transformer operators:
boom.quantized_matmul,boom.quantize,boom.quantized_matmul_q8,boom.matmul_q8_tc,boom.rms_norm_quantize,boom.gelu_mul_quantize,boom.silu_mul_quantize,boom.rms_norm,boom.rms_norm_heads,boom.gelu_fp16,boom.silu,boom.rope,boom.attention,boom.gqa_broadcast,boom.embedding,boom.matvec_f32,boom.matvec_bf16,boom.add_bias,boom.mul,boom.scale,boom.cast_fp16,boom.softcap,boom.argmax,boom.sample_select,boom.slice,boom.flash_attention,boom.kv_append. - Scheduling:
boom.async_weight_prefetch,boom.sync_prefetchare layer-boundary markers for the runtime’s prefetch stream, andboom.fused_regionis a compiler-generated glue region: it takes over the op it replaces (same id, attributes, operands, results, backend) and itsfused_regionnode attribute selects the generated kernel at dispatch. - Training:
boom.adamw,boom.clip_grad_norm,boom.matmul_bwd,boom.softmax_bwd,boom.cross_entropy_bwd,boom.rmsnorm_bwd,boom.rope_bwd,boom.silu_mul_bwd,boom.embedding_bwd, plus the forward primitivesboom.matmul,boom.bmm,boom.permute,boom.index_select,boom.reduce_sum,boom.silu_mul,boom.cross_entropy.
Consumers
A decode graph ends at the model’s logits. What reads them is declared on the graph rather than written as nodes by the emitter:
consumer |
the reader |
|---|---|
"logits" |
the output, whole |
"precap_logits" |
the output before its trailing monotone map (the logit softcap), which the reader applies itself |
"argmax" |
the index of the largest element, reduced in the graph |
"sample_select" |
the best candidates, selected in the graph |
consumer_list = N adds that only a runtime list of at most N indices along the output’s innermost dimension is eligible (a grammar’s allowed set); it is valid with the two selecting consumers.
resolveConsumer(module) turns the declaration into the graph that reader needs, and runs before the phase’s passes on every path to the lowering. It runs the demand analysis (below) from the declared consumer, removes what the reader does not need, narrows what is read only on the index list to the list’s capacity (the weight matmul producing it is marked gather), and appends the argmax node. finishConsumer(module) runs after the passes and appends the sample_select node, which reads the finished logits buffer. A graph that still declares a consumer does not lower.
When an unresolved graph is lifted into a boom.program, the declaration moves to the program’s boom.sample (stochastic, allowed_list), which is then the reader the analysis starts from.
Demand
analyzeDemand(op) is a backward data-flow analysis over a module, a graph or a program. For every SSA value it computes how closely the value is observed — not at all, only ranked (order), or for its elements (exact) — and, per dimension, which indices are read: none, an interval, a runtime list of bounded length, or all. It runs together with MLIR’s dead-code analysis, so a reader on an unreachable path demands nothing, and it does not modify the IR.
Demand flows through a node according to what the op declares:
| trait | statement | effect on demand |
|---|---|---|
Elementwise |
element i of the result depends only on element i of the operands | operands get the result’s region |
Monotone |
elementwise, one operand, non-decreasing | a ranking of the result is a ranking of the operand |
RowIndependent |
slice r along the leading dimension depends only on slice r of the operands | the row set is kept |
NoMemoryEffect |
the op has no effect of its own | an unread result demands nothing |
A weight matmul keeps the row set of its activation and records which weight rows are read. An op that declares nothing is opaque: it is kept, and its operands are demanded whole. A trait is a statement about the operator’s semantics; declaring one on an op whose kernel does not have the property makes a rewrite unsound.
Stages
analyzeStage(op) (BoomStage.h) computes, for every SSA value under a module, a graph or a program, the inputs of a generation the value depends on:
| source | enters at |
|---|---|
prompt |
a program’s arguments other than the cache handles |
counter |
a loop-carried value that an iteration replaces |
token |
the result of boom.sample, boom.draft, boom.accept |
cache |
a cache handle |
It is a sparse forward analysis on the same framework as the demand analysis, solved with the same reachability. A result depends on what the operands depend on; an op not declared free of memory effects depends on every source; boom.step is transparent. A standalone graph has no loop to read from, so its arguments are taken by name: token_id is {token}, pos is {prompt, counter}, a cache handle {cache}, anything else every source.
The set orders into a stage, the earliest moment the value can be computed:
| stage | sources | computable |
|---|---|---|
model |
none | at load |
request |
prompt |
once per generation |
schedule |
+ counter |
for every iteration, ahead of time |
token |
+ token |
once the iteration’s token is chosen |
state |
+ cache |
in the iteration, in order |
recordStages(op) writes the result on each unit as stages: (tensor id, source bits) pairs in ascending id order, with prompt, counter, token and cache as bits 0 to 3. A stage is a fact about what a tensor means, so it stays true through any pass that keeps the tensor; recording again keeps, for each tensor, the sources both recordings name. The lift into a program carries the attribute to the step, and the lowering carries it into planner::Graph::facts (planner/facts.h), where a tensor with no entry reads as every source. The facts take no part in the model fingerprint or the plan.
Effects
The KV cache is a memory-effect resource, not a side channel: boom.attention reads the prefix and appends the current row, boom.flash_attention reads only (the batched append is deferred to boom.kv_append, which writes only), so MLIR can order them. The three cache-touching ops carry their !boom.kv<layer> handle as their last operand.
Pass pipeline
resolveConsumer(module) runs first (see Consumers), then runBoomPasses(module, phase, skip_kv_append_fusion) runs one phase’s passes over the module’s graph, verifying after every pass. A pass order is significant: later passes consume the nodes earlier ones produce. A pass failure fails the plan — the IR is never lowered as if the phase had run.
| phase | passes, in order |
|---|---|
| Decode | QuantizeOncePass, RmsNormQuantizePass, GeluMulQuantizePass, SiluMulQuantizePass, FuseGeluMulPass, BiasIntoGemvPass, ResidualAddIntoGemvPass, FuseResidualEpiloguePass, FuseSiblingGemvPass, FuseGlueRegionsPass, FuseHeadsNormRopeCastPass, FuseKvAppendIntoPrepPass, BufferReusePass, AsyncPrefetchPass |
| Verify | the decode list with FuseKvAppendIntoPrepPass skipped: a verify step appends one row per query position, which a single-row prep region cannot express |
| Prefill | RmsNormMmqPass, FuseAddRmsNormMmqPass, FuseRmsNormAddPass, FoldScaleIntoGluePass, GeluMulMmqPass, SiluMulMmqPass, FuseSliceScalePass, FuseGeluMulPass, FuseSiluMulPass (standard numerics profile only), ScaleIntoDensePass, FuseHeadsNormRopeCastPass, PrefillDispatchPass, DenseLayoutPropagationPass, BufferReusePass, AsyncPrefetchPass |
| Training | BufferReusePass only (semantic-neutral for a traced forward+backward graph) |
What each pass does:
| pass | effect |
|---|---|
QuantizeOncePass |
insert one quantize before the first quantized_matmul consumer of an activation; later matmuls share it |
RmsNormQuantizePass |
rms_norm with one reader (quantize) becomes rms_norm_quantize |
GeluMulQuantizePass / SiluMulQuantizePass |
gated gelu_fp16/silu × up followed by quantize becomes one fused op |
FuseGeluMulPass |
the dense-projection form of the same gated multiply |
FuseSiluMulPass |
the SwiGLU counterpart, prefill only and only under the standard numerics profile (a down projection with no mmq activation layout, e.g. Q6_K, leaves the chain for it) |
BiasIntoGemvPass |
add_bias over a single-reader gemv becomes the projection’s bias_slot epilogue |
ResidualAddIntoGemvPass |
add(proj, residual) becomes a two-input projection epilogue |
FuseResidualEpiloguePass |
the same treatment where the residual add is produced by a norm (sandwich norm / PLE post-norm) |
FuseSiblingGemvPass |
group projections that share an activation |
FuseGlueRegionsPass |
fold a maximal elementwise chain into one boom.fused_region (one generated kernel) |
FuseHeadsNormRopeCastPass |
attention prep chains (norm → rope [→ cast]) in decode and prefill |
FuseKvAppendIntoPrepPass |
fold the K/V append into the attention prep region |
RmsNormMmqPass |
rms_norm + batched q8_1 quantize for the MMQ prefill path |
FuseAddRmsNormMmqPass |
add → norm → quantize as one batched region |
FuseRmsNormAddPass |
norm + add chains the MMQ fold did not take |
FoldScaleIntoGluePass / ScaleIntoDensePass |
a scalar scale after a single-reader chain becomes an epilogue_scale attribute (node position unchanged) |
GeluMulMmqPass / SiluMulMmqPass |
gated activation + batched quantize for MMQ |
FuseSliceScalePass |
strided slice + scale as one elementwise region |
PrefillDispatchPass |
rewrite a batched q8 matmul to the generated int8 tensor-core kernel, per SONICBOOM_MMQ_TC (off default, auto, on) and the SONICBOOM_MMQ_TC_MAX_MB copy budget |
DenseLayoutPropagationPass |
mark the projections with their dense layout (attribute only) |
BufferReusePass |
mark single-use elementwise results the executor may write in place |
AsyncPrefetchPass |
insert prefetch markers when model weights exceed SONICBOOM_VRAM_THRESHOLD_MB (SONICBOOM_ASYNC_PREFETCH=on) |
PruneByDemandPass |
the optional part of resolveConsumer: bypass a monotone map whose result is only ranked, and remove effect-free nodes nothing reads |
Ablation
SONICBOOM_IR_PASS_DISABLE=PassName[,PassName...] skips those passes for the run. Names are the pass class names above. A phase that does not contain a named pass ignores it; an unrecognized name is reported on stderr with the valid names and disables nothing. A disabled pass can also remove a later pass’s fusion (the passes run in sequence), so a plan delta measures that pass and the folds it feeds together.
PruneByDemandPass names the demand-driven pruning inside resolveConsumer. With it disabled an argmax consumer reads the softcapped logits; the index-list narrowing is not part of the switch, because the list is what the consumer asked for rather than an optimization of it.
Adding a pass
- Write the class in
core/compiler/lib/BoomPasses.cppas an MLIR pass overboom::GraphOp. Read the graph through the pass-side view (graphView), which owns the tensor-id ↔︎ SSA-value relation and the tensor table; edit both only through it. - Create replacement nodes with
buildNode, which writes the identity attributes, the authoritative attribute record, and the typed view the verifier checks. Do not construct an op by hand. - Add the pass to the phase’s list in
addPasses, in the position its preconditions require, and add a{"YourPassPass", &DisabledPasses::flag}row tokPassNamesso ablation can name it. - Add a
TESTto the conversion/verifier suite, and a plan golden if the pass changes the node set. - Ensure
boomToGraphstill round-trips: a pass may only produceGraphNodeDescs the planner can rebuild (same tensor table invariants, same attribute kinds).
Verification rules
The verifier (BoomVerify.cpp) rejects, with the offending node named:
- a value whose type disagrees with its
#boom.tensor_descentry (shape and dtype; a!boom.q8_1value is stored as a float32 buffer, so its table entry is float32 with the matching shape); - a
node_attrsrecord that disagrees with the typed attribute view, or an attribute the op may not carry; - a
result_idslist that does not match the op’s arity or the tensor table; - a KV handle anywhere but the last operand of the three cache-touching ops, or whose layer disagrees with the op’s
layer/kv_source_layer; - an unknown op kind, an unsupported dtype, or a quantized shape whose block-contiguous dimension is not a multiple of the block size;
- a
stagesrecord that is not(tensor id, source bits)pairs with the ids ascending and the bits within the four sources (an id the table does not have is allowed, and ignored).