SonicBoom

Weight formats in the graph — design note (Blade side first)

Status: proposal, under revision. Nothing in this document is implemented.

The document exists in the tree because the Boom IR passes point at it: TODO(weight-format) in the ported passes (core/compiler/lib/BoomPasses.cpp:825, :992) and the private ds predicate those passes carry, which they kept from the Graph rewrites they were ported from (core/src/planner/rewrites.cpp, since deleted). Both say the same thing: a weight’s quant format is an emitter-baked integer today, and it should be a property of a type.

All file:line references are to core/ unless stated, and are checked against the boom-passes tip. Statements that are inference rather than observation are marked (inference).

1. Why the graph cannot say what a weight is quantized as

TensorDesc (include/sonicboom/planner/graph.h:41-53) has shape, dtype, size_bytes and four flags — no format. sx::DType (sx/ir.h) has ten members, none quantized. So the only quantized value the dialect can name for a graph tensor is q8_1, and it is named by provenance (the first operand of quantized_matmul_q8), not by a declaration. Design §6.2 wants the opposite: !boom.q4_k … as first-class types for weights too, so the ds-consistency precondition of fuse_rms_norm_mmq_pass becomes a verifier check instead of a private predicate.

2. Where weights live today, and where their sizes come from

Weights are not graph tensors in the model path. The emitter bakes (layer, slot) onto each node and the backend reads model.weights (model/plan.h:22-27; SonicBackend::weight at planner/sonic_backend.cpp:153-177). No Gemma4Model/Qwen2 emitter node carries is_constant — verified by grep: core/src/model/*.cpp never sets it. So no TensorDesc::size_bytes in any model graph describes a weight; every tensor in those graphs is an f32 activation.

Weights live in two shapes (model/weights.h):

On the device: upload_weight (quant/cuda_quant_matmul.cu:612-629) sizes the device copy from W.byte_size() (+256 B tail pad, ggml’s own padding) and caches it by (host pointer, byte size). So the device footprint already follows the packed format. Nothing accounts for it: device_local.capacity_bytes = 0 // unknown; not budget-checked in v0 (planner/resource.cpp:312).

The one path where a weight is a graph tensor is the Blade/SX one: is_constant / is_external (graph.h:49-50), set in adapt_graph (planner/graph_adapter.cpp:167) from sx::tensor_byte_size(type). That function derives the size from dtype alone (sx/ir.cpp:74), so it cannot produce a packed size for a quantized tensor — this is exactly the place a format field is needed.

3. Does any planning, budgeting or allocation logic use a weight’s size_bytes?

The short answer, per path.

Layout as capacity — the existing precedent, where the two meanings are already equated:

Identity and records (this is what breaks first):

Not affected: the KV cache geometry (planner/kv_cache.*) is format-free.

4. Proposal

4.1 Blade / planner side

Add the format to the tensor, keeping today’s convention that a quantized tensor’s logical element type is f32:

struct TensorDesc {
  ...
  sx::DType dtype;        // logical element type (f32 for quantized tensors)
  std::optional<sx::QuantFormat> quant;  // packed format; nullopt = plain
  uint64_t size_bytes;    // bytes the tensor occupies (packed when quant set)
};

Cost of the change: model_fingerprint gains a field → every fingerprint moves → all four plan goldens must be re-recorded once, in one commit, with the plan diff tool showing only fingerprint/format-affected lines. That is expected, not a regression.

4.2 Boom IR side

The dialect’s half of this is already in place as of the boom-dialect work, and it is worth stating exactly what exists before saying what is missing, because the two halves of the review’s ask land in different places:

4.3 The ds predicate — the first check that goes away

ds is the mmq activation layout a weight format selects, and it is the concrete reason this proposal exists:

Once a weight carries its format as a type, all three facts are derivable: ds is “the activation layout this weight format’s mmq kernel reads”, the “no mmq layout” answer is a property of the format (not of an integer’s sign), and the agreement test between two matmuls is a type equality. The predicate then moves from a private function in two implementations of the same pass into the verifier, which is the acceptance criterion for §5 step 3 — a plan goldens diff that shows no change, plus the predicate’s deletion.

5. Suggested order (each step independently verifiable)

  1. TensorDesc + serialization + fingerprint get the field, always empty. No behaviour change; the four goldens are re-recorded once here, and only the fingerprint lines move.
  2. sx::tensor_byte_size becomes format-aware; Blade v0.2 can name a format for a parameter. First step where a real packed weight size exists.
  3. Boom IR: weights become graph values that carry their format type, per-op format declarations and the verifier rules of §4.2 — the step that removes the ds predicate (§4.3).
  4. Re-record the goldens; the round-trip test then proves the format survives.

Steps 1 and 2 are separable from 3; step 3 is where the dialect’s type story becomes what design §6.2 describes. Step 3 is also the largest: weights entering the graph changes the emitter and the backend’s model.weights lookup, and is inference — not costed here.

6. Decisions needed

  1. Do quantized tensors keep dtype = f32 as their logical type (this document’s proposal, matching QuantizedTensor), or does sx::DType gain quantized members? The first keeps the ten dtypes and the existing kernels; the second makes the format a dtype everywhere.
  2. Should Blade be able to type a quantized intermediate (a quantized activation other than q8_1, a quantized KV), or is a quantized tensor only ever a parameter plus the q8_1 activation? This decides whether the verifier needs a general “producer of format F” rule (§4.2 rule 3) or just the two producers that exist.
  3. Should device weight residency be accounted for at all? Today it is a private cache in the .cu with no budget and no plan visibility (resource.cpp:312).
  4. Should step 3 make weights graph values outright, or keep (layer, slot) and attach the format to the node attribute — a smaller change that fixes the verifier but leaves #boom.tensor_desc without a format? (§4.2 gap 1; the two differ in whether the round trip can carry a weight at all.)

Revision