SonicBoom

GGUF v3 Reader — implementation, boundaries, and Gemma 4 gap

Status: reader implemented + unit-tested; real-file sanity-checked. No inference backend built in this task.

Scope

Add an independent, self-contained GGUF v3 reader to SonicBoom. The reader parses the GGUF on-disk format and exposes each tensor’s raw (possibly quantized) bytes by offset + computed size. It does not build a compute graph, schedule work, touch a device backend, or dequantize — those are explicitly out of scope for this task.

Files

The reader is a new core/src/gguf/ subsystem globbed into libsonicboom.so (SONICBOOM_GGUF_SOURCES), mirroring the existing sx/ and planner/ subsystem layout.

What is parsed (GGUF v3)

ggml type table

Ids 0..30 are mapped to (name, block_size, bytes_per_block) — the format reference needed to compute tensor byte sizes (f32, f16, bf16, i8/16/32/64, f64, q4_0/q4_1/q5_0/q5_1/q8_0/q8_1, q2_K…q8_K, and the iq_* types). Unknown type ids yield a nullopt byte size and the tensor is still listed — parsing never hard-fails on an unknown type. No dequantization constants or kernels are implemented; the table is format metadata only.

Dependency boundary

The reader is pure SonicBoom C++23. It has no dependency on ggml, llama.cpp, MLIR, native-torch, or any device backend; the format constants are re-declared by reference to the public GGUF v3 specification (ggml/docs/gguf.md). This is verified by the fact that reader.cpp compiles with only -I core/include and nothing else. It mirrors only the format, never ggml’s execution code.

Error model

ErrorCategory::{Io, Format, Truncated} plus a message and (when known) the byte offset. Corrupted inputs (bad magic, unsupported version, truncated header/string/tensor-info, zero-dim tensors) are rejected cleanly, never via UB.

Test results

tests/core/test_gguf.cpp builds synthetic GGUF files in memory and covers: header + metadata read, all value types (incl. string arrays), the tensor directory, alignment + offsets, raw byte access, boundary checks (out-of-range offset/size → nullopt), unknown types (graceful), corrupted/truncated files, and file-backed load.

Result: test_gguf OK (all checks pass) — compiled and run standalone against reader.cpp; registered in CTest as test_gguf.

Real-model validation

Validated against a real Gemma 4 GGUF file now present in the workspace (models/gemma-4-E4B-it-Q3_K_M.gguf, 4.06 GB, GGUF v3):

file size tensors result
~/Project/velum/models/hift.gguf 83 MB 246 parsed OK (vocoder, f32)
~/Project/velum/models/llm.gguf 2.5 GB 293 parsed OK (transformer LLM, f32)
models/gemma-4-E4B-it-Q3_K_M.gguf 4.06 GB 720 parsed OK (Gemma 4 E4B-It, mixed quant)

The Gemma 4 file parses cleanly end to end: version=3, n_kv=56, alignment=32, general.architecture = "gemma4", general.name = "Gemma-4-E4B-It", context 131072, embedding_length=2560, 42 blocks, GQA heads 8/2, sliding window 512, final_logit_softcapping=30. All 720 tensors resolve to a correct data span (resolvable=720 bad=0 unknown_type=0), proving the header → KV → tensor-directory → offset/alignment → raw-bytes path on a real quantized model.

Tensor dtype mix (the “_M” in Q3_K_M means mixed):

ggml type count
f32 423 (norms, biases, layer_output_scale)
q3_K 169 (token_embd, attn_q/k, ffn_gate/up)
q4_K 123 (attn_output, per_layer_token_embd)
q5_K 4 (attn_v, ffn_down)
bf16 1 (per_layer_model_proj)

Two Gemma-4-specific structure notes relevant to execution: (1) there is no output.weight — the LM head is a per-layer token embedding (per_layer_model_proj.weight [2560×10752, bf16] × per_layer_token_embd.weight [10752×262144, q4_K]); (2) every block carries a learned layer_output_scale (f32 scalar) applied to its residual output. Neither is a plain tied embedding. The reader exposes all of this by name/shape/offset/bytes; none of the dequant or computation is in scope here.

Gemma 4 compute gap (existing capability vs. required)

SonicBoom’s current compute surface, assessed without building an inference backend:

Gemma 4 is a decoder-only transformer. Mapping requirement → current support:

Gemma 4 requirement present gap
Embedding lookup — no gather/embedding op
RMSNorm / LayerNorm — no norm op (only reduce_mean, no pow/sqrt/mul to compose it)
RoPE — no rotary-position op
QKV projection gemm ✓ —
Head split / reshape / transpose reshape ✓ no transpose/split
Scaled dot-product attention gemm + softmax ✓ no mul for the scale, no fused attention
QK-norm — depends on missing RMSNorm
MLP (up/gate/down) gemm ✓ —
Activation (GELU/SiLU/SwiGLU) — no activation op (only relu)
Residual add add ✓ —
LM head gemm ✓ —
Quantized weights (GGUF Q4_K etc.) — no dequant, no quantized GEMM, no quantized dtype in the IR
KV cache / decoding (top-k, top-p, sampling) — not a graph op; absent

Concrete missing operators for a Gemma 4 path: rms_norm (or layer_norm), rope, gelu/silu/swiglu, embedding/gather, transpose, and a mul (attention scaling). On top of the operator set, quantized execution needs dequantization kernels and a quantized dtype — both explicitly deferred here.

None of these are implemented in this task; the deliverable is the reader plus this gap list.