sonicboom/trace/fused
One fused op per training layer, emitted under the S-Expr op names the lowering resolves to Boom IR operators:
fused-softmax -> softmax (last axis) fused-rms-norm -> rms_norm fused-rope -> rope fused-silu-mul -> silu_mul fused-cross-entropy -> cross_entropy (mean NLL, 1-D int targets)
backward! (sonicboom trace backward) recognizes these forward nodes and emits the matching *_bwd operator from the same recording. Embedding lookup is the ordinary index-select on axis 0. The composite versions in (sonicboom trace layers) still decompose into primitive ops; these exist for training graphs that go to the native lowering.
adamw-step! is the one optimizer update that is itself a graph op: one node for the whole parameter/moment update, so a training graph with any number of parameters has one optimizer node per tensor instead of a decomposed arithmetic chain.
fused-softmax
procedure
A numerically stable softmax over the last axis.
Each element becomes exp(x - max) / sum(exp(x - max)), with the max taken over the last axis so large values do not overflow.
Arguments:
x: a tensor.
Returns: a tensor with X’s shape whose last axis sums to 1.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor->list (fused-softmax (list->ctensor '(1.0 2.0))))=> (0.2689414322376251 0.7310585975646973)
PyTorch: torch.nn.functional.softmax
Note: the axis is always the last one; torch.nn.functional.softmax takes a dim.
fused-rms-norm
procedure
Root-mean-square normalisation over the last axis.
Computes x / sqrt(mean(x^2) + EPS) * W.
Arguments:
x: a tensor.w: the weight, shaped as the last axis.eps: keyword; added under the root (default 1e-6).
Returns: a tensor with X’s shape.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor->list (fused-rms-norm (list->ctensor '(3.0 4.0))
(list->ctensor '(1.0 1.0))))=> (0.8485280871391296 1.1313707828521729)
PyTorch: torch.nn.functional.rms_norm
Note: only the last axis is normalised; torch.nn.functional.rms_norm takes a normalised shape.
fused-rope
procedure
Rotary position embedding on the last axis.
Row i is rotated at position POS0 + i, with HEAD-DIM values per head.
Arguments:
x: a tensor [rows, heads * head-dim].head-dim: keyword; the number of rotated values per head (required).theta-base: keyword; the frequency base (default 10000.0).freq-scale: keyword; the frequency scale (default 1.0).pos0: keyword; the position of the first row (default 0).
Returns: a tensor with X’s shape.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor->nested (fused-rope (list->ctensor '((1.0 2.0) (1.0 0.0)))
#:head-dim 2))=> ((1.0 2.0) (0.5403022766113281 0.8414709568023682))
PyTorch: none
Note: PyTorch has no rotary embedding function; the rotation convention differs between implementations.
fused-silu-mul
procedure
The SwiGLU activation silu(GATE) * UP.
Arguments:
gate: the gate tensor.up: the up-projection tensor, same shape as GATE.
Returns: a tensor with GATE’s shape.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor->list (fused-silu-mul (list->ctensor '(0.0 1.0))
(list->ctensor '(1.0 1.0))))=> (0.0 0.7310585975646973)
PyTorch: none
Note: PyTorch has no fused SwiGLU; it is written torch.nn.functional.silu(gate) * up.
fused-cross-entropy
procedure
The mean cross-entropy of class indices TARGETS under LOGITS.
Computes log_softmax over the class axis, gathers the target column of each row and returns the mean negative log-likelihood as a rank-0 tensor.
Arguments:
logits: a float tensor [rows, classes].targets: an int64 tensor [rows] of class indices.
Returns: a rank-0 tensor.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor-item (fused-cross-entropy (list->ctensor '((0.0 1.0) (2.0 0.0)))
(make-ctensor 'int64 '(2) '(1 0))))=> 0.22009484469890594
PyTorch: torch.nn.functional.cross_entropy
Note: the reduction is always the mean and TARGETS are always class indices; torch.nn.functional.cross_entropy also accepts class probabilities and other reductions.
adamw-step!
procedure
One AdamW update of PARAM from GRAD.
M1 and M2 are the first and the second moment tensors and STEP is the 1-based step counter tensor. When PARAM, M1 or M2 is a leaf, its value is replaced by the updated tensor.
Arguments:
param: the parameter tensor.grad: its gradient, same shape.m1: the first moment.m2: the second moment.step: the step counter, a rank-0 integer tensor.lr: keyword; the learning rate, a number (default 1e-3).beta1: keyword; the first moment decay (default 0.9).beta2: keyword; the second moment decay (default 0.999).eps: keyword; added under the root (default 1e-8).weight-decay: keyword; the decoupled weight decay (default 0.01).
Returns: the list (new-param new-m1 new-m2).
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(ctensor->list
(car (adamw-step! (list->ctensor '(1.0 2.0)) (list->ctensor '(0.5 0.5))
(list->ctensor '(0.0 0.0)) (list->ctensor '(0.0 0.0))
(make-ctensor 'int32 '() '(1)) #:lr 0.1)))=> (0.8989999890327454 1.8980000019073486)
PyTorch: torch.optim.AdamW
Note: LR may also be a tensor (a leaf or a symbolic argument), in which case it is an input of the update rather than an attribute; torch.optim.AdamW reads the rate from the optimizer.
fused-clip-grad-norm
procedure
The gradients GRADS scaled so that their global L2 norm does not exceed MAX-NORM.
The norm is accumulated in double over every element of every gradient. When it does not exceed MAX-NORM the results are copies of the inputs; otherwise each gradient is scaled by MAX-NORM / (norm + 1e-6).
Arguments:
grads: a list of gradient tensors.max-norm: keyword; the norm limit (default 1.0).
Returns: one clipped tensor per gradient, in order.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(map ctensor->list
(fused-clip-grad-norm (list (list->ctensor '(3.0 4.0))) #:max-norm 1.0))=> ((0.5999999046325684 0.7999998331069946))
PyTorch: torch.nn.utils.clip_grad_norm_
Note: torch.nn.utils.clip_grad_norm_ scales the .grad of parameters in place; this returns new tensors.
clip-grad-norm!
procedure
Clip the stored gradients of PARAMS by their global L2 norm, in place.
PARAMS is a leaf or a list of leaves, typically the ones backward! filled in. After the call each leaf’s .grad is the clipped gradient. Leaves without an accumulated gradient are skipped.
Arguments:
params: a leaf or a list of leaves.max-norm: optional; the norm limit (default 1.0).
Returns: the list of clipped gradient values.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace fused))
(let ((leaf (make-leaf (list->ctensor '(1.0 1.0)))))
(set-leaf-grad! leaf (list->ctensor '(3.0 4.0)))
(clip-grad-norm! leaf)
(ctensor->list (leaf-grad leaf)))=> (0.5999999046325684 0.7999998331069946)
PyTorch: torch.nn.utils.clip_grad_norm_
Note: operates on the .grad slot of leaf tensors; torch.nn.utils.clip_grad_norm_ operates on the .grad of parameters.