sonicboom/trace/backward
backward! is the training counterpart of grad in (sonicboom trace autodiff). Called on a symbolic loss inside a traced function, it walks the nodes recorded so far backwards from the loss and appends the adjoint nodes to the same recording, so the graph handed to the lowering holds the forward and the backward pass together.
The difference from grad is where the adjoints come from: the fused training ops (matmul, softmax, cross-entropy, RMSNorm, RoPE, SwiGLU, embedding) emit their Boom IR backward operators
matmul_bwd, softmax_bwd, cross_entropy_bwd, rmsnorm_bwd, rope_bwd, silu_mul_bwd, embedding_bwd
instead of being decomposed into primitive ops. Every other op reuses its rule from the op table, so the walk covers the same elementwise, reduction, shape and indexing vocabulary as grad.
Results land in the .grad field of the caller’s leaf tensors (parameters): a leaf passed as an argument is a leaf in the trace too, backward! stores the symbolic adjoint there, the tracer makes it a segment output, and after replay writes the concrete gradient back into the caller’s leaf. zero-grad! clears the slot. Because the training graph is replayed, parameters should be leaves passed as arguments, not free variables computed inside the body.
backward!
procedure
Reverse-mode differentiation of LOSS on the tape being traced.
Walks the nodes recorded so far backwards from LOSS and appends the adjoint nodes to the same recording, so the graph holds the forward and the backward pass together. Each leaf’s gradient is stored in its .grad slot.
Arguments:
loss: a rank-0 symbolic tensor.
Returns: the list of gradients, in the order the leaves were bound.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace symbolic)
(sonicboom trace macro) (sonicboom trace backward))
(define-traced (f x) (let ((loss (sum (mul x x)))) (backward! loss) loss))
(let ((leaf (make-leaf (list->ctensor '(1.0 2.0)))))
(f leaf)
(ctensor->list (leaf-grad leaf)))=> (2.0 4.0)
PyTorch: torch.autograd.backward
Note: Gradients are recorded into the traced graph and land in each leaf’s grad slot when it is replayed.
zero-grad!
procedure
Clear the .grad slot of a leaf or of every leaf in a list.
Arguments:
leaves: a leaf or a list of leaves.
Returns: LEAVES.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace backward))
(let ((leaf (make-leaf (list->ctensor '(1.0)))))
(set-leaf-grad! leaf (list->ctensor '(5.0)))
(zero-grad! leaf)
(leaf-grad leaf))=> #f
PyTorch: torch.optim.Optimizer.zero_grad
Note: Clears the gradient of every leaf it is given; PyTorch’s method clears the parameters of one optimizer and by default sets them to None.
leaf-grads
procedure
The current .grad of a leaf or of every leaf in a list.
Arguments:
leaves: a leaf or a list of leaves.
Returns: the gradient tensor of one leaf, or a list of gradients in LEAVES order.
Example:
(use-modules (sonicboom trace tensor) (sonicboom trace backward))
(let ((leaf (make-leaf (list->ctensor '(1.0)))))
(set-leaf-grad! leaf (list->ctensor '(5.0)))
(ctensor->list (leaf-grads leaf)))=> (5.0)
PyTorch: none
Note: a leaf without a gradient returns #f.