SonicBoom

sonicboom/trace/tensor

Research prototype. Pure Scheme; no libsonicboom. A concrete tensor (“ctensor”) is a dtype, a static shape and an owned over a flat row-major bytevector holding the elements in the dtype’s native representation (native-endian, like libsonicboom’s tensors): IEEE binary32 for float32, IEEE binary16 for float16, bfloat16 for bfloat16, IEEE binary64 for float64, two’s-complement native integers, and one byte 0/1 for bool. Element access converts to and from Scheme numbers (inexact reals for float dtypes, exact integers for integer dtypes, booleans for bool).

The bytevector is the storage, not a serialised view: ctensor-ref and ctensor-set! read and write it in place, and bytevector->ctensor shares it (reshape/view aliasing). Materialising a whole tensor into Scheme values is explicit: ctensor->list / ctensor->nested / ctensor-write are the only such interfaces, and printing streams from the storage.

It exists only so that the reference interpreter and eager mode have something to compute on; it is not a SonicBoom runtime tensor.

make-ctensor

procedure

(make-ctensor dtype shape data)

A tensor built from a dtype, a shape and flat data.

DATA is a flat list or vector of Scheme values, or a bytevector of packed dtype values. A bytevector is copied; to share storage, use bytevector->ctensor.

Arguments:

Returns: a new ctensor.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->list (make-ctensor 'float32 '(2 2) '(1.0 2.0 3.0 4.0)))

=> (1.0 2.0 3.0 4.0)

PyTorch: torch.tensor

Note: The shape is given separately from the flat data; PyTorch infers it from the nesting of its input.

ctensor?

record type

A concrete reference tensor: a dtype, a static shape and owned storage.

The bytes are row-major and native-endian in the dtype’s native representation. ctensor? tests the type; ctensor-dtype and ctensor-shape read the dtype and the shape; ctensor-data exposes the backing bytevector.

PyTorch: torch.Tensor

Note: This is the tracing prototype’s reference tensor, written in pure Scheme. It models a tensor for the tracer at compile time; it is not the runtime tensor of (sonicboom tensor).

ctensor-dtype

record type

A concrete reference tensor: a dtype, a static shape and owned storage.

The bytes are row-major and native-endian in the dtype’s native representation. ctensor? tests the type; ctensor-dtype and ctensor-shape read the dtype and the shape; ctensor-data exposes the backing bytevector.

PyTorch: torch.Tensor

Note: This is the tracing prototype’s reference tensor, written in pure Scheme. It models a tensor for the tracer at compile time; it is not the runtime tensor of (sonicboom tensor).

ctensor-shape

record type

A concrete reference tensor: a dtype, a static shape and owned storage.

The bytes are row-major and native-endian in the dtype’s native representation. ctensor? tests the type; ctensor-dtype and ctensor-shape read the dtype and the shape; ctensor-data exposes the backing bytevector.

PyTorch: torch.Tensor

Note: This is the tracing prototype’s reference tensor, written in pure Scheme. It models a tensor for the tracer at compile time; it is not the runtime tensor of (sonicboom tensor).

ctensor-data

procedure

(ctensor-data t)

The bytevector backing T.

The bytevector is the tensor’s storage, not a copy: writing to it changes the tensor.

Arguments:

Returns: a bytevector of (ctensor-nbytes T) bytes.

Example:

(use-modules (sonicboom trace tensor))
(bytevector-length (ctensor-data (make-ctensor 'float32 '(2) '(1.0 2.0))))

=> 8

PyTorch: none

Note: The result is the raw bytevector, not a NumPy array.

bytevector->ctensor

procedure

(bytevector->ctensor dtype shape bv)

A tensor that shares BV’s storage.

BV must hold exactly one packed element per element of SHAPE in DTYPE. No copy is made: writes through ctensor-set! are visible in BV, and writes to BV are visible through the tensor.

Arguments:

Returns: a new ctensor sharing BV.

Example:

(use-modules (sonicboom trace tensor) (rnrs bytevectors))
(let* ((bv (make-bytevector 8 0))
       (t (bytevector->ctensor 'int32 '(2) bv)))
  (ctensor-set! t 0 7)
  (bytevector-s32-native-ref bv 0))

=> 7

PyTorch: torch.frombuffer

Note: torch.frombuffer also views a buffer without copying.

ctensor-copy

procedure

(ctensor-copy t)

An independent tensor with the same dtype, shape and bytes.

Arguments:

Returns: a new ctensor; changing one does not affect the other.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->list (ctensor-copy (make-ctensor 'float32 '(2) '(1.0 2.0))))

=> (1.0 2.0)

PyTorch: torch.Tensor.clone

ctensor-nbytes

procedure

(ctensor-nbytes t)

The size of T’s storage in bytes.

Arguments:

Returns: an exact integer.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-nbytes (make-ctensor 'float32 '(2 3) '(0.0 0.0 0.0 0.0 0.0 0.0)))

=> 24

PyTorch: torch.Tensor.nbytes

dtype-size

procedure

(dtype-size d)

The number of bytes per element of DTYPE.

Arguments:

Returns: 1, 2, 4 or 8.

Example:

(use-modules (sonicboom trace tensor))
(dtype-size 'float64)

=> 8

PyTorch: none

ctensor-numel

procedure

(ctensor-numel t)

The number of elements in T.

Arguments:

Returns: an exact integer.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-numel (make-ctensor 'float32 '(2 3) '(0.0 0.0 0.0 0.0 0.0 0.0)))

=> 6

PyTorch: torch.Tensor.numel

ctensor-ref

procedure

(ctensor-ref t i)

Element I of T as a Scheme value.

The index is flat and row-major. The value is an inexact real for a float dtype, an exact integer for an integer dtype, and a boolean for bool.

Arguments:

Returns: the element.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-ref (make-ctensor 'float32 '(2) '(1.5 2.5)) 1)

=> 2.5

PyTorch: none

Note: PyTorch indexes with one subscript per axis; this index is flat.

ctensor-set!

procedure

(ctensor-set! t i v)

Store V as element I of T.

The value is converted to T’s dtype: floats round to the dtype’s precision (nearest, ties to even) and integers must be within the dtype’s range.

Arguments:

Returns: an unspecified value.

Example:

(use-modules (sonicboom trace tensor))
(let ((t (make-ctensor 'float32 '(2) '(1.0 2.0))))
  (ctensor-set! t 0 5.0)
  (ctensor->list t))

=> (5.0 2.0)

PyTorch: none

Note: The write is direct; PyTorch item assignment is subscript-based and records autograd history.

ctensor->list

procedure

(ctensor->list t)

All elements of T as a flat Scheme list in row-major order.

Arguments:

Returns: a list with (ctensor-numel T) elements.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->list (make-ctensor 'float32 '(2) '(1.0 2.0)))

=> (1.0 2.0)

PyTorch: torch.Tensor.tolist

ctensor->nested

procedure

(ctensor->nested t)

The elements of T regrouped into nested lists following its shape.

A rank-0 tensor gives its single element rather than an empty list.

Arguments:

Returns: a nested list, or the single element of a rank-0 tensor.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->nested (make-ctensor 'float32 '(2 2) '(1.0 2.0 3.0 4.0)))

=> ((1.0 2.0) (3.0 4.0))

PyTorch: torch.Tensor.tolist

ctensor-write

procedure

(ctensor-write t port)

Write T’s values to PORT as nested lists in shape order.

The values are streamed from storage; no whole-tensor list is built.

Arguments:

Returns: an unspecified value.

Example:

(use-modules (sonicboom trace tensor))
(call-with-output-string
  (lambda (p)
    (ctensor-write (make-ctensor 'float32 '(2 2) '(1.0 2.0 3.0 4.0)) p)))

=> "((1.0 2.0) (3.0 4.0))"

PyTorch: none

list->ctensor

procedure

(list->ctensor nested #:optional (dtype #f))

A tensor built from nested lists.

The nesting gives the shape; a bare number or boolean gives a rank-0 tensor.

Arguments:

Returns: a new ctensor.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->nested (list->ctensor '((1 2) (3 4))))

=> ((1.0 2.0) (3.0 4.0))

PyTorch: torch.tensor

scalar->ctensor

procedure

(scalar->ctensor x #:optional (dtype #f))

A rank-0 tensor holding X.

Arguments:

Returns: a rank-0 ctensor.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-item (scalar->ctensor 3.0))

=> 3.0

PyTorch: torch.tensor

ctensor-item

procedure

(ctensor-item t)

The single element of a one-element tensor.

T may have any shape as long as it has exactly one element.

Arguments:

Returns: the element.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-item (make-ctensor 'float32 '(1 1) '(7.0)))

=> 7.0

PyTorch: torch.Tensor.item

make-leaf

procedure

(make-leaf value #:optional (grad #f))

A leaf with VALUE and an optional gradient.

The value is a ctensor, or a symbolic tensor while the tracer runs.

Arguments:

Returns: a new leaf.

Example:

(use-modules (sonicboom trace tensor))
(ctensor->list (leaf-value (make-leaf (make-ctensor 'float32 '(2) '(1.0 2.0)))))

=> (1.0 2.0)

PyTorch: none

Note: A PyTorch parameter is a tensor that owns its gradient; a leaf keeps the value and the gradient in separate slots.

leaf?

record type

A tensor value and its gradient slot.

A leaf is what a traced training step differentiates. leaf-value holds the tensor (a ctensor, or the symbolic tensor the tracer substitutes for it) and leaf-grad holds its gradient, if any. leaf? tests the type; leaf-value and leaf-grad read the slots and set-leaf-value! and set-leaf-grad! replace them. The object keeps its identity across calls while the slots change.

PyTorch: torch.nn.Parameter

Note: The nearest PyTorch notion is a leaf tensor with requires_grad; here the value and the gradient are two slots of an explicit object.

leaf-value

record type

A tensor value and its gradient slot.

A leaf is what a traced training step differentiates. leaf-value holds the tensor (a ctensor, or the symbolic tensor the tracer substitutes for it) and leaf-grad holds its gradient, if any. leaf? tests the type; leaf-value and leaf-grad read the slots and set-leaf-value! and set-leaf-grad! replace them. The object keeps its identity across calls while the slots change.

PyTorch: torch.nn.Parameter

Note: The nearest PyTorch notion is a leaf tensor with requires_grad; here the value and the gradient are two slots of an explicit object.

set-leaf-value!

record type

A tensor value and its gradient slot.

A leaf is what a traced training step differentiates. leaf-value holds the tensor (a ctensor, or the symbolic tensor the tracer substitutes for it) and leaf-grad holds its gradient, if any. leaf? tests the type; leaf-value and leaf-grad read the slots and set-leaf-value! and set-leaf-grad! replace them. The object keeps its identity across calls while the slots change.

PyTorch: torch.nn.Parameter

Note: The nearest PyTorch notion is a leaf tensor with requires_grad; here the value and the gradient are two slots of an explicit object.

leaf-grad

record type

A tensor value and its gradient slot.

A leaf is what a traced training step differentiates. leaf-value holds the tensor (a ctensor, or the symbolic tensor the tracer substitutes for it) and leaf-grad holds its gradient, if any. leaf? tests the type; leaf-value and leaf-grad read the slots and set-leaf-value! and set-leaf-grad! replace them. The object keeps its identity across calls while the slots change.

PyTorch: torch.nn.Parameter

Note: The nearest PyTorch notion is a leaf tensor with requires_grad; here the value and the gradient are two slots of an explicit object.

set-leaf-grad!

record type

A tensor value and its gradient slot.

A leaf is what a traced training step differentiates. leaf-value holds the tensor (a ctensor, or the symbolic tensor the tracer substitutes for it) and leaf-grad holds its gradient, if any. leaf? tests the type; leaf-value and leaf-grad read the slots and set-leaf-value! and set-leaf-grad! replace them. The object keeps its identity across calls while the slots change.

PyTorch: torch.nn.Parameter

Note: The nearest PyTorch notion is a leaf tensor with requires_grad; here the value and the gradient are two slots of an explicit object.

make-lr-tensor

procedure

(make-lr-tensor #:optional (device (quote cpu)) #:key (value 0.001))

A one-element float32 learning-rate tensor for a traced training step.

Passed to a training step as #:lr, the tensor becomes a graph input rather than an attribute, so a schedule can change the rate without retracing. Read the rate with leaf-value and replace it with set-leaf-value!.

Arguments:

Returns: a leaf holding a rank-0 float32 tensor.

Example:

(use-modules (sonicboom trace tensor))
(ctensor-item (leaf-value (make-lr-tensor 'cpu #:value 0.5)))

=> 0.5

PyTorch: none

Note: PyTorch passes the learning rate as a number to the optimizer.

shape-numel

procedure

(shape-numel shape)

The number of elements of SHAPE.

Arguments:

Returns: the product of the dimensions; 1 for the empty shape.

Example:

(use-modules (sonicboom trace tensor))
(shape-numel '(2 3 4))

=> 24

PyTorch: none

float-dtype?

procedure

(float-dtype? d)

Whether DTYPE is a floating-point dtype.

Arguments:

Returns: #t for float32, float16, bfloat16 and float64; #f otherwise.

Example:

(use-modules (sonicboom trace tensor))
(float-dtype? 'float16)

=> #t

PyTorch: none

int-dtype?

procedure

(int-dtype? d)

Whether DTYPE is an integer dtype.

Arguments:

Returns: #t for int8, uint8, int16, int32 and int64; #f otherwise.

Example:

(use-modules (sonicboom trace tensor))
(int-dtype? 'int64)

=> #t

PyTorch: none

valid-dtype?

procedure

(valid-dtype? d)

Whether DTYPE is one of the dtypes a ctensor accepts.

Arguments:

Returns: #t for float32, float16, bfloat16, float64, int8, uint8, int16, int32, int64 and bool; #f otherwise.

Example:

(use-modules (sonicboom trace tensor))
(valid-dtype? 'bfloat16)

=> #t

PyTorch: none

coerce-value

procedure

(coerce-value dtype x)

Convert X to the Scheme representation used for DTYPE.

A float dtype accepts any real and returns it inexact; an integer dtype accepts an integer value and returns it exact; bool accepts a boolean or a real and returns a boolean.

Arguments:

Returns: the converted Scheme value.

Example:

(use-modules (sonicboom trace tensor))
(coerce-value 'int32 2.0)

=> 2

PyTorch: none