Training and evaluation#

Fit a policy's parameters to labelled data, measure it honestly, compare it with alternatives, and deploy exactly the values you measured.

A policy becomes trainable when it has tensors (values to learn), a loss (what "good" means against your labels) and an optimiser (how to search). A training run fits the tensors on each training fold, scores the policy on the held-out entries, and, with final_fit, refits on everything to produce deployable versions.

# parcel/scale.sb — a box's length, correcting the scan for a scale error kpolicy(cloud: Cloud, sigma_h: Length = 1 mm) -> (length: Gaussian[Length])import policy parcel/boxoptimiser delegateloss    { length: nll }metrics { err: abs_error(length, y.length), lpd: log_predictive_density(length, y.length) }raw = box(cloud, sigma_h: sigma_h)k       = tensor(0.0, lower: -0.1, upper: 0.1, name: "k")sigma_2 = tensor(3 mm, lower: 0 mm, scale: 1 mm, name: "sigma_2")length  = linear(raw.length_raw, coef: k, noise: sigma_2, offset: raw.length_raw)

linear has a built-in fitter, so optimiser delegate solves k and sigma_2 directly on each training fold.

Optimisers#

Declare optimisers in the policy header. The current built-ins are delegate, adam and cma_es, and more built-in optimisers are coming.

optimiser delegate                       # nodes with a built-in fitter fit their own tensorsoptimiser adam(lr: 1e-2, steps: 1000)    # or:optimiser cma_es(sigma0: 0.3)

Declare at most one non-delegate optimiser, such as adam or cma_es; delegate may be declared alongside it. A policy with only delegate makes one pass of delegate fits per split.

Delegate fits#

linear, stack and conditional_logit have built-in fitters: closed-form or convex solves that are exact and fast. Under optimiser delegate, such a node fits its own tensors when all of these hold:

1. its tensors are trainable and used by that node alone, at one call site;
2. exactly one Form-1 loss term targets it: the node's result is bound straight to a declared output, with no other operation in between (length = linear(…) with loss { length: nll }), and the term's kind (nll or mse) is one the fitter supports. Inside an unfrozen imported policy, that policy's own outputs and loss count as targets, matched by name to your y columns;
3. its tensors' bounds are constants;
4. you haven't passed fit: "optimiser" to the node.

Otherwise its tensors go to the global optimiser, such as adam or cma_es. If a trainable tensor ends up with no optimiser that can train it, the editor shows W0520 and training fails with E0520, naming each tensor and the reason. The graph records, for each fit node, whether it is delegated and, if not, why.

Delegates minimise their own loss term; any other term that depends on them steers only the global optimiser. Unless they're static (below), they re-solve inside every global evaluation (every adam step, every cma_es candidate), always on all the training entries (under adam, those not held out for early stopping), never a minibatch. A delegate's regulariser (ridge) is part of the total loss.

Under adam, each delegate needs a way to pass gradients back, chosen in this order: it is static (nothing trainable upstream, so it fits once per fold), envelope (only its own term depends on it and the fit is exact), or implicit (one Newton correction gives the exact implicit gradient, for up to 256 fitted values). If none applies, compilation fails with E0522 (help: use cma_es, or freeze the tensors upstream). stack is never envelope or implicit, so under adam it must be static. cma_es needs no gradients, so this never comes up.

adam#

AdamW with linear warmup, then a cosine (or constant) schedule, and gradient clipping.

• Bounds hold by construction: a bounded tensor is optimised through a sigmoid or softplus reparameterisation, never clipped. Each tensor moves in units of its scale.
• Early stopping watches an inner holdout split off the training fold, never the validation fold, and restores the best checkpoint. A fold with fewer than two groups has no inner holdout, so no early stopping (W0607).
• A non-finite step is skipped and the learning rate halved; ten in a row abort the run (R0601).
• While training, point selections downstream of a trainable tensor use soft (sigmoid) masks, annealed towards hard, so gradients flow. The early-stopping check, validation and metrics always use hard masks, and every fit (each split's and the final one) ends with a hard-mode delegate re-solve on the whole training fold.
• A tensor trained by adam needs a differentiable path to the loss (E0521 names the input that blocks it).
• Panels: loss (train and inner holdout), learning rate, gradient norm, and the trajectory of each tensor (up to 32 values).

cma_es#

Covariance-matrix adaptation evolution strategy (pycma), with BIPOP restarts.

• Bounds always hold (the same reparameterisation as adam).
• Every candidate is scored on the full training fold, with delegates re-solved. A candidate that fails scores +∞; if more than half a generation fails, the run aborts (R0602).
• Masks are always hard, and gradients are never needed, so any node can sit on the path.
• Warning W0530 above 50 trainable values; error E0530 above 500 (use adam for weight tensors).
• Panels: best and median fitness per generation, step size, axis ratio, eigenvalue spectrum, restarts.

Which to use#

Among the current built-ins:

Stochastic nodes (sample) are seeded per entry, node and run, so every candidate and every step sees the same random numbers.

Loss and metrics#

loss    { length: nll, lid: nll(weight: 0.5) }metrics { exact: agreement(lid, y.lid), err: abs_error(length, y.length) }

• Form-1 terms (out: nll | mse) compare a declared output with the same-named y column. Form-2 terms are any expression over outputs, y.<column> and the block's own bindings, which may declare tensors, such as an observation-noise scale.
• The total loss is Σ weight × the mean over usable entries. All terms share one dimension; nll is in nats, computed in canonical SI units, so log densities compare across policies.
• Metrics are evaluated per entry on the validation entries and averaged. They never affect training.
• The full rules are in Header blocks.

Gaussian targets#

A y column of kind Gaussian[T] holds a measurement {mean, sd} instead of an exact value (Gaussian columns). Against it:

• nll (and log_prob in a Form-2 term) integrates the measurement noise: a Gaussian prediction N(μ, σ²) scores −log N(mean; μ, σ² + sd²). A Mixture does this per component, a Grid convolves its density with the noise, Samples widen their kernel by it. With sd: 0 it is the ordinary nll.
• mse uses the mean, as do abs_error, sq_error and coverage; log_predictive_density integrates the noise like nll.
• linear's fit adds sd² to each entry's variance, so its fitted noise is the model's own, without the measurement noise.
• In other arithmetic in a Form-2 term, use its parts: y.length.mean, y.length.sd.

A Gaussian input works the same way from the other side. Here a tape reading with its own sd feeds the prediction, and the targets are noisy too; sigma_1 learns only what neither sd explains:

# parcel/tape_fix.sb — a box's length from a noisy tape readingpolicy(tape_est: Gaussian[Length]) -> (length: Gaussian[Length])optimiser adam(lr: 0.05, steps: 300)loss    { length: nll }metrics { err: abs_error(length, y.length) }sigma_1 = tensor(3 mm, lower: 0 mm, scale: 1 mm)length  = tape_est + noise(sigma_1)

Honest evaluation#

• Validate by group. With repeat scans, group_by the object. Scans of the same box are not independent, and a split that mixes them reports optimistic numbers.
• Early stopping never sees the validation fold. adam stops on an inner holdout taken from the training fold (holdout, by group).
• Every split starts fresh, from the declared initial values, never from earlier trained versions.
• Validation runs in hard mode, exactly as production does.
• Leakage warning. If a frozen import's tensors were trained on entries that are in the current validation fold, the run reports W0540 and lists them in the summary. Pin a version trained on other data, or import the policy unfrozen so its fits re-run on each training fold.
• Stacking needs out-of-fold inputs. stack chooses its weights from out-of-fold predictions: an inner K-fold (inner_k, default 5, by the same group_by) re-solves the delegate fits upstream of its components, so a model that merely nests another can't win by fitting the training data better. That costs inner_k extra fits of each delegate upstream, which the editor shows on the stack node's delegate badge and in its details (×K fits, K = inner_k). Stacks nest at most two deep (E0523). Only delegate fits are re-solved inside the inner folds; tensors trained by adam or cma_es stay at the current values.
• Fitted noise is not sensor noise. linear with method: "ml" (the default) accounts for each entry's own measurement uncertainty, so its σ measures what the model doesn't explain. method: "ols" folds sensor noise into σ.
• In-sample is labelled. Metrics from split none are marked in-sample everywhere they appear.