Separate calculator testing from execution-model validation

Three different review questions

First ask whether the formulas and units are implemented correctly. Next ask whether the model structure is appropriate for the intended use. Finally ask how its outputs compare with observed outcomes that were not selected to make it look successful. These questions require different evidence.

The Federal Reserve’s April 2026 model-risk guidance discusses conceptual soundness and outcomes analysis and supersedes earlier SR 11-7 guidance. It is a reference for this distinction, not a certification standard applied to this publication. The educational workbench has a tested formula and disclosed limits; it does not claim an empirically validated execution coefficient.

Meaningful arithmetic and boundary checks

For a hypothetical $1 million order, $25 million eligible volume, 3 bps fixed charges and k = 20 bps, expected participation is 4%, cost is 7 bps and dollar cost is $700. An independently written calculation should reconcile these values without calling the same function under test.

Other useful checks include zero order producing zero dollar cost, rejection of negative or non-finite inputs, and rejection of positive orders against zero volume. Doubling eligible volume should halve participation. Increasing AUM with fixed weights should increase order notional proportionally. These properties catch mistakes that a single default-case screenshot can miss.

Calibration needs a declared target

Suppose an empirical project seeks to estimate k. It must first define the outcome to be fitted: midpoint-relative slippage, implementation shortfall, or another metric. Fees and spread already included in that outcome cannot be added again without reconciliation. The sample needs intended orders, completed and incomplete outcomes, timestamps and compatible market observations.

Partition the data before adjusting the model, using a method appropriate to chronological dependence and strategy changes. An evaluation set used repeatedly to choose parameters becomes part of model development. Reserve genuinely untouched evidence for the final assessment and keep changes to the model or sample visible in a version history.

A diagnostic residual example

Imagine three hypothetical comparable orders with predicted costs of 6, 8 and 10 bps and observed costs of 5, 12 and 15 bps under the same definition. Residuals, observed minus predicted, are −1, 4 and 5 bps. Mean residual is 2.667 bps and mean absolute error is 3.333 bps.

Those three invented observations demonstrate calculations only. They cannot establish statistical reliability. In a real study, inspect residuals by size, participation, side, volatility, liquidity and completion status. A low overall average error can conceal systematic underestimation on the largest orders, precisely where a capacity conclusion is most sensitive.

Define action thresholds before seeing failures

A review process needs rules for missing data, out-of-range use, persistent bias and material structural changes. The response might be to limit use, seek additional evidence or recalibrate and re-evaluate. Record what triggered the action and which version remains valid for which purpose.

Do not describe successful browser interactions, unit tests or a clean deployment as independent strategy-validation samples. They establish that the publication works as designed. A live capacity claim additionally needs lawful, representative execution evidence and a defensible mapping from those observations to the proposed scale. Until then, retain the label 'illustrative scenario' even when the software is reliable.

Keep in the decision record

• Test arithmetic independently of the implementation.
• Choose the empirical target and evaluation sample before fitting.
• Software release checks are not independent execution-performance evidence.

Source notes

1. Federal Reserve — SR 26-2, Revised Guidance on Model Risk Management, 17 April 2026
2. Federal Reserve — Supervisory Guidance on Model Risk Management

Sources inform the definitions and limitations. Scenario inputs and arithmetic are publication illustrations, not extracted market observations. Access checks are dated in the source inventory.

Continue the analysis