Qualification

qualification

Opt-in, eager numerical qualification of compiled RHS functions.

This module checks user-supplied fixtures against independent equations and optional balance and JAX directional-derivative checks. It does not replace structural validation or establish scientific validity or integration accuracy.

ArrayComparison

Bases: NamedTuple

Reference parity, shape/dtype preservation, and finite-value diagnostics.

FixtureQualification(fixture, evaluator, backend, time, rtol, atol, arrays=(), invariants=(), derivative=None, errors=()) dataclass

Qualification of one named fixture through one evaluator.

All diagnostics are host values: this report is an eager authoring tool, even when the evaluated RHS uses JAX or Torch arrays.

ok property

Whether every requested check passed.

InvariantCheck

Bases: NamedTuple

A residual-to-zero balance check with explicit applicability conditions.

NumericalCheck

Bases: NamedTuple

A named numerical check and any informative failure messages.

QualificationConfig(rtol=1e-07, atol=1e-09, derivative_step=1e-05, derivative_rtol=0.0001, derivative_atol=1e-06) dataclass

Reference and derivative tolerances for eager qualification.

Defaults assume double precision. Choose wider tolerances and a larger finite-difference step for float32 fixtures.

__post_init__()

Validate controls.

Raises:

Type	Description
ValueError	If a tolerance or finite-difference step is invalid.

Source code in src/op_system/qualification.py

def __post_init__(self) -> None:
    """Validate controls.

    Raises:
        ValueError: If a tolerance or finite-difference step is invalid.
    """
    for name in (
        "rtol",
        "atol",
        "derivative_rtol",
        "derivative_atol",
        "derivative_step",
    ):
        value = getattr(self, name)
        numeric = isinstance(value, Real) and not isinstance(value, bool)
        if (
            not numeric
            or not math.isfinite(value)
            or value < 0
            or (name == "derivative_step" and value == 0)
        ):
            bound = "positive" if name == "derivative_step" else "non-negative"
            msg = f"{name} must be finite and {bound}"
            raise ValueError(msg)

QualificationFixture

Bases: NamedTuple

Named structured state/parameter fixture and optional state direction.

QualificationReport(fixtures) dataclass

Inspectable results for all requested fixture/evaluator combinations.

ok property

Whether all fixtures and requested evaluators qualified.

to_dict()

Return JSON-compatible fields, including per-fixture pass/fail status.

Returns:

Type	Description
dict[str, object]	A mapping suitable for json.dumps(..., allow_nan=False).

Source code in src/op_system/qualification.py

def to_dict(self) -> dict[str, object]:
    """Return JSON-compatible fields, including per-fixture pass/fail status.

    Returns:
        A mapping suitable for ``json.dumps(..., allow_nan=False)``.
    """
    return {
        "ok": self.ok,
        "fixtures": [
            {
                **asdict(item),
                "ok": item.ok,
                "arrays": [value._asdict() for value in item.arrays],
                "invariants": [value._asdict() for value in item.invariants],
                "derivative": item.derivative._asdict()
                if item.derivative
                else None,
            }
            for item in self.fixtures
        ],
    }

qualify_rhs(compiled, fixtures, reference, *, evaluators=('flat', 'structured'), config=None, invariants=())

Compare compiled layouts against independently supplied equations.

Parameters:

Name	Type	Description	Default
compiled	CompiledRhs	Compiled RHS to inspect; structural validation is separate.	required
fixtures	Sequence[QualificationFixture]	Named structured state/parameter fixtures. Supply a JAX state direction to request an additional derivative check.	required
reference	ReferenceRhs	Independent reference(t, state, params) returning derivatives keyed by the same state templates.	required
evaluators	Sequence[Evaluator]	Requested layouts. Unavailable layouts produce failed records rather than silently falling back to another evaluator.	('flat', 'structured')
config	QualificationConfig | None	Reference and derivative tolerances. Defaults assume double precision; adjust the tolerances and difference step for float32.	None
invariants	Sequence[InvariantCheck]	Residual-to-zero callbacks with stated conditions and individual absolute tolerances, such as closed-system balance.	()

Returns:

Type	Description
QualificationReport	Eager host diagnostics for every requested combination. Fixture-based
QualificationReport	agreement is not a proof of scientific validity or solution accuracy.

Raises:

Type	Description
ValueError	If fixtures/evaluators are empty, fixture names/time are invalid, or an invariant lacks valid conditions/tolerances.

Examples:

>>> import numpy as np
>>> from op_system import compile_spec
>>> compiled = compile_spec({
...     "kind": "expr",
...     "state": ["x"],
...     "equations": {"x": "-rate*x"},
... })
>>> fixture = QualificationFixture(
...     "decay", {"x": np.array(4.0)}, {"rate": 0.5}
... )
>>> def reference(time, state, params):
...     return {"x": -params["rate"] * state["x"]}
>>> report = qualify_rhs(compiled, [fixture], reference)
>>> report.ok
True
>>> len(report.fixtures)
2

Source code in src/op_system/qualification.py

def qualify_rhs(  # ruff: ignore[too-many-arguments]
    compiled: CompiledRhs,
    fixtures: Sequence[QualificationFixture],
    reference: ReferenceRhs,
    *,
    evaluators: Sequence[Evaluator] = ("flat", "structured"),
    config: QualificationConfig | None = None,
    invariants: Sequence[InvariantCheck] = (),
) -> QualificationReport:
    """Compare compiled layouts against independently supplied equations.

    Args:
        compiled: Compiled RHS to inspect; structural validation is separate.
        fixtures: Named structured state/parameter fixtures. Supply a JAX
            state ``direction`` to request an additional derivative check.
        reference: Independent ``reference(t, state, params)`` returning
            derivatives keyed by the same state templates.
        evaluators: Requested layouts. Unavailable layouts produce failed
            records rather than silently falling back to another evaluator.
        config: Reference and derivative tolerances. Defaults assume double
            precision; adjust the tolerances and difference step for float32.
        invariants: Residual-to-zero callbacks with stated conditions and
            individual absolute tolerances, such as closed-system balance.

    Returns:
        Eager host diagnostics for every requested combination. Fixture-based
        agreement is not a proof of scientific validity or solution accuracy.

    Raises:
        ValueError: If fixtures/evaluators are empty, fixture names/time are
            invalid, or an invariant lacks valid conditions/tolerances.

    Examples:
        >>> import numpy as np
        >>> from op_system import compile_spec
        >>> compiled = compile_spec({
        ...     "kind": "expr",
        ...     "state": ["x"],
        ...     "equations": {"x": "-rate*x"},
        ... })
        >>> fixture = QualificationFixture(
        ...     "decay", {"x": np.array(4.0)}, {"rate": 0.5}
        ... )
        >>> def reference(time, state, params):
        ...     return {"x": -params["rate"] * state["x"]}
        >>> report = qualify_rhs(compiled, [fixture], reference)
        >>> report.ok
        True
        >>> len(report.fixtures)
        2
    """
    controls = config or QualificationConfig()
    if not fixtures or not evaluators:
        msg = "at least one fixture and evaluator are required"
        raise ValueError(msg)
    names = [item.name for item in fixtures]
    if len(set(names)) != len(names) or any(not name.strip() for name in names):
        msg = "fixture names must be unique and non-empty"
        raise ValueError(msg)
    if any(not math.isfinite(item.time) for item in fixtures):
        msg = "fixture times must be finite"
        raise ValueError(msg)
    if any(name not in {"flat", "structured", "block"} for name in evaluators):
        msg = "unknown evaluator"
        raise ValueError(msg)
    if any(
        not item.condition.strip() or not math.isfinite(item.atol) or item.atol < 0
        for item in invariants
    ):
        msg = "invariant checks need conditions and finite non-negative tolerances"
        raise ValueError(msg)
    return QualificationReport(
        tuple(
            _qualify_fixture(
                compiled, item, reference, evaluator, controls, invariants=invariants
            )
            for item in fixtures
            for evaluator in evaluators
        )
    )