Drift between recalibrations
A scoring engine that recalibrates on a weekly or monthly cadence is silent about what happens in between. Most regime shifts (a marine heatwave, a sudden stratospheric warming, an unexpected snowpack collapse) play out in days: fast enough to change how activities get called, slow enough that nobody catches them until the next refit.
The value is plain. Do not wait for the next retraining to discover the model is wrong more than usual. Partners who price against these scores need us to detect sustained degradation on a daily monitoring cadence, and to keep an evidence trail that shows we did.
Two monitors, kept separate
Forecast-skill drift and decision-calibration drift are different questions and are tracked separately. What is active today is the decision-BSS monitor. The forecast-skill monitor is planned and gated on data quality.
Decision-calibration drift
Movement in the decision Brier Skill Score: Goable's suitability probability against the operator-reported outcome (ran or cancelled). This is what the live monitor watches. A decision-BSS decline can also come from non-weather causes (operator policy, user mix, demand, staffing, pricing), so a flag is a signal to investigate, not proof the forecast itself degraded.
Forecast-skill drift
Movement in the meteorological skill of the forecast itself, scored against a physical reference. Not active yet: it depends on a representative physical reference per cell, which is not established for most cells.
Only on cells it can evaluate
- Drift monitoring activates only after a cell has accumulated enough daily decision-BSS history to evaluate. Before that the cell is unmonitored, not silently passing.
- A flag reports the cell, the length of the decline and whether a recalibration candidate was triggered. It does not claim a sample size or confidence interval it does not have.
- The decision-BSS monitor needs operator-reported outcomes on the cell. A meteorological-skill monitor would additionally need a representative physical reference per cell.
- Pooling sparse cells into a hierarchical low-volume monitor is a future direction, not something implemented today.
What is implemented, honestly
The structured drift_flag ships on score responses today.
Runs only for cells with sufficient daily decision-BSS history to evaluate.
The platform is pre-revenue; paired operator outcomes are still accumulating, so coverage is partial.
Depends on a representative physical reference per cell.
Available to contracted and admin tenants.
A mapping reference for partner review, not a certification.
A severity ladder of actions
The monitor is a lower one-sided CUSUM over each cell's daily decision-BSS series (slack k = 0.5σ). Severity is which configured threshold h the cumulative statistic's peak crossed, not a single-day excursion. Each level below is defined by the response it triggers.
| Level | Trigger | Response |
|---|---|---|
| watch | CUSUM peak crossed the watch threshold. | Logged to the drift_events ledger and surfaced inline in the score response's drift_flag. No alert, no recalibration. |
| warning | CUSUM peak crossed the warning threshold. | Everything at watch, plus an operations alert and a scoped recalibration candidate for that cell only (no platform-wide refit). Scores on the cell carry an elevated drift_flag. |
| critical | CUSUM peak crossed the critical threshold; sustained, multi-day decline. | Everything at warning, plus paging escalation and automatic charter regeneration, stamped with the event. |
confidenceDetail.drift_flag
Every /v1/score response on a cell with an open drift event carries an inline flag. Your client reads it and decides: widen tolerances, route to a human reviewer, or simply surface "we are less confident than usual here" to the end user. No separate API call, no polling. Confidence is held down while the flag is set.
metric: "decision_bss" and reference_type: "operator_outcome" make explicit that this is decision-calibration drift against operator outcomes, not a pure meteorological forecast error.
Detect, candidate, validate, roll back
Detect
The daily CUSUM crosses a configured threshold on a cell. A drift_events row opens.
Candidate
A warning or critical launches a scoped recalibration candidate for that cell only. recalibration_triggered reports that the candidate was launched, nothing more.
Validate
The candidate curve is scored against a held-out Brier-Skill-Score gate. It qualifies only if it beats the incumbent curve.
Promote or roll back
It ships only on a pass. Otherwise the previous curve persists. A worse model is never silently served, and the safety gates are never altered by this loop.
A warning triggers a scoped recalibration candidate for the affected cell. Promotion requires automated held-out validation and remains rollbackable. The previous curve persists on a failing candidate.
An open event auto-resolves only after the CUSUM statistic stays below the watch threshold for five consecutive daily evaluations. Resolution sets resolved_at; the drift_events row is preserved with its opening, peak severity, actions, recalibration outcome and recovery evidence.
A model-monitoring record for partner review
The SPC charter is a versioned document covering the monitored cells, baseline windows, severity ladder, escalation and recalibration policy, and the event log for the requested range. Available in two formats.
Machine-readable
Versioned JSON-LD with a published schema, schema version, issuer, event ids, model / profile / provider versions, methodology version and integrity metadata. Ingestible by a partner's model-review tooling.
POST /v1/admin/drift/charter (Accept: application/ld+json)Reader-friendly
Pure-JS vector render (pdfkit) embedding the decision-BSS time series with the zero-skill reference line and the event annotations.
POST /v1/admin/drift/charter (Accept: application/pdf)Intended use: supporting evidence. The charter can be attached in a delegated-authority or model-review process. It does not itself enter a binding application, a filing or a pricing approval.
A mapping reference, not a certification
The charter is structured to support evidence relevant to model-performance monitoring and change-control review, mapped to selected model-governance expectations used in insurance and delegated-authority review. The framework names below are a mapping reference. Partners remain responsible for determining regulatory applicability and compliance.
Relevant to ongoing model-performance monitoring and model change management: an independent, automated monitor with severity-based escalation and a preserved event log.
Relevant to documenting the forecast-model risk component and the process that watches it.
Relevant to a documented process for detecting and responding to deteriorations in model performance, with the drift_events ledger as the supporting evidence.
Due-diligence support
If you are evaluating Goable as a parametric primitive provider, the SPC charter, the forecast-verification records and per-cell skill estimates can support technical due diligence for parametric products. They do not constitute a policy, a pricing approval, a regulatory filing or delegated authority, and Goable does not present itself as an authorised MGA or coverholder. The partnerships desk routes you to the right contracting and DPA shape.