Calibration is a verifiable claim
A score that doesn't carry its skill is a marketing number. So we keep two questions apart, because they have different references and different failure modes. Meteorological verification asks whether a forecast variable matched a physical reference. Decision verification asks whether Goable's suitability probability matched what an operator later reported. The first is a physics check; the second is a behavioural one. Every audit-logged score carries the providerChain that produced it, so a degraded cell can be traced to a specific source mix, not just a location. The verification view is the source of truth for our own recalibration loop and for the publicly inspectable metrics below.
Two references, never merged
A forecast variable measured against a physical reference. ERA5 is a reanalysis: a consistent historical and regime baseline for backtesting, not a live truth. Real observations are a separate layer, used only where they are representative and fresh.
Goable's decision output measured against a consented, operator reviewed outcome. The outcome is an operational and behavioural signal, not a meteorological observation, and it carries its own caveat in every row.
An operational suitability outcome
The reliability question is not "when the engine says 70% favourable, how often did the activity run". It is: when the engine estimates a 70% probability of suitability, how often did comparable, weather-attributable sessions proceed under a documented outcome policy? That is an operational suitability outcome, not a clean forecast validation, because many things other than the weather decide whether a session runs.
- Demand and no-show
- Staff and equipment availability
- Operator policy and risk appetite
- Safety culture
- Group skill and composition
- Logistics and access
- Unobserved local conditions
- The operator having already acted on the forecast
What Brier, BSS and CRPS actually measure here
The binary event is weather_attributable_session_suitable. BS is the mean squared gap between the forecast probability and that outcome; BSS scales it against a climatological reference, so positive beats climatology and zero matches it. Unobserved sessions are labelled unobserved and excluded from the count, never treated as a negative. A cancelled session is split into cancelled_weather and cancelled_other by the operator's outcome policy, so demand cancellations do not read as forecast misses.
A cell's BSS is withheld below a documented minimum sample rather than shown as a noisy zero. Reliability bins are pooled over a rolling window and reported on held-out outcomes, per cell at horizons 24, 72 and 168h.
CRPS is computed over the ensemble suitability distribution where an ensemble exists, and reduces to absolute error against the observed outcome for a single-forecast cell. A 0 to 100 verdict is not itself a predictive distribution, so we do not claim CRPS over the verdict; it scores the distribution of ensemble suitability members against the realised outcome.
The confidence number decomposes
Every forecast-mode score carries a confidenceDetail block. The scalar confidence is a deterministic multiplicative fold of its components, so you can see exactly why a number is held down.
forecast_skillA horizon-decaying skill prior: ~0.90 nowcast, ~0.78 at 24h, down to ~0.51 at the 16-day limit. An honest prior today, to be replaced by measured per-stratum skill as outcomes accrue.provider_agreementAgreement across the consensus constituents. 1.0 for a single source — one provider is not penalised for being alone.profile_maturityProvisional 0.75, reviewed 0.85, calibrated 0.95. Rises as the profile is calibrated against outcomes.hierarchical_calibrationSub-spot 1.0 down to base 0.85, by how specific the resolved calibration level is.data_coverageFraction of the scoring weight that was backed by real data at score time.skill_calibrationOptional. Present only when a skill-conditioned curve was used.drift_flagSet when the drift monitor has flagged the cell. Confidence is held down while it is set.
Publicly inspectable, honestly bounded
Verification rows are published as line-delimited JSON at GET /v1/research/verification/export under CC BY 4.0. These are publicly inspectable verification metrics. Full recomputation is possible only for releases that include the documented input fields, label policy and methodology version.
The k greater than or equal to 10 contributor threshold, the 1 km grid generalisation and the 90-day lag are minimum disclosure controls, not a guarantee of anonymity under GDPR Recital 26. Rows are released only after a disclosure-control review, because a rare cell tied to a sub_spot_slug could still reveal a single operator's commercial signal. The public export prefers a grid cell over a sub_spot_slug, with a higher threshold before any sub-spot is exposed.
Coverage is a partial map while the platform is pre-revenue. As consented outcomes accumulate, cells move from provisional to reviewed to calibrated, and each score already carries a profile_maturity field so consumers know whether they are reading a calibrated number or an expert-set one.
Working on a verification paper?
If you are using the verification export in research or want a custom cut, the partnerships desk is the fastest path. We will set you up with a research API key and co-author the methods section when it helps.