By StationClosed · Updated · Sources and corrections
“80% accurate” is incomplete without knowing what was predicted, what counted as the reference and which cases were tested. Predicting an open station is different from giving useful advance warning of a closure. This guide explains the numbers without treating any one of them as a safety guarantee.
Current status and future warnings are different tests
A current-status model estimates open, advisory or closed now. Exact agreement measures how often its level matches the reference at that same time. It does not measure how much warning a driver received.
A closing-soon prediction names something expected to happen later. To grade it fairly, the predicted time window must finish before the outcome is known. The public warning accuracy report concerns the tracker's warnings and recorded outcomes; an internal comparison against an upstream status is a different evaluation. Neither is direct confirmation of what a station's staff actually did.
The reference itself matters. Agreement with an automated source can identify differences between systems, but both may miss a real-world closure. A driver report can add context, yet its time and meaning also need checking.
A worked example: 100 station checks
The following figures are invented for teaching. They are not StationClosed's live performance, and they simplify the task to closed versus not closed.
| Reference result | Predicted closed | Predicted not closed |
|---|---|---|
| Closed | 12 correct alerts | 8 missed closures |
| Not closed | 3 false alerts | 77 correct negatives |
- Exact agreement: 89%. The 12 correct alerts plus 77 correct negatives make 89 correct checks out of 100.
- Precision: 80%. Of the 15 closure calls, 12 matched the reference: 12 ÷ (12 + 3). It answers “when the model says closed, how often is that right?”
- Recall: 60%. The model caught 12 of the 20 reference closures: 12 ÷ (12 + 8). It answers “how many closures did it catch?”
- Critical success index (CSI): about 52%. This is 12 ÷ (12 + 3 + 8). It counts hits against hits, false alerts and misses, leaving correct negatives out.
The 89% headline can sound strong while the system still misses eight of twenty closures. Precision and recall make that weakness visible.
Why a 97% score can still mislead
Imagine a different sample in which 97 of 100 checks are open. A model that always says open has 97% agreement, but catches none of the three closures. That is why a target such as 97% agreement should not be pursued by suppressing difficult alerts or excluding missed events.
The reverse trade-off also matters: calling every station closed catches every reference closure but generates many false alerts. A useful improvement needs to show its effect on both false alarms and missed closures, not just one favourable percentage.
What a fair evaluation needs
- Unseen time periods. Tune on earlier data, then evaluate on later data that was not used to select the model. Do not let a closure observed this afternoon influence a prediction supposedly made this morning.
- The same cases for comparisons. A source covering only easy stations should not be compared directly with one covering every station. Compare their overlapping station-times and report coverage.
- Enough relevant events. Millions of mostly open checks are not millions of independent storms. Long-lived conditions produce repeated observations. Look at the number of closure events, days and regions represented.
- Separate hazards and levels. A model can be strong on cold advisories but weak on wind closures. Overall agreement hides that difference. Check closure recall and the cause of the missed calls.
- Clear handling of missing data. Unknown, delayed and unhealthy-source periods should be visible, not quietly counted as correct open cases.
The tracker now supports evidence-gated source selection for its advisory model. A change needs sufficient paired examples, an improvement in CSI and no loss of precision or recall on both earlier complete days and a later complete validation day. That guardrail limits unsupported source switching; it does not establish a particular achieved score or replace a broader held-out evaluation.
What to do with an accuracy number
Read its date range, prediction horizon, reference and sample size first. Ask whether it measures warnings like the one you are using. A country-wide average cannot promise the same result at one station during a rare event.
Use weather warnings to prepare and recheck conditions. Use the Flex app and station staff to confirm pickup arrangements. If you report a mismatch, include the station code, time zone, what was displayed and what you actually observed; do not turn an uncertain observation into a confirmed closure.
Related: Public warning results · Method and limits · How sources and corrections are handled