Module II asked how we infer and update an uncertain state. Module III asks what a forecast distribution can support, and how trust changes near rare events, altered baselines, and unfamiliar conditions.
Module III · Exploration 9Predicting a Changing Planet
A probabilistic forecast does not select one future. It describes how plausibility is distributed across several possible futures, and makes a claim that can be tested only across many comparable cases.
Enter the Forecast Lab ↓
Watch first · short video
See why a forecast distribution can be more honest and more useful than a single predicted future.
chance of rain
A single best estimate hides uncertainty in the current state, future forcing, model parameters, and model structure. An ensemble makes some of that uncertainty visible by generating several plausible evolutions.
Thin paths show plausible futures; the band contains the central 80%.
This outcome may fall near the center or in the tail. One outcome does not verify the distribution. Verification requires repeated forecast–outcome pairs.
Leave bias or the missing pathway unchanged, then compare the same forecast design at three ensemble sizes.
More members reduce sampling noise. They do not repair model bias or restore possibilities omitted from the ensemble design.
Among many comparable cases assigned a 70% probability, the event should occur about 70% of the time.
forecast probability ≈ observed frequency
Calibration asks whether stated probabilities agree with what happens across repeated comparable cases.
A forecast has useful resolution when it gives different probabilities in situations whose event frequencies really are different.
Here “resolution” is a forecast-verification term. It does not mean spatial resolution.Calibration gives probabilities meaning. Resolution makes them informative.
About sharpness. For continuous forecasts, sharpness describes how concentrated a predictive distribution is. A narrow distribution is not automatically good—it must also remain calibrated.
Each verification case contains a forecast probability pᵢ for a defined event and an observed outcome oᵢ, where oᵢ = 1 if the event occurs and oᵢ = 0 otherwise. One case cannot tell us whether a probability system is trustworthy.
Repeat the forecast many times:
(p₁, o₁), (p₂, o₂), …, (pN, oN)
The diagnostics below ask different questions about these same forecast–outcome pairs.
Group cases with similar forecast probabilities. Within each group, calculate the mean issued probability p̄ and observed frequency ō, then plot (p̄, ō).
Small samples produce noisy estimates, especially for probabilities issued rarely.
Case 1: forecast 13%; event did not occur; Brier penalty 0.017.
The same 100 realized outcomes are used for both systems: 40 lower-risk cases (10%), 40 baseline cases (30%), and 20 higher-risk cases (70%). Their expected total event frequency is 30%.
System A produces one point near (30%, 30%). System B produces three points near (10%, 10%), (30%, 30%), and (70%, 70%).
Brier score on the same cases: System A 0.214 · System B 0.166. Lower is better.
If both systems are calibrated, why is System B more useful?
calibrated ≠ informative
The ideal forecast does both: its probability statements are calibrated, and they distinguish situations whose risks genuinely differ.
The Brier score gives a smooth squared penalty.
Proper scores reward honest probabilities, but different scores emphasize different mistakes.
For continuous outcomes, scores such as the continuous ranked probability score compare the full predictive distribution with the observed value.
Verification told us whether probabilities behave honestly. Decision problems ask something different: how should uncertain information be used when events are rare and mistakes have different consequences?
How often does the event occur before using this warning system?
Among event cases, what fraction receive a warning? Also called sensitivity or hit rate.
Among non-event cases, what fraction incorrectly receive a warning—not the fraction of warnings that are false.
Of all events, how many did we detect?80 / 100 = 80%
Of all warnings, how many were followed by the event?80 / 575 ≈ 14%
These percentages answer different conditional questions.
A warning system can detect most rare events and still produce many false alarms because the non-event population is much larger.
Usefulness depends on the base rate, warning-system performance, and consequences of misses and false alarms.
Suppose the forecast probability itself is already trusted. We now face a separate question: should we act?
Protection has cost C. Assume it avoids event loss completely.
Expected cost = CIf the event occurs, loss is L. With forecast probability p:
Expected loss = pLProtect when C < pL, equivalently p > C/L.
The same scientifically valid 20% probability can rationally support different actions because the actions have different costs and consequences.
forecast probability ≠ decision
The forecast supplies p. The decision threshold C/L comes from consequences.
It assumes one event, two actions, known costs and losses, full protection, no side effects, no uncertainty in cost estimates, and no competing users or equity considerations.
Its purpose is not to prescribe a real decision. It shows mathematically why a probability alone cannot determine an action.
Weather forecasts, seasonal outlooks, and climate projections condition on different information, operate over different horizons, and should be interpreted and evaluated accordingly.
Where might a storm be on Friday?
Which temperature category is favored next season?
How may mid-century statistics differ under a scenario?
Every summary preserves some information and hides something else. Choose the form that matches the question without pretending the discarded uncertainty has vanished.
Threshold probability. Useful when a specific consequence begins beyond a threshold.
A summary is a projection of the distribution, not a replacement for it.
A probability is a statement about repeated comparable cases. A well-calibrated 70% event should fail roughly 30% of the time.
Always predicting the climatological frequency can be calibrated and nearly uninformative. A useful forecast should add information without sacrificing calibration.
A forecast describes uncertainty. A decision also includes consequences, costs, values, resources, and risk tolerance.
Here, we treated a consequential event as a threshold and asked whether its probability was trustworthy. The next exploration moves into the tail itself: why some events become rare, amplified, or extreme, and why those mechanisms should not be confused.
Every forecast, outcome, regime, and warning case on this page is synthetic. The rainfall ensemble is a conceptual five-day system, not an operational weather forecast. The verification cases are independent and use stable statistical relationships unless the background-shift control is activated.
Random experiments use deterministic pseudo-random seeds so forecast systems can be compared on the same cases. Ensemble probabilities are direct member fractions. Reliability uncertainty bars are approximate sampling intervals. The warning grid reports expected category counts from the selected rates.