Exploration 6 reconstructed a hidden snapshot from incomplete measurements. Exploration 7 asks what changes when the hidden state keeps moving while new observations arrive.
Module IILearning from an Incomplete Planet
Exploration 7 · When Models Meet Data
How should an uncertain forecast change when an uncertain observation arrives?
A model can carry a complete estimate forward, but its uncertainty grows. A new measurement brings evidence, but it is partial and uncertain too. Data assimilation combines the two, produces an updated estimate, and then begins the cycle again.
A spacecraft does not stop while engineers decide where it is. Between sightings, a navigation model must carry an uncertain state forward.
One number is not enough to describe that moving state. Three numbers locate the spacecraft in space, and three more describe how quickly it is moving in each direction. Together they form a state vector: a compact list of the quantities needed to continue the calculation.
As the model carries the estimate forward, small errors accumulate and uncertainty grows. An optical sighting may provide a direction. A radio signal may provide a distance or show how quickly that distance is changing. Neither measurement reveals the entire state, and neither is perfectly precise.
The navigation system does not replace the forecast with the measurement. It first calculates what the instrument should have reported if the forecast were correct. Their mismatch provides evidence about how the forecast may be wrong.
The corrected estimate usually lies between forecast and observation. How far it moves depends on the uncertainty in both. That updated estimate becomes the starting point for the next forecast.
Propagation is continuous. The update occurs when an observation arrives.
An observation does not replace an evolving estimate. It changes it.
This forecast–compare–update cycle organizes the rest of the exploration. The same logic appears in weather prediction, ocean-state estimation, and every problem in which models and measurements must be combined repeatedly through time.
The question
How should an uncertain model forecast and an uncertain observation be combined to estimate a changing state?
You will answer this in two steps.
A · One state: How far should an uncertain forecast move toward an uncertain observation?
B · Connected state: How can observing one variable update another variable that was never measured?
Experiment
Forecast, observe, update
The forecast provides a complete estimate with uncertainty. The observation supplies partial and noisy evidence. Your task is not to choose one and discard the other, but to decide how they should be combined.
Conceptual synthetic modelNot an operational navigation, weather, or reanalysis system
One state · one update
How far should the estimate move?
Forecast and observation are two uncertain distributions on the same state axis. Apply the update to see how their relative precision determines the compromise.
Outline style and direct labels distinguish all three sources, not color alone.
Forecast–observation mismatch2.55
Weight on observation85%
Updated estimateApply
Updated uncertaintyApply
The mismatch determines the direction of the correction. Relative uncertainty determines how far the estimate should move.
What changedAdjust the two estimates, then apply the update. The synthetic truth remains hidden until afterward.
The forecast–observation mismatch is often called the innovation.
Show the mathematics
K = Pf / (Pf + R)
ma = mf + K(y − mf)
Pa = (1 − K)Pf
The innovation y − mf is the difference between the observation and the forecast. The gain K determines how much of that difference is applied. Greater forecast uncertainty produces a larger correction; greater observation uncertainty produces a smaller correction.
Reflect
What changed your estimate?
How far did the observed state move?Begin with either experiment. Your latest settings will generate a concise interpretation here. The innovation sets the direction; forecast and observation uncertainty determine how far the state moves.
Earth connection
A changing planet is estimated the same way
The spacecraft example contains the basic logic, but Earth-system estimation is vastly larger. Forecast models carry millions of connected variables forward; balloons, aircraft, satellites, ships, radar, and ocean floats provide partial and uncertain evidence.
A · Weather analysis
Building the atmosphere now
A forecast supplies a complete but imperfect atmospheric state. The system calculates what each instrument should have measured, compares that value with the report, and uses modeled relationships to update connected variables.
An analysis is the updated estimate produced after observations are assimilated.B · Reanalysis
Reconstructing a history
A reanalysis repeats the cycle through an archive of past measurements using a broadly consistent system. It produces continuous fields where observations were incomplete, but it is a model–data reconstruction, not a global instrument or literal record of direct measurements.
A smooth historical map can contain information carried by the model across places, variables, depths, and times.
C · Diagnostics
When disagreement repeats
One large innovation can reflect a rare event, instrument error, an imperfect observation operator, model error, or a mismatch of scale. Repeated patterns may reveal bias, overconfidence, memory in errors, or an incorrect information pathway.
Innovations are clues, not verdicts.
Innovation statistics alone do not identify one unique cause.
Observations through time
Two ways to use observations through time
The difference between filtering and smoothing is which observations are allowed to inform the state at a particular time.
t0
t1observation
t2observation
t3observation
t4observation
t5observation
Filtering
Estimate the present using observations available up to that time.
p(xt | y1:t)
Past observations + present observation → present estimate. Later observations are not used because they have not arrived yet.
FilteringTarget: nowUses data available up to nowReal-time estimation and prediction
SmoothingTarget: an earlier timeUses data before and after that timeRetrospective reconstruction and reanalysis
Representing uncertainty
How can uncertainty be represented?
A model state is uncertain, so instead of carrying only one best estimate we can carry many plausible states.
01
Represent uncertainty
The spread of ensemble members approximates forecast uncertainty.
02
Propagate uncertainty
Advance every member through the model; the cloud may stretch, rotate, grow, or shrink.
03
Estimate relationships
Variables that vary together across members provide a pathway for one observation to update another variable.
Filtering and smoothing describe how observations are used in time.Ensembles describe one way to represent and propagate uncertainty and estimate relationships.An ensemble method can be used for filtering, smoothing, or both.
The wind–pressure relationship explored above can be estimated from an ensemble. Ensembles are useful, but they are not the only way to represent uncertainty.
What this model leaves out
A consistent update still depends on its assumptions.
This Lab uses linear relationships, Gaussian uncertainty, and a very small synthetic state. Real observing systems can be nonlinear, indirect, irregular, and biased. Observation errors may be correlated. Model errors may persist through time. Forecast relationships estimated from a finite ensemble may be noisy or false.
A mathematically consistent update is optimal only within its assumptions. If the observation model, uncertainty estimates, or information pathways are wrong, the system can become more confident while becoming less accurate.
Operational weather analyses and reanalyses add quality control, nonlinear observation operators, bias correction, localization, inflation, model-error treatment, and large numerical solvers.
What to remember
Four ideas to carry forward
01
Models and observations have complementary strengths.
Models connect variables, places, and times. Observations anchor an estimate to the physical world.
02
Both sources are uncertain.
An observation is evidence, not an exact replacement for the forecast.
03
The update occurs when information arrives.
Between observations, the model propagates the state and its uncertainty. At an observation time, the estimate is revised.
04
Information follows modeled pathways.
Those pathways can recover hidden variables, or spread a correction incorrectly.
Data assimilation is navigation through uncertainty: forecast, observe, update, and continue.
Sources, method, and synthetic-model note
R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” 1960.
E. Kalnay, Atmospheric Modeling, Data Assimilation and Predictability, 2003.
G. Evensen, “The Ensemble Kalman Filter: Theoretical Formulation and Practical Implementation,” 2003.
H. Hersbach et al., “The ERA5 Global Reanalysis,” 2020.
NASA historical material on Apollo onboard and midcourse navigation, including Kalman-based navigation procedures.
Every state, observation, uncertainty, and truth value on this page is reproducible synthetic teaching data. The one-state experiments use a scalar linear Gaussian update. The sequential experiment uses a small evolving state with seeded disturbances and a retrospective linear smoother. The connected-state experiment uses a two-variable Gaussian forecast in which cross-covariance carries a wind update to unobserved pressure. These examples isolate the logic of data assimilation; they are not operational spacecraft, weather, or reanalysis systems.
Continue the argument
The remaining uncertainty immediately begins to move
An update may reduce uncertainty now, but it does not eliminate it. The next exploration follows what the remaining uncertainty means for the future and how it becomes a probabilistic forecast.
