Scoreboard

How far off is each model, on average, as the forecast reaches further ahead? Every forecast is compared with what the airport actually reported. Lower is better; a bias near zero means no systematic lean.

Typical error (RMSE) by lead time

Root-mean-square error. Grows as the forecast reaches further ahead.

Bias by lead time

Mean of forecast − observed. Above zero: too high / warm; below: too low / cold.

Summary

About sample sizes. n is the number of forecast–observation pairs behind each number. Thousands of pairs give stable scores; a few hundred can swing from day to day, and differences of a few hundredths between models are within the noise. Lead times with fewer than 30 pairs are left out of the charts, and rows with fewer than 100 pairs are greyed out. Small regions (Polar, Oceania) and rarely reported variables (gusts, sea-level pressure) have far fewer pairs than global temperature.

Head to head

Is one model really better than another, or is the gap just noise? Here two models are compared only on the forecasts both made — same airport, same hour, same forecast lead — and the difference comes with a 95% confidence interval.

Loading comparisons…

Difference A − B by lead time

RMSE of A minus RMSE of B on their common sample. Below zero: A is more accurate. Whiskers: 95% confidence interval.

By lead time

Common pairs, both scores and the difference with its 95% interval; last column: share of airports where A has the lower RMSE.

All pairs

Each cell: RMSE of the row model minus RMSE of the column model (all lead times, selected variable, window and region), each pair on its own common sample. Click a cell to open that pair above.

row model significantly betterrow model significantly worseno significant difference / interval not available

Ranking on one common sample

Every model scored on exactly the same forecasts: the hours and airports that all of them verified.

Why a common sample?

Models are not always verified on the same hours: one may have started later, skipped a run, or lack a gust forecast. A model scored on calmer days looks better than it is. Comparing only forecasts that both models made for the same airport, hour and lead time removes that advantage.

What the interval means

The 95% confidence interval is the range the true difference most plausibly lies in, given how much the result changes from day to day. If it does not include zero, the difference is significant: it is very unlikely to be luck. Otherwise we call it “no significant difference”.

How it is computed

Errors on the same day and in the same region move together (one weather system), so pairs are not independent. We resample whole days per region (runs of consecutive days, a block bootstrap, 2,000 times) to get the interval. With only a few days of data the intervals are rough. Details in DATA.md.

What do these numbers mean?

Bias

The average of forecast − observed. It tells you whether a model leans one way: +0.8 °C means it is, on average, 0.8 °C too warm. Errors in both directions cancel, so a model can have zero bias and still be poor.

MAE

Mean absolute error: the average size of the miss, ignoring direction. “On a typical hour this forecast is 1.4 °C off.” It is the easiest score to read.

RMSE

Root-mean-square error. Like MAE, but squaring the errors first makes big misses count more. Always ≥ MAE; a large gap between the two means occasional big busts.

Wind direction scores are circular (the error between 350° and 10° is 20°) and only use hours with at least 5 kt observed. See methodology for the matching rules.