How StationTruth works

Weather apps rarely tell you how often they are wrong. StationTruth checks two of the world's main global forecast models — ECMWF and GFS — against what thousands of airports actually measure, every hour, and publishes the results. It then uses that track record to correct each model's habitual errors, and checks whether the correction really helps.

• In short
• Sources
• Matching
• Scores
• Bias correction
• Phase 2: Analysis
• TAF verification
• Limitations
• Licences & credits

In short

1. Every hour we collect the latest weather reports (METARs) from about 5,000 airports.
2. Twice a day (00 and 12 UTC runs) we fetch the forecasts of five global models for the next 72 hours — three physics-based (ECMWF IFS, NOAA GFS, DWD ICON) and two machine-learning “AI” models (ECMWF AIFS, NOAA AIGFS) — and read off the value at each airport.
3. When the forecast hour arrives, we compare: forecast − observed. That difference is the error.
4. Averaging those errors over 7 and 30 days tells us which model is better, where, and for which weather variable.
5. The average error of the last two weeks is subtracted from new forecasts. Those corrected forecasts (and a blend of all of them) are scored exactly like the raw ones.

Sources

Observations — METAR

Routine (METAR) and special (SPECI) airport reports from the aviationweather.gov data cache run by NOAA's Aviation Weather Center, collected hourly. Each raw report is decoded with MetarParse, our open-source METAR parser (the same library behind MetarParse Live). Where a US report carries the T remark group, temperature and dew point are taken to 0.1 °C instead of whole degrees; sea-level pressure comes from the SLPppp remark.

Forecasts — physics vs AI

Two kinds of global model are compared. Physics models solve the equations of the atmosphere on a grid; AI models (marked AI) are neural networks trained on decades of past analyses that step the weather forward directly. All runs at 00 and 12 UTC are used, out to 72 h.

• ECMWF IFS (physics): the Integrated Forecasting System high-resolution forecast from ECMWF's open data service, 0.25° grid, every 3 h.
• NOAA GFS (physics): the Global Forecast System at 0.25°, from the NOAA Open Data Dissemination bucket on AWS, every 3 h.
• DWD ICON (physics): Deutscher Wetterdienst's global model on its native ~13 km icosahedral (triangular) grid, from opendata.dwd.de, every 3 h. Each airport takes the distance-weighted mean of the three nearest grid cells.
• ECMWF AIFS AI: ECMWF's operational Artificial Intelligence Forecasting System (single deterministic run), open data, 0.25°, every 6 h. It has no gust field.
• NOAA AIGFS AI: NOAA's operational AI Global Forecast System (a GraphCast-based model initialised from the GFS analysis), 0.25°, every 6 h, from NOAA's AI weather-model bucket on AWS. It has no gust or cloud field.

Fields: 2 m temperature and dew point, 10 m wind (speed and direction from the u/v components), gusts (ECMWF 10fg, GFS surface gust, ICON hourly maximum VMAX_10M), mean-sea-level pressure, cloud cover. Values on the regular grids are bilinearly interpolated from the four surrounding grid points to each station. The AI models output every 6 h, so they are scored at 6-hourly lead times; a model without a variable is simply not scored for it.

Matching forecasts to observations

• A forecast valid at time T is paired with the observation at the same station closest to T, within ±30 minutes. If a routine METAR and a SPECI are equally close, the METAR wins.
• Variables verified: temperature, dew point, wind speed; gusts only when both the forecast and the observation report one; pressure only where the station reports sea-level pressure (the forecast MSLP is compared with the observed SLP, not with QNH); wind direction only when the observed wind is at least 5 kt, using the circular difference (350° vs 10° is a 20° error).
• Error = forecast − observed. Positive means the model was too warm / too windy / too high.
• All times are UTC. Units: °C, knots, hPa, degrees.

On the map, “Error now” compares the latest observation with the forecast for that hour from the most recent run that covers it, linearly interpolated in time between the 3-hourly steps.

Scores

For every model, region, variable and lead time we report the number of pairs n and three numbers built from the errors e = forecast − observed:

• Bias = mean(e) — does the model lean one way?
• MAE = mean(|e|) — how far off is it on a typical hour?
• RMSE = √mean(e²) — like MAE, but large misses weigh more. RMSE² = bias² + variance of the error, so removing bias can only lower RMSE if the bias is stable.

Windows are the last 7 and 30 days of verified pairs. Regions: Europe, North America, Central America & Caribbean, South America, Africa, Middle East, Asia, Oceania and Polar (|latitude| > 66.5°). The “best model here” on the map is the model with the lowest 7-day temperature RMSE at that station among those with enough verified pairs.

Bias correction (with no look-ahead)

Many forecast errors are systematic: a model grid cell is on average higher than the airport, a coastal station is mixed with sea, a valley gets colder at night than the smooth model terrain allows. These repeat day after day and can be learned.

For each model, station, variable, lead-time bucket (0–24, 27–48, 51–72 h) and time-of-day bucket of the valid time (00–05, 06–11, 12–17, 18–23 UTC), the correction is the mean error over the previous 14 days of verified pairs whose run started before the run being corrected. Only information that was available when the forecast was issued is used — no look-ahead — so the corrected forecasts are scored fairly, as if they had been issued in real time.

• Each layer of the estimate is weighted by its own sample size (shrinkage, n/(n+k)), so a correction starts small after the first verified run and grows as evidence accumulates, instead of switching on at a fixed sample size.
• Every model <id> has a corrected twin <id>_bc (ecmwf_bc, gfs_bc, icon_bc, aifs_bc, aigfs_bc). blend is the average of all corrected forecasts available for that hour (each from its model's latest run, at most 12 h old); at 3-hourly lead times between the AI models' 6-hourly steps it averages the physics models only.
• Corrected forecasts are verified exactly like the raw ones, so the scoreboard shows whether the correction actually helps — and where it doesn't.

Phase 2: Analysis

The Analysis page goes from points to a map: it combines the latest airport observations with the model forecasts to estimate the current weather everywhere on a regular grid, not only at the stations. Where airports are dense the analysis follows the observations closely; far from any station it falls back towards the model. It shows how the model's errors are organised in space — a cold valley, a too-windy coast — and is built only from data that were available at the time.

TAF verification

Airports also issue their own human-written forecasts, TAFs. The TAFs page checks them against what the airport then reported — wind, visibility, cloud and weather — and puts them side by side with the global models at the same airports, so you can see where the forecaster's local knowledge beats the raw model and where it does not.

Limitations

• Point vs grid. A 0.25° model cell (about 25 km) represents an area average; an airport thermometer measures one spot. Some “error” is simply that difference, especially on coasts, in mountains and around cities. It is still the error a user of the raw model at that place would see.
• No height correction. Model terrain and station elevation can differ by hundreds of metres. We deliberately do not apply a lapse-rate adjustment: raw scores show the model as delivered, and the bias correction learns the height difference along with everything else.
• Airports only. Stations are airports, mostly in flat, open terrain, and they are densest in North America and Europe. Global averages are weighted by where airports are, not by area or population.
• Observation quality. Outside the US, METAR temperatures are rounded to whole degrees, and occasional bad reports slip through. Gust and sea-level pressure reporting differ by country, so those samples are smaller and regionally biased.
• Simple correction. A fixed 14-day mean cannot follow sudden regime changes (a cold outbreak, snow cover appearing) and does nothing about random errors. It is a baseline, not a state-of-the-art post-processing system.
• Short history. Scores over a few days, or for small regions, carry real sampling noise — check n before reading much into small differences.
• Not for operational use. StationTruth is an independent, experimental project. Do not use it for aviation, navigation or safety decisions; use official forecasts and briefings.

Licences, attribution & credits

ECMWF open data

Copyright © 2026 European Centre for Medium-Range Weather Forecasts (ECMWF). Source: www.ecmwf.int. This data is published under a Creative Commons Attribution 4.0 International licence (CC BY 4.0). This covers both ECMWF IFS and ECMWF AIFS forecasts. StationTruth has modified the data: it is interpolated to station locations and, for ecmwf_bc, aifs_bc and blend, bias-corrected. ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.

DWD ICON

Source: Deutscher Wetterdienst. ICON forecasts are taken from the DWD open-data server under DWD's terms of use; StationTruth has modified the data (interpolated to station locations and, for icon_bc and blend, bias-corrected).

NOAA GFS, AIGFS and aviationweather.gov

GFS and AIGFS forecasts and METAR observations are produced by the US National Oceanic and Atmospheric Administration (NOAA) and are in the public domain. NOAA does not endorse this project.

Software

StationTruth is open source under the MIT licence: github.com/JerryWoof/StationTruth. Observations are decoded with MetarParse (MetarParse Live shows every airport's current weather). Data collection, verification and this site run on GitHub Actions and GitHub Pages and are regenerated every hour. Map: Leaflet with Esri Canvas tiles (© Esri, HERE, Garmin, © OpenStreetMap contributors); charts: uPlot. No trackers, no cookies.