Forecast Calibration

What it is

A forecaster who says "thirty per cent chance of rain" is making a claim that can be checked, though not on a single day. Collect every day on which that figure was given. If the forecaster is calibrated, rain fell on about three in ten of them. The same test applies at every probability: ten per cent days should be wet about one time in ten, ninety per cent days about nine times in ten. The 1977 paper that this page rests on calls the property reliability.

Calibration concerns the match between stated odds and outcomes, and nothing else. It does not say the forecaster knew much. A forecaster who gave the long-run average chance of rain on every single day would be calibrated and of no use, because the forecast would never change from one day to the next. Daniel Kahneman, Olivier Sibony and Cass Sunstein make the same point in Noise, in a passage on forecasting tournaments (pp. 264 to 265): two forecasters can both be perfectly calibrated while only one of them is useful. The scoring rule they describe, the Brier score, dates to Glenn Brier in 1950 and rewards calibration and resolution together. Resolution is the ability to tell one situation from another, to say ten per cent on one day and ninety on the next and be right about the difference.

Weather forecasting is the standard example because it supplies what calibration needs: a great many forecasts of the same kind, and an outcome that arrives the next day. In 1977 Allan Murphy and Robert Winkler summarised American operational and experimental programmes in which forecasters gave probabilities for whether precipitation would occur and for daily maximum and minimum temperatures. Their conclusion was that weather forecasters can formulate such forecasts in a reliable manner. The forecasts were subjective probability forecasts, meaning forecasters' own judgements, so the result is about forecasters.

In effect

A retailer's weather-driven sales forecast is two forecasts chained together, and only the first is a probability forecast with a long record of being scored. The article What a Weather Forecast Does for Sales sets the 1977 result beside the retail studies.

In a 2020 study of daily sales in 673 stores, Florian Badorf and a co-author found that adding weather forecasts improved daily sales forecasts up to seven days ahead, with the benefit shrinking at longer horizons. In a 2017 study led by Sebastian Steinker, adding weather data to the forecasting models of a European online fashion retailer cut forecast errors by 8.6 to 12.2 per cent on average, and by up to 50.6 per cent on summer weekends, compared with a baseline that ignored the weather.

Those figures are reductions in error, not statements about calibration. The abstracts of the retail studies do not report whether stated probabilities held, and the sales forecasts are point estimates of units. The weather input can be trusted at its stated odds. The step from weather to units has to be estimated from the retailer's own sales, store by store and product by product, which is the subject of weather-sensitive demand. Forecasts of the long-run shape of the retail market have a different record, set out under retail forecasting.

What it does not say

It does not say a weather forecast is accurate in the sense of being sharp. Calibration is about honest odds. A forecast can be honest and vague.

It does not say the 1977 result holds today. It concerns American programmes of that period and subjective probability forecasts for precipitation and for daily maximum and minimum temperatures. It was read here at the level of its published abstract. It does not cover longer ranges, other variables, or the numerical models now used.

It does not say a calibrated input makes the downstream forecast good. In the retail studies the benefit of adding weather shrank with the forecast horizon, and how much it helped depended on the retailer, the season and the day of the week.

And it does not say weather forecasters are well calibrated about everything. The result is specific to the quantities they are scored on.

---