In this article, we discuss the important question of subseasonal forecasting skill, which means we are considering long-range forecasts with a lead time of three to six weeks ahead.
Subseasonal Forecasting Skill: Introduction
When we think about the skill of a short to medium range weather forecast, then we usually think in terms of Mean Average Error (MAE), or perhaps Mean Absolute Percentage Error (MAPE). The results can then be presented in terms of the average error for each day of lead time.
It is then often useful to compare the forecast skill to that of a relevant climatology. On average, what we find is that at around 12 days ahead, the skill of the forecast matches the skill of climatology, which means that at this lead time and beyond, the forecast is no longer providing an advantage over climatology or, in other words, the forecast appears to have no skill.
That being the case, how is it that skilful subseasonal forecasts are a reality?
Subseasonal Forecasting Skill: Forecast Windows
At lead times approaching 14 days ahead, forecast skill becomes so low, that it is no longer meaningful to consider individual days. It becomes necessary to aggregate skill over several days or, in the case of the World Climate Service, whole weeks.
When we do this, not only do we find forecast skill beyond 12 days ahead, but the World Climate Service has also been able to demonstrate useful skill in forecasts with lead times of up to six weeks ahead.
Subseasonal Forecasting Skill: The Value of Ensemble Forecasting
There is one more important way in which assessing subseasonal forecast skill is fundamentally different from the methods used to evaluate the skill of medium range forecasts.
Medium range skill measures are typically deterministic, which means the best available single forecast is compared against the finite number that was recorded in the observation. By taking this approach, some of the value in the initial forecast is completely overlooked.
Both medium range forecasts and subseasonal forecasts are arrived at by a process known as ensemble forecasting. The many individual forecasts (members) which make up an ensemble forecast, provide two dimensions to the forecast: one is the ensemble average, which is easy to use, as we can treat it in the same way as a single, deterministic, forecast. The other is the spread (or range) of the members of the ensemble forecast. The spread contains additional information which can improve the forecast if the data is processed in an appropriate fashion.
Subseasonal Forecasting Skill: Probability Forecasting
The low skill of a subseasonal forecast is such that it is simply too ambitious to forecast a single value precisely. What we can do, however, is to assign each member of the ensemble to a category or bin. The World Climate Service divides the ensemble members between three equally likely bins or terciles: above normal, near normal, and below normal (ANB).
By considering the number of forecasts that fall into each tercile, the WCS generates probability forecasts for each of the three terciles. For example, if it happens that there are 51 members in the ensemble, and the forecast places 17 members in each tercile, then each tercile has a 33.33 percent chance of occurring. This happens to be exactly the same probability we would expect by chance, and so in this case, the forecast would have no signal.
If however, the 51 members were distributed 25 / 17 / 9 into the A, N, and B terciles respectively, then the probabilities of each tercile percentage would be 49 / 33.3 / 17.7. We can see that above normal has a much greater chance of occurring than chance alone: 49 percent vs 33.3 percent. At the same time near normal would have a normal chance of occurring, and below normal would be much less likely to occur than normal: 17.7 percent vs 33.3 percent.
Subseasonal Forecasting Skill: Reliable Probabilities
When producing probability forecasts, it is essential that the forecasts are reliable. Forecasts produced by centers such as the ECMWF are not reliable at the point of delivery. It is only through a process of careful calibration, that the World Climate Service ensures the forecasts are reliable.
Reliable probability forecasts possess a magical quality in the sense that they are always correct at the rate indicated by the probability regardless of lead time. For example if a probability forecast of 40 percent occurred in lead week six, then it would still occur at least 40 percent of the time.
Subseasonal Forecasting Skill: Expressing Skill
The measure used to show subseasonal forecasting skill of probability forecasts is: Fraction of Forecasts Verifying as Correct. The benchmark is thus 33.3 percent as by chance alone, each of the ANB terciles would occur at this rate.
World Climate Service forecasts in which a tercile was predicted with greater than 33.3 percent proved to be correct 39-49 percent of the time (Figure 2). Thus the forecasts have skill because a system with no skill would only be correct 33.3 percent of the time.
The subset of forecasts in which a tercile was predicted with greater than 50 percent proved to be correct 51-69 percent of the time, which shows that these forecast probabilities are reliable – when you see a forecast of 50 percent, you can be confident that it will occur 50 percent (or more) of the time.
Figure 3 shows the same data as Figure 2 but this time for lead week six. Note that the CFSv2 was not reliable in Sep/Oct/Nov, although this may be due to the sample size being too small.
We can also see that the World Climate Service Multi-Model which combines the CFSv2 and ECMWF forecasts, is always better than any single model. This is one example as to how the World Climate Service provides additional value.
Subseasonal Forecasting Skill: Statistical Forecasts
Another source of subseasonal forecasting skill is provided by statistical forecasting. This method is completely independent of dynamical modelling. The World Climate Service Sub-R statistical model utilizes an automated algorithmic process which matches current weather patterns (such as; sea level pressure, sea surface temperature, land surface 2m temperature, ice concentration, and geopotential height at 500mb, 200mb, and 10mb) with historic patterns. Matches are then rolled forward in time to create a forecast.
The Sub-R model shows modest subseasonal forecasting skill at various times of the year, with lead times of three to six weeks. The Sub-R model becomes particularly valuable when it is combined with the WCS Multi-Model Ensemble. It has been shown that when the Sub-R and MME agree on the forecast tercile, then that MME forecast is much more likely to be correct.
Subseasonal Forecasting Skill: Analog Forecasts
Another important source of subseasonal forecast skill is analog forecasting. Analog forecasting is based on the characteristic weather associated with the various climate indices. If we have confidence that a particular climate index will be in a particular phase at some future date, then we can use the historic weather anomalies as a forecast.
Although it is difficult to quantify the skill of analog forecasting, this method is widely practiced in weather forecasting, and analog forecasting has, in many cases, proven to be just as successful as dynamical modeling forecasting.
The World Climate Service has created a powerful subseasonal analog system which enables users to instantly map historic anomalies for a wide range of climate indices.
Subseasonal Forecasting Skill: Conclusion
In the subseasonal timeframe, it is necessary to consider forecast windows of a week, rather than the individual days referenced in short to medium range forecasting. It is also useful to split the forecast into terciles, which enables all of the available information within the ensemble output to be captured.
By carefully calibrating model output, the World Climate Service is able to generate reliable probability forecasts. This means that the tercile forecasted will be correct at the rate indicated by the probability, regardless of lead time.
World Climate Service subseasonal probability and statistical forecasts possess skill out to lead week six, and are particularly skillful when used in combination.
The World Climate Service also enables users to effortlessly research analog forecasts and thus tap in to another valuable source of subseasonal forecasting skill.
The World Climate Service produces a twice-weekly subseasonal report as part of its Trading Markets product which is aimed at energy traders. Each report summarises the latest signals from all sources of information. Dynamical modelling is featured via the WCS Multi-Model Ensemble. A unique analog forecast is then presented along side the subseasonal model maps for comparison. The World Climate Service statistical Sub-R forecast is also referenced. By combining all sources of subseasonal forecasting skill, subscribers are presented with the most likely evolution across the lower 48 over the subseasonal period (lead time weeks three, four and five).