Meteorology and weather forecasting are still relatively young scientific endeavors, but the tremendous value of accurate forecasts has allowed a rapidly growing global weather industry to develop mature, well-established tools and techniques that produce today’s forecast information. However, recent rapid advances in Machine Learning are promising dramatic changes in the way forecasts are produced, and may prove to be the most significant fundamental change in meteorological practice in many decades.
The Established Orthodoxy
Ever since the pioneering work of Vilhelm Bjerknes and Lewis Fry Richardson, the modern enterprise of weather prediction has been heavily focused on developing faster and better ways of solving the fundamental equations of atmospheric motion and thermodynamics, and on interpreting and applying the results of these calculations. Modern supercomputers like those at ECMWF and NOAA perform many billions of calculations per second to look forward in time, repeatedly calculating the three-dimensional evolution of the atmosphere according to the laws of nature.
The process is intrinsically simple: physics tells us how the atmosphere works in space and time, and we use enormous computers to apply the (simplified) rules over and over again, advancing a few seconds into the future with each step.
Using this recipe, and relying on large financial investments by mostly governmental organizations, weather “modelers” have been able to achieve the seemingly impossible: predict the weather with a remarkable degree of accuracy from hours to days ahead, depending on the application. Moreover, by running the same process many times with small initial differences, the ensemble of solutions provides extremely useful information about the range of possibilities and the uncertainty in the forecast. Nearly the entire modern weather enterprise is built around these “dynamical” ensemble forecasts, and large segments of the global economy rely critically on this information.
But now, in 2023, a change is underway that seems likely to radically reshape the nature of weather prediction in the years ahead. It can justifiably be called a revolution in weather forecasting, and it seems to have emerged in the blink of an eye.
The new challenge to the established orthodoxy of weather prediction is Machine Learning.
Machine Learning in Weather Forecasting
In its broadest sense, Machine Learning refers to the paradigm of using a machine (computer) to discover statistical rules for converting input data into corresponding outputs. The problem could be as simple as tagging images: the computer is presented with a large set of images and corresponding tags, and it “learns” how to assign the right tag based on the content of the images.
Weather prediction is a rather obvious candidate for Machine Learning (ML). Firstly, the problem is fundamentally simple: given today’s weather, what is tomorrow’s weather going to be? And secondly, we have a fairly long (50+ years), detailed, and accurate history of global weather data that provides many examples of today’s weather paired with tomorrow’s weather. Just put the computer to work to learn the rules of forecasting.
Notice that the ML approach to weather prediction is completely different from the traditional physics-based method. Rather than using physical laws to tell us what “must” happen next with the weather (irrespective of the past), ML uses the history of what “did” happen in the past to look forward from the present time. (However, it’s worth noting that ML does actually rely on the traditional modeling methods, because it’s the traditional models that provide the accurate history as well as the current best-guess of the state of the atmosphere.)
From a practical standpoint, a major advantage of Machine Learning in weather forecasting is the massively reduced time to create a ML forecast. Current state-of-the-art physics-based models take hours to run on some of the world’s largest computers, consuming a lot of energy, but an ML-based forecast can be generated in mere seconds on a standard GPU computer. (Developing the ML model in the first place is another matter entirely, of course, but once it’s created the forecasts are simple to generate.) The enormous time and cost savings of running ML models is a key aspect that will help revolutionize the weather enterprise.
2023: The State of Play
Despite the seeming simplicity of the Machine Learning approach to weather prediction, the science of artificial intelligence and ML is still a young field, and it is only in recent years that serious attempts have been made to develop comprehensive weather forecasts with ML methods. In 2021, ECMWF published a “roadmap” for the future of ML at ECMWF, but even then the rapid pace of change was not anticipated. But then, as documented in the following ECMWF blog post, everything changed with the publication of groundbreaking results from large technology companies such as NVIDIA and Google:
Responding quickly to the new developments, ECMWF began publishing forecast graphics from three of the leading ML models in September 2023, and they also released software so that anyone can run the models and explore the forecasts for their own applications. With this open approach to the new ML methods, innovation will continue to accelerate.
Hurricane Lee Forecasts
With realtime ML forecasts now freely available from ECMWF, Hurricane Lee of September 2023 provided a natural opportunity to evaluate the forecasts in a forward-looking mode. Beginning on September 6, we documented the ML predictions of Hurricane Lee in a Twitter thread:
All three of the latest @ECMWF #MachineLearning forecasts are showing #HurricaneLee perilously close to the North American coastline 10 days from now.
This is going to be a *very* interesting real-time test of the new technology. pic.twitter.com/IpmAAFQ0Zd
— World Climate Service (@WorldClimateSvc) September 6, 2023
At the time of writing, Hurricane Lee is just 24 hours away from approaching the North American coastline near the western tip of Nova Scotia (although the storm will likely no longer be classified as a hurricane at landfall). The final path of the storm northward toward landfall, rather than northeastward across the open Atlantic, is a notable success for the Machine Learning models, which have consistently signaled a landfall somewhere between Cape Cod and eastern Nova Scotia. Check out the progress of the ML forecasts in the Twitter thread above.
Physics versus Machine Learning
The potential value of the ML forecasts is highlighted by the fact that the leading ensemble models (i.e. traditional forecasts) showed great uncertainty about the eventual fate of Lee at a lead time of 7-10 days. For example, the September 6 ECMWF ensemble showed a very wide range of solutions, and the September 8 forecast showed about half of the members passing to the east of Nova Scotia.
The latest ECMWF weather model tracks for Lee 🌀 https://t.co/4QEEkHvAuR pic.twitter.com/2zSuNaz5NF
— ECMWFbot (@ECMWFbot) September 6, 2023
The latest ECMWF weather model tracks for Lee 🌀 https://t.co/4QEEkHvAuR pic.twitter.com/CMhAQYATd8
— ECMWFbot (@ECMWFbot) September 8, 2023
It is important to emphasize, however, that this small sample of forecasts from the ML models can’t be used by itself to prove that they are already “better” than the traditional ensemble models. The ML method may have been “lucky” in this case, while other cases may be far less amenable to the ML approach. It’s also very likely that some aspects of the ML forecasts are unrealistic, because the models are not constrained by physical laws; they are statistical solutions, not real-world solutions. Nevertheless, published results have amply demonstrated the strong capability of the new ML models in direct comparison to traditional methods, and Hurricane Lee provides an illustration of the potential in real time.
Today’s Forecast: A High Probability of Machine Learning
In summary, weather prediction using Machine Learning has suddenly emerged as a legitimate competitor to traditional physics-based methods, and publicly-available ML forecasts are attracting attention as potentially valuable tools for real-world forecasting problems at this time. Given the remarkable pace of innovation in just the past few years, it seems likely that major ML advances still lie ahead, and it is not difficult to imagine that ML will be able to replace the physical models for at least some applications in the relatively near future.
Will weather forecasts ultimately be revolutionized by ML?
From the perspective of the forecast developers and the global weather enterprise, the answer is undoubtedly yes – the revolution is already under way, and is proceeding at breakneck speed.
From the perspective of the end user, the answer is less clear. The quality and accuracy of weather forecasts may not improve dramatically, at least not yet, but the potential is exciting. If nothing else, the last few years have clearly demonstrated that technological advances are difficult to anticipate, and the future may surprise us all.