FourCastNet (FCN) is a data-driven weather forecasting model that offers a 5,000x computational speedup over traditional numerical weather prediction (NWP) models. Trained on forty years of reanalysis data, FCN uses neural operators and spherical harmonic transforms to generate 0.25-degree horizontal resolution weather forecasts. In terms of annual global mean anomaly correlation coefficient, FCN matches the performance of the Integrated Forecasting System (IFS), a physics-based NWP model. Because this metric is averaged in space and time, it is unsuitable for assessing the accuracy and quality of extreme weather forecasts. Furthermore, a known challenge with machine learning models is that they produce smoothed forecasts. To evaluate the fidelity of FCN’s extreme weather forecasts, we present a comprehensive suite of extreme weather diagnostics.

First, we evaluate deterministic forecasts from FCN and IFS. We assess these models’ ability to successfully predict rare events, such as those greater than the 95th percentile of historical climate. We compare FCN’s and IFS’s performance on the tails of the distribution. Second, we evaluate perturbed ensemble forecasts from these two models. We show how our suite of metrics evaluates key features of an ensemble weather forecast: reliability, resolution, sharpness, skill, spread, and discrimination between extreme and typical weather. Finally, we compare FCN and IFS on case studies of extreme weather events with return periods greater than 10 years. As machine learning is rapidly adopted in weather forecasting, we present ways to apply our open-source extreme diagnostics to other data-driven weather models.

References: FourCastNet (https://arxiv.org/abs/2202.11214; https://arxiv.org/abs/2208.05419; https://arxiv.org/abs/2306.03838)