OVERALL PERFORMANCE MEASURES
4th QUARTER METRIC COMPLETED:
Evaluating the U.S. Water Cycle in Global Kilometer-Scale Simulations Produced on an Exascale Computer
Product Definition
Global kilometer-scale models have emerged in the last decade (Stevens et al. 2019), holding great promise to improve modeling of water cycle processes and extreme events that have large societal impacts. By explicitly resolving deep convection, which is ubiquitous over the tropics and in the midlatitudes during the warm season, kilometer-scale models can produce more realistic probability distributions of rain rates, reducing the over-abundance of drizzle and the low-bias of intense rainfall in lower-resolution climate models (Satoh et al. 2019). Through changes in the latent heating and cloud‑radiation effects, improving modeling of deep convection is also important for improving simulations of large-scale atmospheric circulations and, hence, the spatial distribution of clouds and precipitation (Zhou et al. 2022). Collectively, improvements in modeling the rain rate distributions and atmospheric circulations are expected to have notable impacts on how well the Earth’s water cycle is simulated through more realistic representations of the energy balance at the top of the atmosphere and how precipitation is partitioned among evapotranspiration, runoff, and soil water storage at the land surface.
Recently, the Energy Exascale Earth System Model (E3SM) project (Leung et al. 2020) has developed a global kilometer-scale model called Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM) (Caldwell et al. 2021). The latest implementation, SCREAMv1 (Donahue et al. 2024), includes innovative algorithmic and software engineering advancements to run on Frontier, an exascale computer capable of performing more than one quintillion (1018) calculations per second. This has resulted in groundbreaking performance, achieving computational throughput exceeding one simulated year per day for global simulations at 3.25-km grid spacing (Taylor et al. 2023). With this computational performance, it is now feasible to run longer simulations beyond the 40-day boreal winter and summer simulations reported in the DYnamics of the Atmospheric general circulation Modeled On Non-hydrostatic Domains (DYAMOND), the first intercomparison project of global storm-resolving models (Stevens et al. 2019).
Extending the demonstration of the SCREAMv0 kilometer-scale modeling achieved through regional refinement summarized in the 2024 Second Quarter Performance Metric Report, this fourth quarter report focuses on the evaluation of the U.S. water cycle in SCREAMv1 global kilometer-scale simulations. While the SCREAMv0 regional refined simulations featured in the second quarter report are short simulations of 5-50 days produced on the National Energy Research Supercomputing Center (NERSC)’s Perlmutter using central processing units (CPUs), here SCREAMv1 is used to perform two 13‑month‑long, kilometer-scale global simulations on the graphics processing units (GPUs) of Frontier at the Oak Ridge Leadership Computing Facility (OLCF) and Perlmutter at NERSC. The two simulations were configured identically with prescribed sea surface temperature and sea-ice extent, except nudging is applied to one of the simulations to constrain its atmospheric circulations by a 50-km-resolution global reanalysis while the other is a free-running simulation after model initialization.
In this report, we track weather features in the 13-month-long SCREAMv1 simulations including atmospheric rivers (ARs), tropical cyclones (TCs), extratropical cyclones (ETCs) and mesoscale convective systems (MCSs) that are known to produce heavy precipitation and flooding in different regions of the U.S. The results for ARs, TCs, and ETCs are qualitatively compared with the performance of E3SMv1 at 10- km and 25-km resolutions (Harrop et al. 2023) and with similar results for E3SMv1 and other models reported in previous studies (e.g., Kim et al. 2022, Gore et al. 2023). Feng et al. (2023) tracked MCSs in the DYAMOND models, including SCREAMv0, and found that tropical MCSs are generally well simulated in global kilometer-scale simulations, despite noticeable differences among the models. Here the MCS results are also qualitatively compared with previous studies on evaluation of MCSs in E3SM and other models over the contiguous U.S. (CONUS).
Comparison of the two SCREAMv1 global kilometer-scale simulations with observations shows that the model can produce realistic statistics and characteristics for all four types of storms. More specifically, the observed spatial distributions of storm frequency over CONUS and the surrounding oceans are generally well reproduced. By constraining the atmospheric circulations towards those depicted by a global reanalysis, storm frequency is generally better simulated in the nudged simulation compared to the free-running simulation, underscoring the importance of correctly modeling the large‑scale circulations for skillful simulation of storm frequency. While this is true for ARs, ETCs, and MCSs, the nudged simulation produced TC frequency comparable to the reanalysis, but both are biased low compared to observations, while the free-running simulation better captured the higher TC frequency and intensity in observations. The latter suggests that by better resolving TC processes and their influence on the atmospheric environments, kilometer-scale simulations are more skillful for TC modeling than a global reanalysis with data assimilation, but at a coarser resolution. As for storm characteristics such as storm area and precipitation, both simulations are generally skillful and comparable, although some deficiencies have also been identified for future improvements. Through qualitative comparison with climate simulations at 25-50-km grid spacing reported in the literature, the SCREAMv1 global kilometer‑scale simulations clearly demonstrate superior skill in modeling the four types of storms (ARs, ETCs, TCs, and MCSs) that are the main causes of extreme precipitation and flooding.
Product Documentation
This report documents our effort to evaluate the U.S. water cycle simulated by SCREAMv1 configured at 3.25-km grid spacing globally, with 128 vertical levels and a model top at 40 km. Simulations were performed with a model top-level timestep of 100s and a dynamic time step of 8.33s. Descriptions of the model processes, model coupling, and computational strategies are provided by Donahue et al. (2024). Two 13-month-long simulations have been performed on the Frontier and Perlmutter GPUs. The simulations cover August 1, 2019 to September 1, 2020, with the simulation period chosen for its neutral phase in the El Niño-Southern Oscillation (ENSO), a dominant interannual mode of climate variability that has important impacts on the U.S. water cycle. These simulations were initialized using the atmospheric conditions from the European Center for Medium-Range Forecast Reanalysis version 5 (ERA5) at 25-km grid spacing (Hersbach et al. 2020), with sea surface temperature (SST) and sea-ice extent prescribed as the lower boundary conditions from the UK Met Office Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA) at 0.1ox0.1o and daily resolution. Other than constrained by the observed SST and sea-ice extent, the control simulation is a free‑running simulation with interactive land and sea-ice thermodynamics after the model initialization.
Noting that model biases in simulating water cycle processes such as storms and precipitation may result from model biases in simulating the large-scale atmospheric circulations as well as model biases in simulating local processes such as turbulence, clouds, and land-atmosphere interactions (Feng et al. 2022, Harrop et al. 2023), the control simulation is compared with another simulation in which model biases in the large-scale atmospheric circulations are suppressed. The latter is achieved by configuring a simulation that is identical to the control simulation except that the winds are nudged towards those of the Modern-Era Retrospective analysis for Research and Applications version 2 (MERRA-2) (Gelaro et al. 2017) at 50-km resolution throughout the 13-month simulation. These simulations with and without nudging, referred to as “nudged” and “free-running”, respectively, are evaluated against observations and compared to understand the role of large-scale circulation biases in modeling storm frequency and characteristics.
Storms including ARs, TCs, ETCs, and MCSs are tracked in both simulations. The TempestExtremes feature tracking tool (Ullrich et al. 2021) is applied to track ARs, TCs, and ETCs using model outputs (sea level pressure, geopotential height at 200 and 500 hPa, column integrated moisture flux, 10-m wind speed, and surface geopotential) on a coarsened grid (ne120) corresponding to roughly 25-km grid spacing. For MCSs, the PyFLEXTRKR feature tracking tool (Feng et al. 2023) is applied to model outputs (hourly brightness temperature and precipitation) interpolated to a 10-km latitude-longitude grid. Various statistics are calculated for comparison with those of the ARs, TCs, and ETCs tracked in ERA5 and described in Harrop et al. (2023). TC statistics are further compared against those of the International Best Track Archive for Climate Stewardship (IBTrACS) observations (Knapp et al. 2010). For MCSs, statistics are compared against those derived from the 10-km, hourly combined geostationary satellite infrared brightness temperature (Tb) and GPM IMERG precipitation data set (Feng et al. 2021a). The simulations are evaluated using observations for the same overlapping period. Since each simulation covers only one year, we caution about the statistical robustness of the model-observation differences and recommend that further evaluation using decade-long simulations be performed soon.
3rd QUARTER METRIC COMPLETED:
Demonstrating and Evaluating Sub-Kilometer Regional-to-Urban Integrated Pluvial and Fluvial Flooding Simulations Driven by Kilometer-Scale Extreme Event Precipitation
Product Definition
Flooding is one of the costliest natural hazards, inflicting billions of dollars in annual damages. Low-lying coastal areas are particularly prone to a range of flood hazards, including direct runoff from heavy rainfall (pluvial flood), river flooding (fluvial flood), and coastal storm surges and high tides (coastal flood). When these flood drivers overlap spatially or temporally, they can lead to “compound flooding”, which often results in more severe damage, as exemplified by Hurricane Irene (2011) ‒ a storm that caused unprecedented compound flooding in many Mid-Atlantic areas. With projected increases in extreme precipitation, intensified hydrological cycle, sea level rise, and continued coastal development, urban coastal populations and infrastructures face escalating risks. Accurate flood predictions are crucial for effective flood risk management and resilience strategies in these areas.
Sponsored by the U.S. Department of Energy (DOE), a coastal flood modeling framework (DHSVM‑FVCOM-RIFT) has been developed for regional-to-urban integrated modeling of fluvial, pluvial, and coastal flood processes at sub-kilometer spatial resolution. This framework couples regional hydrology (DHSVM – Distributed Hydrology Soil Vegetation Model), coastal hydrodynamics (FVCOM – Finite-Volume Community Ocean Model), and urban hydrodynamics (RIFT – Rapid Infrastructure Flood Tool) models. The term 'sub-kilometer regional-to-urban’ refers to simulations that span extensive regional landscapes to detailed urban environments at spatial resolutions from 150 meters to 10 meters. In this report, we demonstrate that driven by kilometer- or near-kilometer-scale atmospheric forcing, the DHSVM-FVCOM-RIFT framework skillfully predicts sub-kilometer-scale flooding across river, coastal, and urban systems, even during complex compound flooding events.
The modeling capability and accuracy are demonstrated in two distinct U.S. coastal areas through: (1) an event-scale modeling of the 2022 compound flooding event in the Puget Sound region and (2) long-term modeling (1985-2019) of integrated fluvial, coastal, and pluvial flooding across the Delaware River, Bay, and Philadelphia. The sub-kilometer modeling capabilities were evaluated for each regional demonstration using available observational data. For example, DHSVM was evaluated using observational streamflow data, FVCOM was evaluated using observational tide gage data, and RIFT was evaluated using observational instream water surface elevation data. For the Puget Sound region in the Pacific Northwest, the modeling focused on a single compound flooding event in 2022. This event, characterized by concurrent king tides, rain-on-snow river flooding, and storm surges, caused extensive flooding across the Puget Sound region, especially in Seattle’s Duwamish District. The modeling of this event was conducted using DHSVM-FVCOM-RIFT, driven by 3-km atmospheric forcing from the Simple Cloud-Resolving Energy Exascale Earth System (E3SM) Atmosphere Model (SCREAM) with regional refinement. The modeling framework accurately captured the tidal ranges and timing, as well as influence from storm surges and river discharge. Analysis of tide, river, and surge effects on coastal flooding indicates the importance of modeling the strong nonlinear interactions between the forcing mechanisms.
In the Delaware coastal region, we used the DHSVM-FVCOM-RIFT framework to conduct 35-year continuous simulations of watershed-coastal processes, aiming to provide a more comprehensive evaluation of flood modeling accuracy over extended time scales. Due to the computational demands of generating kilometer-scale atmospheric forcing, we used near-kilometer-scale atmospheric forcing from the WRF-TGW dataset. This dataset offers 12-km, 40-year historical climate simulations that are dynamically downscaled from climate reanalysis data using the Weather Research and Forecasting (WRF) model. Through extensive evaluations against available observational data, including river discharge, coastal water level, and urban flooding, this modeling framework consistently demonstrates skillful flood prediction across both coastal regions. Notably, we also demonstrate the capability of this framework for property-level assessments of probabilistic flood hazards, flood drivers, and infrastructure flood exposure in Philadelphia. Overall, we found that river flooding plays a significant role in Philadelphia flooding, contributing to 73% of events, either as a single driver or as a component in compound driver cases. We also identified three compound flood events involving concurrent surge, pluvial flooding, and river flooding, highlighting the complexity of flood drivers in Philadelphia and emphasizing the need for models that accurately represent the interactions between multiple flood drivers.
Product Documentation
This report documents DOE's advanced capability in simulating regional-to-urban flooding from integrated pluvial, fluvial, and coastal impacts, using the DHSVM-FVCOM-RIFT modeling framework. The modeling capability and accuracy are demonstrated in two distinct U.S. coastal areas through: (1) an event-scale modeling of the 2022 compound flooding event in the Puget Sound region, driven by 3-km SCREAM atmospheric forcing; and (2) long-term modeling of integrated fluvial, coastal, and pluvial flooding across the Delaware River, Bay, and Philadelphia, driven by 12-km WRF-TGW climate simulations (1985-2019).
2nd QUARTER METRIC COMPLETED:
Demonstrating and Evaluating Kilometer-Scale Regional Refinement Modeling Capability in Simulated Water Cycle Extremes in the U.S.
Product Definition
Increasing the horizontal grid resolution in atmosphere models has been shown to lead generally to improvements in modeling extreme events such as extreme precipitation produced by thunderstorms (e.g., Pope and Stratton 2002, Roekner et al. 2006, Wehner et al. 2010, Jong et al. 2023). At higher resolutions, atmosphere models can produce stronger vertical motions, for example, by better resolving the temperature and moisture gradients associated with fronts and tropical cyclones and representing the orographic uplift associated with mountains, and subsequently the higher condensation rates needed to generate intense precipitation associated with storms. However, atmosphere models simulate the precipitation processes using physics parameterizations such as cloud microphysics and convection with resolution- and time step-dependent behaviors, which complicate the relationship between model resolution and the simulated mean and extreme precipitation (e.g., Kopparla et al. 2013, Yang et al. 2014, Wehner et al. 2021).
Global kilometer-scale models that have emerged in the last decade (Stevens et al. 2019) hold great promise to further improve modeling of extreme weather events by explicitly resolving deep convection, a major mechanism for generating extreme precipitation, and potentially improving the simulation of mesoscale and large-scale atmospheric environments for the storms. Recently, the Energy Exascale Earth System Model (E3SM) project funded by the U.S. Department of Energy developed a global kilometer-scale model called Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM) (Caldwell et al. 2021). Implementation of SCREAM, including innovative algorithmic and software engineering advancements to run on the Frontier Exascale computer, has demonstrated a groundbreaking performance of computational throughput > 1 simulated year per day (SYPD) (Taylor et al. 2023). However, running decadal-to-century-scale simulations using global kilometer-scale models remains computationally challenging, especially for quantifying the statistics of extreme events and their future changes (which typically requires large ensemble simulations). To circumvent such computational challenges, SCREAM has a regional refinement capability that allows kilometer-scale modeling to be performed in regions of interest within the context of global modeling at coarser resolutions outside those regions. This capability was first demonstrated in a study that used SCREAM with regional refinement over the central-eastern U.S. for simulating the strong winds and convection associated with a derecho event (Liu et al. 2023).
Building on the demonstration of Liu et al. (2023), this report documents the configurations and evaluation of SCREAM using regional refinement for kilometer-scale simulations of cold-season and warm-season storm events that produce extreme precipitation across diverse geographical regions in the U.S. Three examples are included to demonstrate the regional refinement capability of SCREAM to simulate (1) atmospheric rivers that made landfall in the U.S. Pacific Northwest Puget Sound basin that produced flooding, (2) mesoscale convective systems and isolated deep convection in the central-eastern U.S. that produced heavy precipitation in the region, and (3) a blizzard in the northeastern U.S. that produced snow and ice that damaged energy infrastructure. The results provide clear evidence that using regional refinement at 3.25-km grid spacing, the SCREAM model provides a scientifically robust and computationally efficient capability for simulating a variety of extreme weather events, including atmospheric rivers, mesoscale convective systems, and winter storms, across different geographies and regions of the contiguous U,S. Such capability is critical for understanding water cycle extremes such as flooding caused by heavy precipitation, rain-on-snow events, and compound events, as well as ice storms that cause disruptions and power outages.
Product Documentation
This report documents the modeling of water cycle extremes in the U.S. using SCREAM with regional refinement capability for kilometer-scale modeling. This capability is demonstrated using three examples of simulating cold-season and warm-season storms in the U.S. underlined by different processes. SCREAM is configured for a global domain at 25-km grid spacing with regional refinement at 3.25-km grid spacing covering part of the U.S. and the surrounding ocean where the storms developed. As SCREAM does not include any parameterization of deep convection, which is needed at 25-km grid spacing, nudging was applied to constrain the SCREAM simulations with the European Center for Medium-Range Forecast Reanalysis version 5 (ERA5) (Hersbach et al. 2020) outside the refined region. Initialized using atmospheric conditions from ERA5, the simulations, each covering 5-50 days, were compared with observations to demonstrate the model skill in simulating important water cycle aspects of the storm events.
1st QUARTER METRIC COMPLETED:
Developing and demonstrating a Kilometer-Scale Land Simulation Capability for Modeling the Terrestrial Water Cycle over the Contiguous U.S.
Product Definition
Kilometer-scale (k-scale) modeling allows for explicit modeling of physical processes that are poorly represented in climate models with typical resolutions of 25-100 km, thus providing opportunities to significantly improve the accuracy of climate simulations (Slingo et al. 2022). The recent advancements in computing power are making k-scale regional and global simulations using Land Surface Models (LSMs) and Earth System Models (ESMs) increasingly feasible (Condon et al. 2020, Caldwell et al. 2021). Groundwater is a vital human water resource that provides 20-30% of global freshwater withdrawals (Döll 2009). Previous k-scale LSM studies for specific watersheds or basins have demonstrated the impacts of fine scale-structures on terrestrial hydrologic processes including groundwater dynamics (Maxwell and Kollet 2008, Fan et al. 2013).
However, the parameters of LSMs within ESMs being run at the k scale are typically derived from coarse-resolution data sets or outdated data sets. Consequently, k-scale modeling may not accurately represent fine-scale land surface heterogeneity unless high-resolution land surface parameters at the kilometer or finer scales are used. Additionally, LSMs will have to be recalibrated at k scale to accurately simulate terrestrial processes (Ruiz‐Vásquez et al. 2023) and the recalibration is expected to have a significantly large computational cost at such high spatial resolution.
In this report, we develop a first-of-a-kind k-scale global land simulation capability and demonstrate this for modeling the terrestrial water cycle over the contiguous U.S. in ESMs. To enable this capability, we first develop a new set of global land surface parameters at 1-km resolution by using the newest high‑resolution data sources for multiple years by combining the latest and most accurate available global data sets to provide input data for ESMs, including the U.S. Department of Energy’s flagship Energy Exascale Earth System Model (E3SM) (Leung et al. 2020). Second, we produce an initial 5-year simulation using E3SM Land Model version 2 (ELMv2) over the contiguous United States (CONUS) at 1-km resolution using the newly developed land surface parameters. A spatial scaling analysis was performed to underscore the value of the high-resolution land surface parameters, and AI/ML methods were used to identify the most important land surface parameters and climate conditions that drive the spatial variability at k scale and hence, spatial information loss as resolutions were coarsened in ELMv2 simulations. Using the newly developed 1-km input data, the k-scale simulations show significant spatial heterogeneity of soil moisture, latent heat, emitted longwave radiation, and absorbed shortwave radiation, which is expected to have important effect on modeling of land-atmosphere interactions. Third, we demonstrate significant improvements in the ELM-simulated water table depth over CONUS by calibrating ELMv2’s subsurface drainage parameterization for k-scale modeling.
Product Documentation
This report documents a k-scale land simulation capability, which includes the development of global k-scale land surface parameters for LSMs (2.1), the application of the newly developed surface parameters in k-scale ELMv2 simulation over CONUS and analysis that demonstrates the importance of the k-scale surface parameters data set (2.2), and the calibration of ELMv2’s subsurface hydrologic processes to improve the prediction of groundwater at a 1-km resolution over CONUS (2.3).