High-order, property-preserving semi-Lagrangian tracer transport and physics-dynamics-grid remap in the E3SMv2 Atmosphere Model
We describe two new methods that together increase the computational efficiency of the E3SM Atmosphere Model (EAM) by a factor of approximately two in multiple configurations and on multiple platforms. EAM consists of the High-Order Methods Modeling Environment (HOMME) Spectral Element (SE) dynamical core (dycore) and the EAM physics and chemistry parameterizations. The most expensive parts of EAM are tracer transport in the dycore and parameterizations. Two related methods speed up each of these: high-order, property-preserving, remap-form, semi-Lagrangian tracer transport (SL transport) and high-order, property-preserving physics-dynamics-grid remap (physgrid remap). SL transport replaces the dycore's Eulerian flux-form method, enabling much greater simulation time per computational communication round. Physgrid remap enables running parameterizations on a grid that approximately matches the effective resolution of the SE dycore. This procedure replaces running parameterizations on the denser dynamics grid. These new methods work seamlessly both in the standard EAM configuration and in the Regionally Refined Mesh (RRM) configuration. The E3SMv2 low-resolution and RRM coupled configurations use these features. On E3SM's Chrysalis system, based on AMD Epyc 7532 processors, the low-resolution coupled model achieves over 41 simulated years per day (SYPD) on 105 nodes, and the North America RRM coupled model achieves over 12 SYPD on 100 nodes.