Ensuring Climate Simulation Reproducibility in the Exascale Era
In a recent paper, researchers at USDepartment of Energy’s Oak Ridge National Laboratory present a short simulation ensemble-based statistical testing framework that indicates that a frozen model configuration of DOE’s Energy Exascale Earth System Model (E3SM) was reproducible after many months of improvements to its support software infrastructure on Titan. Numerous short simulation ensembles with controlled changes to the model were conducted to empirically quantify the power (probability of correctly detecting a false null hypothesis) of the testing framework and the magnitude of change in the climate statistics that the test can confidently detect.
Using a short simulation ensemble framework applied to a realistic climate model configuration, DOE researchers were able to use a statistical testing methodology to establish that the simulation results of an older model configuration—which was not directly perturbed—are not significantly impacted through a multiyear model development cycle. This is reassuring given that bit-for-bit reproducibility cannot be expected with new hybrid architectures. Statistical power analysis of the testing strategy suggests that it can detect changes or successfully reject the null hypothesis of equality of distribution when known climate changes are introduced.
DOE researchers presented a methodology for solution reproducibility for the Energy Exascale Earth System Model (E3SM) during its ongoing software infrastructure development to prepare for exascale computers. The non-linear chaotic nature of climate system simulations precludes traditional model verification approaches since machine precision differences—resulting from code refactoring, changes in the software environment, etc.—grow exponentially to a different weather state. They leverage the nature of climate as a statistical description of the atmosphere in order to establish model reproducibility. The researchers evaluate the degree to which two-sample equality of distribution tests can confidently detect the change in climate from minor tuning parameter changes on model output variables in order to establish the level of difference that indicates a new climate. They apply the test to a section of the model's development cycle wherein no intentional science changes have been applied to its source code. An ensemble of short simulations that were conducted using a verified model configuration against a new ensemble with the same configuration but with the latest software infrastructure (Common Infrastructure for Modeling the Earth, CIME5.0), compiler versions, and software libraries were compared. They also compare these against ensemble simulations conducted using the original version of the software infrastructure (CIME4.0) of the earlier model configuration, but with the latest compilers and software libraries, to test the impact of new compilers and libraries in isolation from additional software infrastructure. The two-sample equality of distribution tests indicates that these ensembles indeed represent the same climate.