On July 16-18, 2024, the DOE Office of Science Advanced Scientific Computing Research Program (ASCR) held the Scientific Discovery through Advanced Computing (SciDAC) Partnership Program Principal Investigator (PI) Meeting in Rockville, Maryland. Building on over two decades of tradition in facilitating interdisciplinary exchanges and collaborations, the event brought together mathematicians and computer scientists supported by ASCR as well as domain scientists from various research programs under DOE’s Office of Science and Office of Nuclear Energy, including the Biological and Environmental Research Program (BER). The plenary sessions on day 1 and day 3 highlighted the breadth of research being performed with DOE’s leadership computing facilities. Strategies to tackle challenges and opportunities were also discussed. The breakout sessions of individual research partnership programs on day 2 provided a venue to dive deeper into discussions in specific research domains.

BER’s breakout session was kicked off by a set of comments from Dr. Dorothy Koch, the Associate Director of BER, who expressed her appreciation of the challenges but also the importance of interdisciplinary research and reminded the audience that with patience and perseverance, the pay-off for science can be enormous. SciDAC is a powerful and unique program that only DOE offers to earth system modeling. Dr. Gary Geernaert, Director of BER’s Earth and Environmental Systems Sciences Division, commended SciDAC projects for their role to “set the stage for what’s next”. BER’s Program Manager for Earth System Model Development (ESMD) and the BER-ASCR SciDAC Partnership Program, Dr. Xujing Davis, summarized the history of SciDAC investment and its long term and profound scientific impacts in advancing the Energy Exascale Earth System Model (E3SM) as well as broad climate and earth system modeling science. ASCR SciDAC Partnership Program Manager, Dr. Lali Chatterjee, and the ASCR Program Manager for RAPIDS Institute, Dr. Kalyan Perumalla, also attended the BER Breakout Session and expressed their strong support for this valuable partnership.

The seven projects (Fig. 1) currently supported by the BER-ASCR SciDAC partnership are approaching the end of Year 2 of the 5-year tenure. Summaries presented by the PIs or their representatives showed that all projects had successfully integrated expertise from the two SciDAC Institutes, FASTMath and RAPIDS, leveraging advanced methods of spatial and temporal discretization, uncertainty quantification, machine learning (ML), and high-performance computing to address model development and evaluation challenges relevant for E3SM’s atmosphere, ocean, land/hydrology, sea ice, and land ice components.

SciDAC/E3SM coordination

To facilitate the coordination and collaboration between the BER SciDAC projects and the E3SM project, three members of the E3SM leadership team who are also part of the SciDAC projects, Wuyin Lin, Luke Van Roekel, and Gautam Bisht, were invited to present highlights and priorities of the E3SM development in the next few years. The presentations were followed by an open discussion, in which the attendees recognized the increased adoption by E3SM of code libraries developed by ASCR and the challenges related to software evolution. It was also pointed out that E3SM and the ecosystem projects have different levels of technical readiness for their products and different levels of risk tolerance in code development, causing hurdles in the process of incorporating new features from the ecosystem into E3SM’s main code repository. Embedding E3SM team members in SciDAC projects has been highly beneficial for addressing these differences, for facilitating code integration, and for increasing the appreciation of computational work.

Three additional topics were discussed during the breakout: (1) collaboration between computational and physical scientists, (2) use of artificial intelligence (AI) and ML in Earth system modeling, and (3) computational resources for SciDAC projects.

Collaboration between computational and physical scientists

The collaboration between computational and physical scientists is the key feature and strength of the SciDAC program. The need for establishing common languages and building trust through intensive communication was widely recognized in the SciDAC community. It was seen across projects that tasks involving new collaborations needed significantly longer spin-up than those based on existing collaborations. Cultivating researchers capable of interfacing two disciplines typically required enthusiasm, patience, and commitment over multiple years. It was understood that the process was slow but necessary and highly rewarding.

AI and ML in Earth system modeling

The use of AI/ML techniques was mentioned in many sessions at the PI meeting, both plenary and in breakouts. Project briefings during the BER breakout indicated that most projects had started experimenting with AI/ML. Although none of the projects have AI/ML as a central focus, the ongoing or envisioned activities included using AI/ML for analysis outside the E3SM code (i.e., for pre-processing model input or postprocessing model output), using AI/ML to represent processes in E3SM (e.g., AI/ML replacement of a parameterization), using AI/ML to emulate the entire E3SM or a component of it (e.g., for automatic parameter tuning), and methodology development or theoretical considerations on the math side. An important outcome of the open discussion was that a gap in understanding and communication was identified. The AI/ML field had been evolving very fast, introducing new language and concepts that required correct interpretation to the scientific domains. For example, it was not uncommon for Earth scientists to have the perception that more extensive uses of AI/ML would imply giving up on process-level understanding or mechanistic models based on first principles and differential equations. The AI/ML experts, in contrast, emphasized the crucial role of physical understanding in making AI/ML applications successful, and they expressed a strong enthusiasm for deep collaboration with the domain scientists. Initial efforts were made during the meeting to start an ongoing AI/ML dialog between the BER project teams and the SciDAC Institutes.

Computational resources

On the topic of computing resources, the BER project teams expressed the wish for significant increases in computing resources beyond what they currently had access to through the National Energy Research Scientific Computing Center (NERSC) and institutional computing. Some projects were able to benefit from Director’s Discretion projects at the Oak Ridge Leadership Computing Facility, and it might be useful to make use of the Early Career Track of the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program and find ways to leverage resources from the ASCR Leadership Computing Challenge (ALCC). A challenge to be addressed is the contrast between some of the facilities’ preference for capability jobs (i.e., individual, very large computing tasks) and the Earth system scientists’ need to execute capacity jobs (e.g., large ensembles of smaller computing tasks) to tackle variabilities and uncertainties in the physical system.

More details such as the Agenda can be found at the event website. Materials presented at the BER breakout can also be found on a Confluence page.

Contact

• Hui Wan, Pacific Northwest National Laboratory
• Matt Hoffman, Los Alamos National Laboratory

# # #

This article is a part of the E3SM “Floating Points” Newsletter. Read the full August 2024 edition.