Detecting Modeling Problems Early and Quickly
Weather and climate models are large computer code structures that solve complex systems of mathematical equations. Researchers developed a new, objective, and computationally efficient method to determine the quality of the results from an atmosphere model when they don’t match identically, digit for digit in essentially identical runs.
Weather and climate models that provide important predictions for society are often developed and maintained by a large team of scientists working collaboratively. These teams thrive with objective and efficient testing methods to ensure the quality of the code and the computing environments during and after development.
After revising the code, or updating the software and hardware environment, there may be times when it is no longer possible to obtain numbers identical “digit for digit” to previously verified results. In these situations it is very important, and non-trivial, to distinguish whether these are “noise-level” differences from discrepancies that are caused by unintended coding errors or computing-environment problems. Existing methods that evaluate these discrepancies with long-term statistics of model results are too computationally expensive to use for daily testing during phases of very active model development. A team of researchers led by scientists at Pacific Northwest National Laboratory developed a new method just as robust as exiting methods, but hundreds of times cheaper in computing expensive. Test results from a new model or computing environment are considered “changed beyond noise level” when the numerical error calculated against a benchmark is found to be inconsistent with previously verified values. In previous work, the researchers showed that the new method was effective for the Community Atmosphere Model. The researchers expect that the underlying concept for this development will be applicable to additional atmosphere and geophysical models.