Convection-permitting models with km-scale grid spacing can explicitly simulate organized convective storms and their associated extreme weather. These models are often considered the future of Earth system models and have been used to study the impacts of climate change on extreme precipitation and windstorms. Here, we comprehensively examined tropical mesoscale convective system (MCS) characteristics in the DYAMOND models (Stevens et al. 2019) for both summer and winter phases by applying 8 different feature trackers to the simulations and observations. Although different trackers produce substantial differences (up to a factor of 2) in observed MCS frequency and their contribution to total precipitation, model-observations differences are more consistent among the trackers. We find that the DYAMOND models are generally skillful in simulating tropical mean MCS frequency, with multi-model mean biases smaller than 30%. Most models underestimate the MCS precipitation amount and their contribution to total precipitation relative to observations (Fig. 1a), with smaller multi-model mean biases over land (6%-20%) than over ocean (9%-28%), though large variability exists among individual models (Fig. 1b). MCS diurnal cycle and cloud shield characteristics are better simulated than precipitation. Most models overestimate MCS precipitation intensity and underestimate stratiform rain contribution, particularly over land. Models also show a wide range of precipitable water in the tropics compared to reanalysis and satellite observations, and many models simulate a higher sensitivity of MCS precipitation intensity to precipitable water (Fig. 1c,d). Potential paths towards more process-oriented model diagnostics to better understand the differences in simulated MCS and precipitation characteristics will be discussed.