Tropical forest dynamics play crucial roles in the global carbon, water, and energy cycles. Dynamic global vegetation models are the primary tools to simulate terrestrial ecosystem dynamics under current and future climate. However, realistically simulating the dynamics of species competition and coexistence remains a significant challenge. The overarching goal of this study is to improve modeling of plant species coexistence using the Functionally Assembled Terrestrial Ecosystem Simulator (FATES), a demographic vegetation model implemented in the Energy Exascale Earth System Model (E3SM) land model (ELM), ELM-FATES. Specifically, we explore whether: (1) plant trait correlations established from field measurements improve ELM-FATES simulations; (2) machine learning-based surrogate models can well emulate ELM-FATES behavior and optimize parameter selections to improve species coexistence modeling. An initial ensemble of ELM-FATES experiments (i.e., EXPs-1) conducted over a tropical forest site at Manaus, Brazil, show that considering the observed trait correlations slightly improves ELM-FATES simulations of water, energy, and carbon fluxes, but degrades the simulation of species coexistence. Using eXtreme Gradient Boosting (XGBoost) based surrogate models trained on EXPs-1, we optimize the trait-related parameters in ELM-FATES to reduce model errors relative to field observations including latent and sensible heat, gross primary production, and above ground biomass. The surrogate model selected parameters were used to conduct another ensemble of ELM-FATES experiments (i.e., EXPs-2). The probability of experiments yielding plant species coexistence greatly increases from 20.6% in EXPs-1 to 73.1% in EXPs-2. Further filtering the coexistence experiments by observations, EXPs-2 still has 33.0% experiments left, much higher than 1.4% in EXPs-1. EXPs-2 also better captured the observations than EXPs-1. Our study demonstrates the benefits of using machine learning models to improve the species coexistence modeling in ELM-FATES, with important implications for modeling the response and feedback of ecosystem dynamics to climate change.