The El Niño-Southern Oscillation (ENSO) is the dominant mode of interannual variability in the climate system, altering the global distribution of temperature and precipitation. These altered climate patterns impact terrestrial regions both locally and in remote regions via atmospheric teleconnections. Due to the sensitivity of photosynthesis to changes in temperature, precipitation and radiation, ENSO has large impacts on the carbon cycle and the magnitude of carbon uptake by terrestrial vegetation at interannual timescales. We quantify the ENSO impact on gross primary productivity (GPP) and identify the physical drivers of the terrestrial ENSO response for different regions around the world. The ten CMIP6 Earth System Models we analyzed showed different patterns of ENSO impacts on GPP across the tropics, but models showed a clear reduction in carbon uptake by terrestrial vegetation during El Niño years. This reduced carbon uptake is primarily driven by a suppression of GPP in the Amazon Basin due to reduced precipitation and increased surface temperatures. However, three models (IPSL-CM6A-LR, MIROC-ES2L, and MPI-ESM1-2-LR) exhibit a weaker reduction, or even an enhancement, to GPP in the Amazon Basin via a strong sensitivity of photosynthesis to increased sunlight during El Niño years. Outside the tropics, models showed more similar GPP responses to El Niño, although the contribution to global carbon variations was generally smaller. Our results suggest that improved constraints on the sensitivity of tropical forests to variations in radiation may enable more robust model predictions of the impact of ENSO on carbon fluxes.