Emerging evidence shows the negative impacts of heat stress on human health, labor productivity, and labor participation. Agricultural labor, accounting for about a quarter of global employment, is particularly vulnerable to heat stress due to the high work intensity and outdoor conditions. However, existing studies of climate impact on agriculture have primarily focused on crop impacts, often neglecting the impacts on agricultural labor and the consequential market responses. Here, we address both crop yield and labor productivity responses to future climate change and investigate their implications on global and regional agriculture. In particular, we use a coupled modeling framework to translate climate projections (HadGEM2-ES and GFDL-ESM2M under Representative Concentration Pathway 6.0) first to crop yield responses (GEPIC and LPJmL) and labor productivity responses (labor-heat response functions), and then to global economic outcomes (GCAM). Our results suggest that heat-induced agricultural labor productivity loss increases over time, accelerating in the latter half of the century. The climate impact on agriculture through labor response may exceed that through crop response, notably in Africa and Southeast Asia. Our results also show that climate impacts on crop yield and labor productivity are highly sensitive to the choices of climate models, crop response models, and labor-heat response functions. This study underscores that omitting heat-induced labor productivity shocks underestimates the overall climate impact on global agriculture.