Realistic representation of land carbon sink in climate models is vital for predicting carbon climate feedbacks in a changing world. Although soil erosion removes organic carbon from land and has increased substantially since the onset of agriculture, it is rarely included in the current generation of climate models. Using a numerical model with soil erosion represented, we estimate that soil erosion displaces 5% of newly fixed organic carbon by plants to land downslope annually in the continental US but this ratio can be as large as 40% in the lower Mississippi river basin and the Cascades. About 12% of the eroded soil organic carbon is eventually exported to inland waters, which is equal to 14% of the model simulated net carbon gain by terrestrial ecosystems. By comparing the amount of organic carbon exported to rivers with the organic carbon export to oceans, we indicate that a large fraction of the eroded organic carbon in inland waters has been mineralized. Importantly, with a direct comparison of eroded and exported soil organic carbon and land net carbon uptake, we find that climate models which ignore soil erosion are likely using heterotrophic respiration to offset erosional carbon loss implicitly. But as soil erosion and heterotrophic respiration are shown to respond differently to a warming climate, this unrealistic compensation would lead to biased predictions of future land carbon sink.