Uncertainty in the representation of biomass burning (BB) aerosol composition and optical properties in climate models contributes to a range in modeled aerosol effects on incoming solar radiation. Depending on the model, the top-of-the-atmosphere BB aerosol effect can range from cooling to warming. By relating aerosol absorption relative to extinction and carbonaceous aerosol composition from 12 global BB observations to 9 state-of-the-art Earth System Models/Chemical Transport Models, we identify varying degrees of overestimation in BB aerosol absorptivity by these models. Modifications to BB aerosol refractive index, size, and mixing state improve the Community Atmosphere Model version 5 (CAM5) agreement with observations, leading to a global change in BB direct radiative effect of -0.07 W m-2,and regional changes of -2 W m-2 (Africa) and -0.5 W m-2 (South America/Temperate). Our findings suggest that current modeled BB direct radiative effects are biased high, largely due to treatments of aerosol mixing state. Model challenges in representing the properties of BB aerosols with different ageing time scales are discussed.