Urban surface and near-surface air temperatures are known to be often higher than their rural counterparts, a phenomenon now labeled as the urban heat island effect. However, whether the elevated urban temperatures are more persistent than rural temperatures at timescales commensurate to heat waves has not been addressed despite its importance for human health. Combining numerical simulations by a global climate model with a surface energy balance theory, it is demonstrated here that urban surface and near-surface air temperatures are significantly more persistent than their rural counterparts in cities dominated by impervious materials with large thermal inertia. Further use of these materials will result in even stronger urban temperature persistence, especially for tropical cities. The present findings help pinpoint mitigation strategies that can simultaneously ameliorate the larger magnitude and stronger persistence of urban temperatures.