Atmospheric rivers (ARs), intrusions of warm and moist air, can effectively drive weather extremes over the Arctic and trigger subsequent impact on sea ice and climate. What controls the observed multi-decadal Arctic AR trends remains unclear. Here, using multiple sources of observations and model experiments, we find that, contrary to the uniform positive trend in climate simulations, the observed Arctic AR frequency increases by twice as much over the Atlantic sector compared to the Pacific sector in 1981-2021. This discrepancy can be reconciled by the observed positive-to-negative phase shift of Interdecadal Pacific Oscillation (IPO) and the negative-to-positive phase shift of Atlantic Multidecadal Oscillation (AMO), which increase and reduce Arctic ARs over the Atlantic and Pacific sectors, respectively. Removing the influence of the IPO and AMO can reduce the projection uncertainties in near-future Arctic AR trends by about 24%, which is important for constraining projection of Arctic warming and the timing of an ice-free Arctic.