Atmospheric rivers (ARs) are long and narrow filaments of intense moisture transport in the atmosphere. When ARs intrude into the Arctic, the enormous amount of warm and moist air carried by them can effectively drive weather extremes over the Arctic and trigger subsequent impact on sea ice and climate. Yet, what controls the observed multi-decadal Arctic AR trends remains unclear. In a recent study, using multi-sources of observations and model experiments, we found that anthropogenic forcing alone would drive a uniform positive AR trend over the Arctic due mostly to the increasingly moistened atmosphere. However, in contrast to the spatial distribution of anthropogenically driven AR trends, the observed one shows an uneven pattern with ARs increasing by twice as much over the Atlantic sector compared to the Pacific sector in 1981-2021. Using both maximum covariance analysis and inter-member regression approaches, we found that this discrepancy can be reconciled by the observed phase shifts of Interdecadal Pacific Oscillation (IPO) and Atlantic Multidecadal Oscillation (AMO). The observed positive-to-negative phase shift of the IPO is expected to reduce ARs over the Pacific sector but increase ARs over the Atlantic sector. Meanwhile, the observed negative-to-positive phase shift of the AMO drives AR increase over the Atlantic sector and decrease over the Pacific sector. The combination of the observed IPO and AMO phase shift thus favors the AR increase over the Atlantic sector and reduction over the Pacific sector. Given the strong control of the IPO and AMO in the decadal variability of Arctic ARs, we further demonstrate that removing the influence of the IPO and AMO can reduce the projection uncertainties in near-future Arctic AR trends by about 24%. Our findings have implications for constraining projections of Arctic warming and timing of an ice-free Arctic.