We develop a novel method for efficiently deploying Convolutional Neural Networks (CNNs) on arbitrary unstructured grids. Specifically for the geophysical community, we illustrate its applicability to the icosahedral spherical mesh, allowing for deep learning tasks directly performed in the spherical domain. Compared to regular-grid based conventional CNNs that operate on the Latitude-Longitude map of a spherical signal with significant distortions, CNNs operating on the native spherical mesh do not suffer map distortions, especially closer to the polar regions. We also show the flexibility of our mesh-based CNN operations, for which we can define operators such as convolution, transpose-convolution, pooling, dropout etc., which are the basic building block constituting a CNN. Hence CNN architectures developed for the regular-grid domain can be easily adapted for use in the unstructured grids. As an illustration of the efficacy of such algorithms, application-wise, we illustrate the use of icosahedral spherical mesh based CNN for the semantic segmentation of climate patterns (tropical cyclones and atmospheric rivers).