Prostate cancer is one of the most common malignant tumors in men. Accurate prostate segmentation is essential for prostate cancer diagnosis and intervention. However, the variation in prostate shape, appearance, and size makes the task challenging, given the limit of the annotated data. There are works proposed applying convolutional neural networks in the literature for the task. In this paper, we propose a method using multi-scale and Channel-wise Self-Attention (CSA) to re-calibrate the feature maps from multiple layers. By embedding the multiscale CSA on the skip-connection in a UNet structure, called as UCAnet, we show the consistent improvement of the prostate segmentation in Dice, IoU and ASSD. For comparison, we also investigate the single-scale CSA in the networks, and incorporate the vision transformer to test if a transformer would boost the performance. Experiments on a public dataset with 204 prostate MRI scans show that UCAnet achieves the best performance and outperforms other state of art methods for prostate segmentation such as ENet, UNet, USE-Net and TransUNet.