Feature attention is a technique used in deep neural networks to provide a discriminative processing of the various regions in an image based on their significance for enhancing the image restoration performance. In this paper, we develop a novel image restoration network, in which the feature maps extracted by the network are recalibrated using a pixel-wise feature attention and the recalibration process is guided by the structural and textural information of the image resulting from the Otsu’s method for its segmentation. It is shown that using this segmentation guidance strategy for recalibrating feature maps is indeed helpful in enhancing the quality of the restored images. The proposed image restoration network outperforms the state-of-the-art light-weight image restoration networks on benchmark datasets.