Colorization for far infrared image is a very challenging task in which feature detection is difficult because of the lack of details compared with visible image. In this paper, we propose a high quality far infrared image colorization method based on generative adversarial network. An efficient pre-processing module is used to improve the quality of colorized image quality in low light environment. In addition, since conventional loss function is not sufficient enough for far infrared image colorization, we propose a composite loss function that combines pixel-wise, adversarial and attention losses. Our proposed method is fully-automatic and robust to image pair misalignments. Quantitative and qualitative experiments demonstrate that our proposed method significantly outperforms existing approaches on the KAIST multispectral pedestrian dataset, achieving more natural and plausible colorized images especially in low light environment.