In this paper, we propose a post-training optimization (PTO) strategy to refine correspondence measurement in the end-to-end optimized framework. The proposed PTO strategy introduces a pseudo loss function to well approximate the target loss and guide the direction of updates. We further develop a video colorization method that incorporates PTO and optical flow to guarantee high-fidelity colorized frames in theory. Experimental results demonstrate the proposed method achieves state-of-the-art PSNR performance in video colorization on the DAVIS dataset and common test sequences for video coding. Furthermore, the proposed method is employed into video compression and achieves competitive rate-distortion performance with HEVC.