This paper presents a method to utilize the movement relationship between adjacent frames for video-based stereo matching. Unlike existing video-based algorithms that need complicated calculation to fuse the information between frames, the proposed method uses the existing motion vector in video encoder or decoder to reconstruct the initial disparity of the target frame from a reference frame. By disparity plane reconstruction, disparity can be initialized in an easy way, such that unnecessary steps in the stereo matching pipeline can be skipped. By taking advantages of the motion relationship between frames, the proposed scheme can achieve better performance with less computation compared with the original method.