We propose an end-to-end video compression method for P-frame in sub-sampled color spaces with the contributions summarized as follows: We introduce the motion estimation and motion encoder-decoder on YUV420 to produce compact optical flow for Y, U and V components. Furthermore, we present the motion compensation with resolution alignment and enhancement on each component. We proposed the residual encoder-decoder with the consideration of cross-component correlation. A proper loss function emphasizing the importance of the Y component is utilized. Experimental results demonstrate that the proposed method outperforms the previous end-to-end video compression method in sub-sampled color spaces.