In this paper, we introduce a deep learning based approach for optimizing the yield prediction process of spring wheat, using multispectral images. We assessed both the temporal features to find the most valuable time to take images, as well as the contribution of spectral bands. We processed full stage images from four site-years of a wheat breeding project, and determined the prediction accuracy of the image-based predicted yields and compared them to the harvested yields taken in the field. These results could be a tool for the development of more efficient sensors and strategies for data collection in plant phenotyping.