We propose a saliency and error feature fusion (SEFF) method for objective image quality assessment . Two image features, the error between reference image and distorted image, the subjective saliency of distorted image, are taken as input of the CNN. The evaluation score of image quality is obtained through a conventional CNN that trained with frequently-used public databases for IQA. The proposed method possess a basic structure of the CNN only and reduces the volume of training data remarkably compared with state-of-art. Experimental results show that the proposed method has better consistency with human subjective perception than existing deep learning methods.