The application of deep learning (DL) in various brain computer interface (BCI) systems has achieved great success, but the results on the attention classification task are still not satisfactory. In this paper, an end-to-end mixed neural network model was proposed to classify the attention and non-attention mental states from multi-channel electroencephalography (EEG) data without any preprocessing. In our experiment, we mainly perform inter-subject transfer learning techniques as a classification strategy. Evaluated on different electrodes combinations of a publicly available dataset, the proposed model outperforms these baseline methods while maintaining relatively low computational complexity. The improved performance on attentive mental state classification task is meaningful for disease treatment and attention enhancement.