In this paper, we proposed an auto-encoder framework based on the BiLSTM and the self-attention for abnormal driving detection, both of them are embedded in the model. The BiLSTM is used for estimating the long-term correlations in the sequence, and the self-attention is used for exploring the internal relationship of spatial-temporal features. As a result, the auto-encoder is capable of restructuring features error using small and representative features. Experimental results show that the proposed framework outperforms the other baselines with recall and F1-score as 96.2% and 95.0%, respectively.