An energy-efficient convolutional neural network accelerator is proposed for real-time segmentation in autonomous electric vehicle system. The computation of semantic segmentation with high-resolution image makes it difficult for real-time operation in time-critical and resource-constrained AEV. To facilitate real-time implementation in AEV, this paper proposes two key features: 1) A compressed multi-object Depth-fused Trilateral Network with dilated convolution and depthwise separable convolution that reduces 90% of the overall computation of baseline and achieves 94.73% accuracy on KITTI Road dataset; 2) An energy-efficient CNN accelerator, which supports 5 types of CONV’s, achieving 1.33× higher throughput than previous processor.