This paper describes a software and hardware co-design approach for implementing a neural network to improve radar signal processing. At the algorithm level, we propose using the ResNet10 model structure and other optimizations trained using the angle-Doppler spectrum of returns at each range. The FPGA implementation is carefully optimized to better tradeoff performance and energy efficiency. Experimental results show our approach achieves better performance than conventional methods and exceed the requirement by more than2.5×. Meanwhile our energy consumption is much lower than other platforms like GPU. Our optimization methods can be applied to other CNN structures for efficiency improvement.