A resource-efficient neural network based face detector using 1.5-bit frame-to-frame delta quantization with diagonal spatial feature extraction method is proposed in this paper, which is designed for resource-limited always-on camera sensors. The proposed architecture completes analog-domain frame difference for motion sensing, which triggers digital-domain feature extraction. Based on the sparse and effective features, a lightweight convolutional neural network is devised as a face detector. Simulation results show that the proposed method achieves 93.6% accuracy using only a 50×50 pixel array. Meanwhile, the conservative estimated power consumption of the proposed method can be reduced by 14× compared to the state-of-the-art work.