A low-power, highly-secure always-on face recognition (FR) processer is proposed for secure applications such as user authentication. We newly propose a branch net-based early stopping FR (BESF) processor for the purpose of adversarial attack prevention and low power consumption. It shows a recognition accuracy of 83.10% and 71.97% under the fast gradient signed method (FGSM) and projected gradient descent (PGD) attack. BESF processor can reduce 30.85% to 74.82% power consumption by using clock-gating during the FR scenario. Unified pointwise convolution and depthwise convolution processing element adopts layer-fusion to reduce the external memory access (EMA) by 88.0% and achieves 74.1% higher throughput. Furthermore, noise injection layers are added between every bottleneck layer in order to reduce the FGSM/PGD attack success rate by 9.29% and 20.0%. The processor is simulated with a 65 nm CMOS process with a 3.0 mm × 3.0 mm chip size. It consumes 0.22-0.89 mW power at 1 frames-per-second (fps) and shows 95.5% FR accuracy in LFW dataset.