With the development of deep learning, face recognition is attracting more and more attention in both industry and academia. Hardware implementation of face recognition systems on heterogeneous embedded devices, however has been rarely studies. In this paper, an embedded face recognition system is designed and implemented on FPGA SoC platforms. A hardware-software partition method is first introduced by analyzing the ratio between computation and memory access of critical tasks in the system. Several acceleration methods are then exploited to optimize the hardware implementation. The face recognition system is implemented on Xilinx FPGA MPSoC ZCU102 with 97.6% recognition accuracy and 296 ms latency. The neural network VIPLFace, as the most time consuming part of the system, has a 74 ms latency, 71x faster after hardware-software co-design.