In this paper, we design an efficient skeleton detector and a lightweight dynamic hand gesture recognition network to establish the contactless HCI system. To construct the whole system, we have to collect the required dynamic hand recognition dataset for application purpose. We propose a two-stage skeleton-based CNN model to facilitate it. This system is applied to the home appliance control system. The simulation results show that it can achieve 99.4% accuracy in the dynamic hand gesture recognition task on our dataset and outperform other methods on SHREC dataset. Also the whole system achieve at least 13 FPS on the embedded system in practice.