We propose a hardware-friendly RGB-D head pose estimation system with fewer parameters. The total number of parameters is 0.19 M, including RGB and depth path, which is 50% lower than FSA-Net. We outperform advanced methods in terms of the Yaw angle and average error by introducing an attention module and feature decouplers. We achieved 3.1 and 3.5 MAE on yaw and average, which is 22.69% lower than QuatNet and 7.4% lower than FSA-Net. The inference speed is 0.92 ms per pair RGB-D images, which is 8% faster than FSA-Net.