Recently, Reinforcement Learning (RL) has shown great performance in solving sequential decision making and control in dynamic environments problems. Despite its achievements, training Deep Neural Network (DNN) based RL is expensive in terms of time and power. In this paper, we propose a novel hardware architecture for RL agents based on the learning hierarchical policies method. We show that hierarchical learning with several levels of control improves RL agents training efficiency. Our method is important for efficient learning of policies for RL agent, especially when the target platform is a resource constraint embedded device.