Sparsity of data and weights appears in many convolution neural networks (CNN) and recurrent neural networks such as long-short term memory (LSTM). In this paper, we design a sparsity-aware deep learning hardware accelerator exploiting both data and weight sparsity in CNN and LSTM models. The proposed hardware accelerator significantly reduces memory accesses and computations, leading to much lower power consumption.