This paper performs an in-depth study of the errorresiliency of Neural Networks(NNs). Our investigation found that Binary Neural Networks (BNNs) are more error-tolerant than 32-bits NNs. Further, we found that in BNNs, errors in Batch Normalization Parameters (BNPs) are more sensitive to network accuracy than binary weights. We discuss the reason behind such behavior in the paper. Then, based on this fact, we propose a split memory architecture for low power BNNs, suitable for IoTs. In the proposed split memory architecture, weights are stored in area-efficient 6T SRAM, and BNPs are stored in robust 12T SRAM.