This paper introduces an approach to reduce the computational cost of object detection in CNN accelerators at the cost of very small additional memory overhead. Results show that the TempDiff method achieves up to 37% computation reduction with 1.1% accuracy drop on the SSD(VGG16) network, under both VIRAT and ImageNet-VID datasets. Similarly, 18.3% (35.8%) computation reduction at 3.3% (3.2%) memory overhead, and 3.8% (6.8%) accuracy drop in YOLOv1 (VGG16) (SSD (VGG16)) is achieved under the CAMEL dataset. Furthermore, up to 58% computation reduction with 2% accuracy drop and 3.7% memory overhead were achieved for YOLOv3-Tiny network under the ImageNet-VID dataset.