This paper presents a deep-learning-based inventory management system for smart refrigerators. The proposed method generates augmented images by considering various positions, angles, and illumination conditions with automatically generated labels. The cameras mounted on the refrigerator side check for incoming and outgoing objects. The results indicate that the accuracy of object recognition improved when multiple cameras were used. The proposed system was tested with 25 types of objects that included several fruits and drinks. Evaluation reveals that the proposed inventory management system achieved an accuracy of 84.63% for a set of test cases.