We are interested in adapting neural networks to the case of learning new classes in a sequential fashion. We propose to improve the learning procedure of one-versus-all invertible neural networks, a state-of-the-art method in class-incremental learning, to reduce its memory impact. We conduct our experiments on the CIFAR-100 dataset for which we learn each class one after the other. Our results show that the proposed approach is able to perform with a similar accuracy whilst reducing the memory cost.