Physical fault injection attacks are becoming an important threat to computer systems, as fault injection equipment becomes more and more accessible. In this work, we propose a new strategy to detect fault attacks in cryptosystems. We use a recurrent neural network (RNN) to detect problems in the program flow caused by injected faults. Our neural network is trained using the instructions of non-faulty operations and therefore, it can protect against both current and future attacks. As a case study, we use two implementations of software RSA. To test the effectiveness of our detector, we propose a collection of fault injection models, where each model represents different types of faults in the instructions. Evaluation results show that we obtain a high detection accuracy in case injected faults lead to changes in the instruction flow and hence, making it difficult to steal secrete keys. Finally, we propose an efficient hardware implementation with only a 6% area overhead compared to a RISC-V processor.