Compilers and code optimisations have specific characteristics that directly impact applications\' code footprint, performance, power efficiency, and reliability. In this scenario, this paper investigates the impact of widely adopted compilers on the soft error reliability of convolutional neural network (CNN) inference models executing on a RISC-V processor. Fault injection campaigns consider two fault targets (registers and memory), two open-source compilers (GCC 8.1.0 and Clang 12.0.1), five code optimisation levels, and two CNN inference models, resulting in 680k fault injections. Results show that optimisation flags can lead to more than two orders of magnitude increase in the occurrence of critical faults.