This paper presents a light-weighted CNN-based BS segment recognizer. The recognizer features a two-staged workflow: the MFCC calculation and the CNN forward propagation. Experimental results show that the proposed recognizer attains the 91.25% and 90.83% mean accuracy for the ‘not-across-subjects’ and ‘across-subjects’ validation, respectively. Moreover, compared with the state-of-the-art LSTM approach, the CNN-based BS segment recognizer is light-weighted with only 20.35k parameters, which is a quarter of that of the LSTM. Due to the lower computation complexity, the CNN-based BS segment recognizer is preferable to be integrated into the wearable patches to reduce the overall power consumption of wearable BS monitoring systems.