This paper presents the architecture and the implementation for a hybrid fixed-point binary convolutional neural network (H-CNN) targeting tactile data processing application. H-CNN combines quantization and binarization operations to achieve a low computational complexity with an acceptable accuracy. When implemented on FPGA, H-CNN architecture achieved a real-time classification within 0.8 ms while consuming 53 mW dynamic power. Compared to existing solutions, H-CNN offers a significant speedup of up to 6875x accompanied with 99.6% energy reduction while recording up to 7% increase in the classification accuracy of touch modalities.