In this paper, a novel multimodal convolutional neural network (CNN) fusion architecture is proposed for heart sound signal classification. Instead of using features from just one domain, general frequency features as well as Mel domain features are extracted from the raw heart sound. The multimodal CNN fusion architecture is individually trained based on the feature maps resulting from various feature extraction methods. These feature maps are then merged for optimizing the diversified extracted features. The proposed method provides an opportunity to explore the optimal selection of features for heart sound classification. Extensive experimentations are carried out, showing that an outstanding accuracy of 98.5% is achieved by the multimodal CNN architecture using the trimodal system, which outperforms the other state-of-the-art approaches.