In this paper, we propose Multi-Scale Group Convo- lution (MSGC) an optimization to the conventional convolutional layer, to address the high computational complexity issue in deploying convolutional neural networks (CNN) on the Internet of Things (IoT) enabled edge sensors. The proposed method reduces complexity by grouping input channels of a convolution layer into smaller groups, thereby reducing the number of intermediate connections and complexity of matrix computations in a CNN. This approach results in a minor performance loss, which is compensated by utilizing a characteristic of group convolution to extract multi-scale features.