Deep neural network technology is a milestone achievement in the field of computer vision. It obtained the performance that the shallow network cannot achieve through the multi-layer network structure and the learning method of reverse adjustment parameters. However, the feature extraction algorithm of the shallow network is very effective and also is more beneficial for deep neural networks. In this paper, we combine the algorithm of the shallow network to proposes the gradient local binary pattern layer(GLBP layer) to replace the first layer of Convolutional Neural Networks(CNNs). The GLBP layer plays a role in initializing the CNNs and can improve network performance without increasing the number and complexity of network layers. In the experiment, using the extracted layer modified by the GLBP feature algorithm to replace other classic deep neural network, 2.65\% and 2.9\% performance improvement were obtained in the WideResNet16-2 and ResNet-101 respectively when training on CIFAR-100 dataset.