The development of lightweight object detectors is essential due to the limited computation resources. To reduce the computation cost, how to generate features plays a significant role. This paper proposes a new lightweight convolution method Cross-Stage Lightweight Module (CSL-M). It combines the Inverted Residual Block (IRB) and Cross-Stage Partial (CSP) concept. Experiments conducted at CIFAR-10 show that the proposed CSL-Net based on CSL-M performs better with fewer FLOPs than the other lightweight backbones. Finally, we use CSL-Net as the backbone to construct a lightweight detector CSL-YOLO, achieving better detection performance with only 43%$ FLOPs and 52% parameters than Tiny-YOLOv4.