In this paper, we propose a three-stream adaptively fusion network which uses paired RGB image and thermal image for crowd counting. The three-stream network is divided into one main stream and two auxiliary streams. We merge a pair of RGB and thermal image to constitute the input of main stream. Two auxiliary streams use RGB image and thermal image respectively as input to extract modality-specific features. Besides we propose Information Improvement Module(IIM) to adaptively fuse modality-specific features with feature extracted from main stream. Experiment results on RGBT-CC dataset shows that our method achieves 20.7%, 14.9%, 11.4%, 8.2%, 20.3% improvement on GAME(0), GAME(1), GAME(2), GAME(3) and RMSE respectively compared with state-of-the-art method.