Binaural sound source localization determines the position of a sound source based on three types of cues that contain head-related transfer functions. Many audio processing systems in our daily work and life rely on SSL such as speech enhancement/recognition, and human-robot interaction to name a few. However, the accuracy of SSL under adverse acoustic scenarios is still hard to be ensured. This paper proposes long short-term memory with a feature extraction technique based on scattering decomposition to improve the estimation accuracy of the loudspeaker position. The obtained results demonstrate that the proposed method achieves great performance in multiple noisy environments in comparison to recent literature methods considering reduced sequence data. Finally, the proposed method shows the capability of regression output to estimate azimuth and elevation angles using only 27% of localizations for training.