Defocus blur detection aims to accurately differentiate the defocus blurred regions and in-focus clear regions in a natural image. In spite of remarkable advances, the disturbing of background clutter and the homogenization of boundary remains as the two most challenging issues. To address these issues, we propose an end-to-end network with an Adaptive Dual Attention Module (ADAM), which is designed to simultaneously capture effective local boundary detail and global semantic information in the channel and spatial dimensions, and dynamically aggregate the multi-level features. Specifically, ADAM pays attention to the correlation between local and global features associated with blur objects enhancing the synthetical feature representation. Furthermore, we design a Detail Optimization Module (DOM), which is subsequently employed to correct the uncertain detection in boundary regions within the low-resolution details. The comprehensive experiments on two common datasets demonstrate our proposed method has superior performance for dealing with defocus blur detection.