This paper presents a closed-loop detection strategy with significant potential to solve the detection defects existed in the residence-times difference method of the coupled core fluxgate magnetic sensors. The external magnetic field (e.g. Earth’s magnetic field), as the main source of these detection defects, has been analyzed by the theoretical investigation. To overcome such deficiencies, the close-loop detection method based on feedback technology is proposed to implement magnetic field compensation and achieve a zero-field running regime. The fluxgate with optimized detection strategy and vertical arrangement structure has been designed, implemented and tested, and the preliminary results showed steady operation and a detection resolution of ±0.2 nT. The optimized detection method proposed in this paper demonstrated a great potential to achieve the optimal detection performances which are embedded in the coupling-induced oscillation of nonlinear dynamics.