In conventional approaches of acoustic vector sensor (AVS) based direction of arrival (DOA) estimation under noisy and reverberant environment, all the raw time-frequency (TF) data are inputted for reliable TF points extraction, which is found not only redundant but also detrimental. In this work, the conventional AVS-based DOA estimation pipeline is reconfigured by introducing a frame-wise-single-source clustering (FWSSC) step which is used to minimize the possible contaminated TF points consumed by the following step such as single source points (SSP) detection and direct path dominant (DPD). Outliers are then removed based on the results of FWSSC. In the proposed FWSSC, the intensity vector instead of the estimated steering vector is used for single-centroid TF point clustering. It is shown that due to the reduced number of potentially contaminated TF points, the accuracy and robustness of DOA estimation can be significantly improved, especially for the cases where the angular differences between the sources are small. The proposed DOA estimation pipeline is applicable to any TF analysis based algorithms.