Abstract: A novel directions of arrival(DOA) estimation method is developed when uncorrelated and coherent sources coexist in nonuniform noise. Its implementation is based on an iterative procedure which includes least squares(LS) problem with respect to the signal subspace and noise nuisance parameters. In our methods, the noise covariance matrix is first iteratively calculated from the array covariance matrix. Then the noise component in the array covariance matrix is eliminated to achieve a noise-free array covariance matrix. The rotational invariance techniques(ESPRIT) can be employed to achieve DOA estimation of the uncorrelated signals. Secondly, the uncorrelated signals are eliminated and a new matrix is constructed based on the spatial difference method. Next, the forward spatial smoothing can be used to realize the covariance matrix rank recovery. Finally, the signal and noise subspaces are achieved by eigendecomposing the noisefree covariance matrix, traditional subspace-based DOA estimation root-MUSIC algorithms can be applied directly. The proposed methods can resolve more signals than the conventional method with better performance. Simulation results demonstrate the effectiveness of the proposed method.