Abstract: ‍ ‍In view of poor accuracy of mixing matrix estimation for underdetermined blind source separation， a mixing matrix estimation algorithm combining （combining density-based spatial clustering of application with noise， DBSCAN） and probability density estimation was proposed in this paper. Firstly， the criterion for single source time-frequency point detection was obtained through vector transformation to detect single source time-frequency points from mixed signal. Secondly， the density-based spatial clustering algorithm was used to cluster single source time-frequency points， so that the number of sound sources and the single source time-frequency points of each category were estimated. Thirdly， the clustering centers of each category were obtained by probability density estimation， and a mixing matrix was constructed. The proposed mixing matrix estimation method does not need to set the number of sound sources in advance and avoids the problem of poor clustering accuracy caused by uneven data distribution. Finally， compressed sensing technology was utilized for the source signal recovery， so as to separate each source signal from mixed signal. Experimental results show that the proposed method can accurately estimate the mixing matrix when the number of sound sources is unknown. And the separated signal has a high quality.