Abstract: A single-channel speech enhancement based on jointly constrained double-layer dictionary learning is proposed to improve the quality of speech in the complex noisy environment.Firstly, the characteristic sub-dictionaries that describe the clean speech and noisy speech are trained. Then, with the new optimization function of discriminative constraints and anti-substitution constraints, a double-layer joint dictionary is trained. The first layer dictionary expresses the separable components of the speech signal and noisy signal, and the second layer expresses easily decomposed components of the speech signal and noisy signal. The constraint of the objective optimization function is used to reduce the occurrence of "cross-projection" phenomenon and the confusion of the signals in the joint dictionary. Furthermore, we can improve the effect of speech enhancement through the double-layer dictionary. The experimental results show that compared with the speech enhancement methods based on the non-negative matrix factorization with sparsity-regularized constraints and the improved wiener filtering, in three aspects including Signal to Noise Ratio(SNR),Perceptual Evaluation of Speech Quality(PESQ) and Logarithmic Spectral Distance(LSD), the proposed method has better performance and can remove noise more effectively.