Abstract:  For the blind separation of the mixtures of communication signals, in consideration of the bit error rate (BER) performance metrics in communication systems, a blind source separation (BSS) algorithm is proposed based on minimum bit error rate criterion in this paper. The basic principle of the proposed algorithm includes that: first of all, the new minimum bit error rate criterion constrained cost function is established according to the derived minimum bit error rate criterion incorporated into maximum likelihood (ML) principle; secondly, by virtue of nature gradient descent search, the blind separation is accomplished through minimizing this built cost function. Simulation experiments and analysis show that, in comparison with the blind separation algorithm based on the cost function of the maximum likelihood principle, the proposed algorithm based on minimum BER criterion constrained cost function can lead to better performance in speed of convergence and separation accuracy.