Abstract: In order to improve the performance of blind source separation, a new geometrical algorithm based on quadratic programming theory was proposed. Since the geometrical properties of the scatter plots of uniform distribution source signals in different processing stages were special, it could be used to separate the mixed signals in the manner of geometry. Suppose that the mixed matrix is non-singular, to begin with, the mixed signals were whiten, and then the datasets which could be classified into two classes were constructed by simply letting the original data move proper distance along an appropriate direction. The principal object was to make the direction parameters of optimal separating hyper-plane be related to the arguments of rotating matrix of the scatter plots. The direction parameters of optimal separating hyper-plane were obtained by solving the problems of quadratic programming. Finally, the rotating matrix was calculated by the estimated direction parameters, and then the source signals were estimated further. Theory analysis and simulation experiments all show that the new method can achieve superior performance in separating the mixed signals.