Abstract: Support Vector Machine (SVM) is a state-of-the-art machine learning algorithm based on the statistical learning theory. It tries to find the optimal classification hyperplane in high dimensional feature space and has a good classification accuracy and generalization performance. Since the training of SVM requires solving a restrained quadratic programming problem, in order to overcome the defect of the classical training method becoming difficult for large datasets, an improved Shuffled Frog Leaping Algorithm(Im-SFLA) is proposed as an alternative to the current classical algorithm, which not only improves the convergence speed and accuracy of SFLA but also provides an effectual approach to the optimal selection of parameters about SVM by means of the global random searching ability of SFLA. In this paper, the prosodic feature, the voice quality feature and the chaos characteristic parameter of the emotional statement are extracted firstly, a data fusion method based on Im-SFLA is proposed and the emotion recognition of pratical speech is researched by use of modified SVM. In the recognition experiments, PCA method, BP neural network and data fusion method are compared under the same testing environment. The test result indicates that the improvement mechanisms provided in this paper bring an outstanding improvement in the classification ability and provide a new method and idea for speech emotion recognition.