Abstract: To estimate the direction of arrival (DOA) of coherent signals, a novel quaternion spatial smoothing (Q-SS) algorithm is proposed using simplified electromagnetic vector sensor array. Since the orthogonality of quaternion is quite suitable for describing the orthogonal relation of a vector sensor, the two-component complex-valued signal of each vector array element constitutes the quaternion-valued signal, and the quaternion model of simplified electromagnetic vector sensor array is first set. This kind of model is more appropriate for vector sensor array signal processing than the traditional long vector model. Then the newly proposed Q-SS algorithm is used to preprocess the incident coherent signals on the basis of the quaternion model. The preprocessing performance is also analyzed, and a quaternion ESPRIT method is applied to estimate the DOA in the sequel. Theoretical analysis show that the proposed algorithm results in a high estimation accuracy in contrast to the traditional spatial smoothing based on long vector model (V-SS). What is more, the Q-SS algorithm is able to decorrelate the coherent noise among the orthogonal components of a vector sensor without any additional procession, while V-SS method does not have the advantage of it. Finally, the performance of the proposed algorithm is evaluated through computer simulation examples, and its efficacy is verified.