Abstract: Aiming at mismatching points problem in the image matching, a theory based on Structural Similarity (SSIM) to eliminate the image mismatching points is proposed. Whole similarity obtained from luminance, contrast and image construction as the image mismatching points elimination standard. Firstly, this algorithm computes matching points neighborhood window’s structure similarity, then eliminate matching points that structural similarity is less than the threshold. Finally, according to the matching points in the image space geometry distribution to further eliminate the mismatching points. Comparing to the existing based on random sample consensus algorithm and based on gray correlation algorithm to eliminate mismatching points, the experimental results show that the proposed algorithm can achieve better effect to eliminate mismatching points. Its comprehensive performance prior to the other two algorithms, and timeliness is also better than the algorithm that based on random sample consensus algorithm to eliminate mismatching points.