Abstract: In classification problem of multiple observation sets, the samples are represented as points on Grassmannian manifolds, with regard to how to exploit the manifold structure to improve the classification performance，a multiple observation sets classification algorithm based on inconsistent similarity measure graph is presented. First of all, considering the characters of global and local data structure comprehensively, an inconsistent similarity measure is constructed, which regards the distinction of within-class and between-class mainly and can effectively reflect the distribution character of actual data clustering. The second step is to obtain the similarity matrix via inconsistent similarity measure graph, after that, the computation of the optimal map is transformed into the search problem of the largest eigenvectors of the Rayleigh quotient by a combined Grassmannian kernel and then the projection matrix is obtained. Lastly, points on the manifold can be mapped into another space, the final classification is completed exploits the nearest neighbor classifier. Three comparative experiments are conducted on ETH-80 object recognition dataset, CMU-PIE and BANCA face recognition datasets, the results prove that the algorithm performs better than traditional algorithm.