Abstract: In classification problem of multiple observation sets, multiple observations of the same object were viewed as a whole for recognition, which respect all single samples equally. Considering the different amount of discriminative information from each single sample, with regard to how to exploit their respective different discriminant information, multiple observation sets classification algorithm based on joint sample weighted sparse representation is proposed. Multiple observation sets are first divided into single samples and each sample is processed separately by a sparse solution, obtained its respective sparsity and residual, then use them jointly to obtain its different reliability. After that, each single sample is weighted by corresponding reliability, the weighted multiple observation sets are reconstructed. Lastly, the classification is completed by global sparse representation. Extensive comparative experiments are conducted on ETH-80 object dataset, CMU-PIE face dataset and BANCA datasets, the results verify the effectiveness of the proposed algorithm, enhancing the recognition accuracy and the robust of the algorithm.