Abstract: In order to overcome occlusion and track the pedestrian accurately, this paper proposes an object tracking algorithm that learns a set of new appearance models for adaptive discriminative object representation. First, we exploit the color invariance as the root feature to obtain the appearance feature. And then, object tracking is taken as a binary classification problem. The correlation of object appearance and class labels from foreground and background is modeled by the analysis from partial least squares, to generate a low-dimensional discriminative feature. As object appearance varies, we learn and adapt a series of appearance models with PLS analysis to achieve the robustness for tracking. Experiments on general data sets demonstrate good performance of the proposed tracking algorithm.