Abstract: Classic Mean Shift based tracking algorithm uses fixed kernel-bandwidth leads to scale and spatial localization inaccuracy. Based on CBWH algorithm which gives the precise location of a moving object, the proposed algorithm generates a color probability distribution with object background weighted model in RGB color space, and calculates the invariant moment to resize the tracking window of the next frame.To meet certain conditions, the background weighted model will be updated timely to adapt to the complex background of tracking. Experimental results show that the proposed algorithm improves the robustness of object tracking by self-adaptive kernel-bandwidth updating.