Abstract: n online visual saliency map based tracking method via structural inverse sparse appearance model is proposed which develops the tracking framework based on a Bayesian framework to improve the performance of sparse representation based trackers. Firstly a correlation visual saliency detection method based on Markov model is formulated to calculate the saliency map of the target template of the current frame. Secondly, we design a structural global and local blocked appearance model to represent the candidates and obtain the adaptive weight of each pitch by mapping the saliency map to the candidate pitches. Finally, we utilize a novel combine mechanism to unite the global and local sparse solutions which is applied to measuring the similarity between the candidates and the template, then the optimum target state can be estimated and tracked under the Bayesian framework. In the procedure, we utilize an inverse sparse representation formulation which enables the tracker to compute the weights of all candidates by solving one optimization problem and this is conducive to improving the performance of our method. Experimental results demonstrate that the proposed algorithm has good robustness and realtime performance.