Abstract: The trajectories for action recognition that were extracted by previous methods contained irrelevant motion changes of background, and the Orientation-Magnitude descriptor of trajectory shapes lacked the robustness due to camera movement. To solve these problems, action recognition by tracking the salient relative motion points was proposed in this paper. Firstly, motion boundary detector which suppressed the camera constant motion was utilized to extract motion features. After processing them by the adaptive threshold, super-pixels which contained salient motion boundaries were defined as relative motion regions. Then a method to track the interest points within relative motion super-pixels was employed to generate trajectories. For the trajectory shape, the pre-defined multiple directional patterns were used to produce distribution statistics of direction of trajectory points. Finally, the descriptors like oriented gradient, motion boundary, oriented statistic and their combined representation were fed into the classifier for recognizing actions, respectively. On KTH and UCF-sports datasets, the extracted trajectories can describe the motion changes of objects, and the directional statistics boosts the robustness to the trajectory shape. Compared with the related literature, our method obtains good recognition performance.