Abstract: In the process of feature extraction, video behavior recognition algorithm has the problem of not focusing the salient area information of video image, which makes the model classification effect not ideal. In order to improve the ability of distinguishing network attention, an algorithm model of video multi-scale temporal sequence behavior recognition incorporating attention mechanism is proposed. The channel-space attention module and the channel attention module are respectively integrated into the video long and short sequence network. In the training process, attention mechanism is introduced to the model to make the network redistribute the weight. The attention mechanism captures the video content and location points of interest and improves the expression ability of the network. Experiments were performed on the Something-SomethingV1 and Jester datasets to verify our behavior recognition method. The results show that the performance of the video multi-scale time-sequence fusion behavior recognition network with robust attention module is effectively improved and shows certain advantages compared with other behavior recognition networks.