Abstract: In order to effectively detect bus drivers’ misbehavior. Paper proposed an algorithm that can detect the irregular behavior of bus drivers while driving, and studied the detection algorithm combined with human pose estimation information for improving the accuracy of detection in the target environment. Finally, a multi-stage hand motion detection method is established. The method consists of three modules. Firstly, human pose estimation module, The Gaussian heat map layer of the human pose estimation network is selected, and the Gaussian heat map information of the human pose is output to achieve the constraint on the detection target space information. Secondly, the hand detection module, based on the CNN detection network, achieves the effect of improving the detection rate of the hand after the human pose Gaussian heat map is integrated at the network input layer. Thirdly, hand motion classification module, by accepting the output of the hand detection module, eliminating the background of interference with the detection result, constraining the feature extraction of the classification network to the local area of the hand, and improving the accuracy of the hand motion classification to the final Human body motion detection effect. In order to verify the multi-stage hand motion detection method proposed in this paper, corresponding experiments have been performed on homemade data sets.