Abstract: Abnormal behavior detection has a wild range of applications in the field of intelligent monitoring systems. A video abnormal behavior detection and localization algorithm is proposed that combines the feature of optical flow and histogram of oriented gradients. Firstly, video background extraction algorithm is used to extract and label the foregrounds which could achieve the prospect segmentation information. Then histogram of oriented gradients and optical flow feature extraction algorithm are selected to extract the foreground images’ histogram and optical flow characteristics, following by support vector machines for the training and testing data. Finally, the abnormal behavior is located with information of the optical flow’s magnitude and the foreground label.