Abstract: Nowadays, video surveillance system has been widely used. People paid more and more attentions on the group anomaly detection of video surveillance, which has become an urgent need to protect public security. In this paper, a new scheme for group anomaly event detection is proposed, in which it is efficient to automatically detect abnormal events in the video surveillance. For the feature extraction, a new feature descriptor called Histograms of Salience Optical Flow (HSOF) is proposed, which is used for the dictionary construction. Then a cluster-based multi-dictionary learning framework is proposed for dictionary optimization, which separates the original large dictionary into some sub-dictionaries. Finally, for the test samples, we can find the most suitable sub-dictionary and calculate the reconstruction error to determine whether the sample is normal or not. Compared with other methods for the group anomaly events detection in crowd scenes, experiments on two datasets show that our proposed method obtains better results.