Abstract: This paper presents a novel algorithm based on single Gaussian model for video background modeling and moving object detection, in which video image is divided into several blocks and pixels in each block are modeled together. Compared to traditional per-pixel background modeling method, the proposed block modeling method matches the characteristics of Gaussian distribution better and more conducive to exploiting the advantages of single Gaussian modeling method. So it enhances the capability to deal with complex background and effectively reduces the complexity in operation. Furthermore, this paper proposes a new CMNO (Compare and Merge the Nearest Ones) strategy to handle the fragment problem in foreground so that the nearest broken agglomerate is merged to construct a new object. The experimental results have verified the effectiveness of proposed method.