Abstract: With the rapid development of high speed railway in China and over the world, the quality requirements of rail tracks become higher and higher. Rail track quality is directly related to the train safety, national economy, security and others. Based on rail track surface images obtained under fixed light sources, we transfer the rail track surface defect detection problem into a low-rank matrix decomposition problem. For the obtained sparse matrix, the accumulation value of each row is calculated. Because the background is removed in the sparse matrix, the absolute row accumulation values of the defect area will be very large. Thus a threshold operation can be applied and the row indexes of the defect areas can be obtained. Finally, binarization operation is applied to the small image block determined by the row indexes and the maximal connected region in the binarization image block can be considered as the defect area. Our algorithm can realize the automatic defect detection and localization. Compared with the exit algorithms, our algorithm achieve a better results under varied illumination and backgrounds.