Abstract: Remote sensing image target detection can provide important usable information for military and civil fields and has become a research hotspot in recent years. Aiming at the problem that the existing target detection technology cannot give attention to both detection speed and accuracy, this paper optimized Faster R-CNN: the lightweight depth separable residual network was used as the backnone network of Faster R-CNN to reduce the number of parameters in the backnone network model. The multi-layer convolution features in the backnone network were fused after local response normalization to enhance the completeness of target feature information and improve the problem that small targets are easy to miss detection. The network model was trained by combining softmax loss function and center loss function to reduce the changes within categories and increase the differences between categories, so that the network model can learn more different target features. The proposed method was verified on VEDAI, NWPU VHR-10, DOTA datasets. Compared with the traditional Faster R-CNN, the detection accuracy of the proposed method is improved by 7.0%.