Abstract: Scene image classification is a popular direction in machine vision. Scene images are characterized with rich content and complex concepts. Existing scene classification algorithms based on deep networks often improve the scene recognition effect by improving the network structure or data enhancement, but ignore the consideration of the relationship between scene elements and object elements in the image. Based on this context, the paper proposes a local feature saliency algorithm based on the analysis of the existing deep network-based scene classification technology. The algorithm aims to jointly consider the scene local features and the object local features, and use the complementary relationship between the two types of different features to optimize them separately to obtain a more discriminative description of the scene features. Experimental results on MIT Indoor67 dataset verified the effectiveness of the algorithm, with an accuracy of 88.88%.