Abstract: ‍ ‍High-quality data is one of the key factors for the success of deep convolution neural network. In the field of computer vision， commonly used image data sets are usually stored in JPEG format. Because of the lossy compression technology， part of the original data information is inevitably lost. It further results in the performance degradation of convolutional neural network trained with compressed data. Therefore， in order to enhance the convolutional neural network， this paper proposes a joint enhancement framework， including the restore module and the task module. The restore module aims to recover the information loss caused by the lossy compression technology. According to the task requirements， the task module focuses on enhancing the compressed image. The joint training of these two modules makes the restoration and enhancement of compressed images more purposeful. Experiments on the image classification task show that our method can effectively restore the compressed image and enhance the performance of convolutional neural network. In addition， the low coupling and substitutability of the two modules in the method make the method applicable.