Abstract: ‍ ‍High-quality images are the fuels for computer vision tasks， but the existence of highlights in real life will cover the texture and color information of the object surface， resulting in substantial image quality degradation. Current deep learning based methods for highlight removal often require a large number of highlight-highlight-free paired images for supervision， while the corresponding highlight-free versions of the highlight images are hard to collect and process. This paper proposes an image specular highlight removal algorithm based on weakly supervised learning， which uses only highlight images in training and achieves promising highlight removal performance. First， a sparse non-negative matrix factorization （NMF） method is used to estimate the highlight region of the image， and a highlight-free reference image is cropped from the highlight-free region. Then， the unpaired highlight and highlight-free images are fed into the jointly trained highlight generation， highlight removal， and image reconstruction modules to optimize the functionality of the respective module. The overall network is trained with a Cyclic Generative Adversarial Network （CycleGAN） architecture and eventually generates a highlight-free image. The natural image datasets SHIQ and LIME are selected for experiments. The experimental results show that the proposed method can effectively remove specular highlights， and the performance is substantially improved compared with the existing weakly supervised learning methods.