Abstract: At present,there exist the problems such as poor model generalization ability, insufficient data, and easy over-fitting in Facial Beauty Prediction research. The Cross-Stitch Network automatically determines the shared layer by activating multiple networks for end-to-end learning, but it ignores the issue of image information priority and is not conducive to Multi-task Learning to distinguish features. Therefore, this paper improves the Cross-Stitch Network, replacing part of its layer network with Self-Attention module and LSTM module, to realize the parameter sharing between layers and between modules. First, we perform image preprocessing, including uniform size, face alignment, image enhancement, normalization, and image cropping. Second, we initialize the constructed Improved Cross-Stitch Network, in which the sharing between layers is called "Micro-sharing" , and the sharing between modules is called "Module sharing". Finally, the trained model is tested. Experimental results show that by the Improved Cross-Stitch Network, the accuracy of Facial Beauty Prediction is as high as 63.95%, higher than the highest accuracy rate of 62.97% by conventional methods. Therefore, the Improved Cross-Stitch Network provides a new idea for Multi-task Learning.