Abstract: The label area of a wine bottle contains its identification, while wine label segmentation can effectively eliminate the interference of the background in the image matching algorithm. Most of the conventional image segmentation algorithms are based on lowlevel features and human defined rules, which are sensitive to the noise and have trouble in processing massive data. In this paper, we first construct a large scale wine label segmentation dataset, then propose a wine label segmentation method based on deep learning to tackle the problems in the conventional algorithm. We design our semantic segmentation model based on a deep residual network with certain skip-layer connections which integrate the low-level and highlevel features together to achieve clearer marginal details of segmentation. Furthermore, we adopt Atrous Spatial Pyramid Pooling(ASPP) to enlarge the receptive field while segmenting multi-scale wine labels. It is demonstrated by the experimental results on our wine label dataset that the proposed wine label segmentation algorithm can achieve high accuracy in real-time.