Abstract: Hand segmentation based on skin color model is a commonly used hand segmentation method. However, this kind of method is easy to be affected by background to cause serious mis-segmentation and is not adaptable to different skin tones. So, this paper proposes a method to roughly detect hand gesture first and then segment hand adaptively. SSD is improved into a hand gesture detection model, where a backbone network based on dilated convolution and a set of Anchor setting scheme are designed, which can preliminarily segment the gesture ROI to avoid the influence of skin color background on the gesture segmentation. Then, the YCrCb gaussian skin color model was built according to the gesture ROI, so that the skin color model could adapt well to different skin colors of gestures. The experimental results demonstrate the robustness and effectiveness of proposed method for gestures with different skin color in a variety of complex scenarios