Abstract: Despite being investigated more and more, it is still challenging to construct a reliable hand-dorsa vein recognition system. One of the challenges is about designing models to recognize hand-dorsa vein pattern precisely even when the images suffer from severe information lost. To address this and formulate a more robust vein recognition model, a hierarchical incomplete vein image inpainting framework is proposed as a cascaded two-stage network. In stage-I, an encoder-decoder network is designed using the dilated convolutional kernel and non-local attention design for better exploiting the contextual information. The coarse prediction from stage-I is then refined with a novel multi-branch encoder-decoder network. By taking the coarse prediction, incomplete vein image, and the binary segmentation map as inputs, the three encoder branches produce complementary semantic features, which are then combined via concatenation to obtain a better semantic representation for the decoder to produce refined images with higher quality. To stabilize the training of the cascaded network, both the adversarial loss and perceptual loss are combined, and two discriminators in terms of global and local design are proposed to guarantee smooth details inpainting. Rigorous experiments are carried out to evaluate the usefulness of the proposed model for the challenging vein image inpainting tasks. On the one hand, both qualitative and quantitative evaluation is performed on hand-dorsa vein image and palmprint image inpainting task. On the other hand, the improvement of recognition results on the specifically-created incomplete vein dataset after inpainting further demonstrates the effectiveness of our inpainting framework.