Abstract: Image annotation aims at assigning a set of semantic labels to describe the content of the image. Aiming at the gap of high-level semantics and low-level features in image annotation, this paper proposed an image annotation methodology based on partial order structure in this paper. At first, calculated the visual score and obtained the adjacent labels. Through the attribute partial order structure (APOS) diagram, the method can get the related semantics of the adjacent label. Specially, the related semantics was used to construct the object partial order structure (OPOS) diagram in the purpose of obtaining the final semantic neighbor set. Technically, set a threshold of the word frequency to select the labels with the higher frequency as the final keywords. More remarkably, the experimental results showed that the method effectively improves precision and recall rate of the annotation.