Abstract: In this paper, we present a n-gram model based on fuzzy representation, in allusion to the problem of data sparsity and sharply of maximum likelihood estimation that the traditional statistical language model confront. We apply it to the lip reading system, combine with Hidden Markov Model (HMM), establish a novel lip movement recognition model HFM(HMM and Fuzzy Language Model). A small vocabulary corpus was built by using the corpus online system developed by the Ministry of Education Institute of Applied Linguistics Computational Linguistics Research Laboratory for carrying out sentence recognition experiments. The experimental results demonstrate that HFM(did not need smoothing) can improve syllable recognition rate by up to 6.5%, and sentence recognition rate by up to 22.7%. In addition, using language model for text stream analysis, instead of blindly text matching, analytical accuracy of single visual flow can be up to 68.7%.