Abstract: The hierarchical B picture prediction structure adopted by multiview video coding scheme (JMVC) introduces extremely high computational complexity. To speed up the encoding process, the exhaustive mode decision is analyzed first, and then a fast inter frame mode decision algorithm for multiview video coding is proposed in this paper. In our approach, the prediction modes are firstly classified according to their size, and the rate distortion costs are calculated from the largest to the smallest. Based on the correlation of rate distortion cost and the size of prediction mode, the mode decision process can be earlier terminated. If the rate distortion cost of the last (larger size) prediction mode is lower than that of the current (smaller size) mode, skip checking the rest prediction modes and take the last one as the optimal mode. On the contrary, if the rate distortion cost of the last (larger size) prediction mode is higher than that of the current (smaller size) one, continue the checking process. Experimental results have shown that the proposed algorithm can reduce the computational load by 81.66%, while keeping almost the same coding efficiency compared to the full search mode decision of JMVC.