Abstract: Rate control in H.264 is a key technology which improves the quality of the compressed video via controlling the bit rate of output streaming. An improved algorithm is proposed on the basis of the JVT-H017 algorithm in this paper. According to the distribution of DCT coefficients in H.264, Cauchy rate module which replaces traditional quadric rate distortion model is adopted into the rate control. Moreover, a prediction algorithm which is used to predict the image complexity via PSNR ratio and MAD ratio is also proposed in this paper, and then the frame bits and QP can be adjusted based on the predicted results. Thus, in the case of complex movement or scene change, this prediction method can overcome the inaccuracy of MAD prediction method in the JVT-H017 algorithm because of the reduced correlation in the adjacent video frames. Simulation results show that, compared with the algorithm in JVT-H017 and reference 5, the new algorithm not only achieves more accurate rate control, but also obtains more stable output bit rate and better PSNR value of reconstructed image.