Abstract: In hand-free telephone systems and mobile communication devices, the loudspeakers often work with large volume and are easily in overload status, so obvious nonlinear acoustic echo occurs. This phenomenon is more usual in devices with small- or micro-loudspeakers. The general AEC (Acoustic Echo Cancellation) algorithm cannot cancel the nonlinear echo, thus the communication speech quality degrades significantly. The nonlinear echo has many extra high-frequency harmonic distortions, and these distortions break the linear constraint of a full-band adaptive filter algorithm. Since the general AEC algorithms are derived to minimize the full-band error, their performances are very sensitive to the nonlinear distortion. This paper proposes a subband AEC algorithms based on polyphase filter-bank, which transforms the fullband error into subband errors, and those harmonic distortions become additive noise in corresponding subbands. In those bands that distortions are small, the subband adaptive filter can convergence normally. In simulation and real tests, the results show that when the nonlinear distortion occurs, the proposed method achieves significantly better ERLE (Echo Return Loss Enhancement) than the typical time-domain and frequency-domain methods. For speech signals with obvious nonlinear distortion, ERLE improves about 10 dB.