Abstract:  Aiming at the conflict between convergence speed and interference cancellation ratio(ICR) of radio frequency(RF) domain self-interference cancellation algorithm which based on the minimum mean square error (MMSE) criterion in full-duplex systems, proposing an improved time-varying step-size normal least mean square algorithm. This method on the basis of the self-interference cancellation construction, building nonlinear relationship between timevarying step-size and modified sigmod function, the autocorrelation of real-time error signal and time parameter t are utilized to vary step-size in order to better convergence speed and ICR. Analysis and simulation show that RF domain ICR of proposed method reached to 88 dB higher than similar algorithms at least 1.5 dB and the convergence speed and the ability to resist sudden interference are significantly improved on the full-duplex 2FSK systems model which interference to signal ratio(ISR) is 80 dB, step interval Δt=1/32 ms, signal to noise ratio Eb/N0=10 dB.