Abstract: As an important tool for processing non-stationary signals, the ambiguity function (AF) has been widely used in radar signal processing, sonar technology, etc. It does well in estimating linear frequency modulation signals. However, it fails in estimating the Quadratic FM signal which is required in many fields. As the generalization of the Fourier transform, the fractional Fourier transform has attracted widespread attention these years. In this paper, in order to estimate parameters of Quadratic frequency modulation signals, we discuss the ambiguity function based on the fractional Fourier transform. Some new basic but important properties of the fractional ambiguity function are discussed, such as symmetry and conjugation property, shifting property and Moyal formula. As well the relationships between the fractional ambiguity function and other time-frequency analysis distributions are derived, including the classical ambiguity function (AF), the Wigner distribution function based on the fractional Fourier transform, the short-time Fourier transform (STFT), and the wavelet transform (WT). At last the fractional ambiguity function is applied for estimating the Quadratic frequency modulation signal. The simulation indicates that this new arithmetic is feasible and effective.