Abstract: ‍ ‍As the number of wireless communication devices increasing rapidly， the shortage of wireless spectrum resource is getting worse. Integrated sensing and communication （ISAC） technology realizes sensing function and communication function at the same time， by using integrated signal， which relives the shortage of spectrum resource and saves the hardware resource at the same time， so that the ISAC technology attracts more attention. The non-orthogonal multiple access （NOMA） based ISAC system has a better performance than traditional ISAC system， but it is a difficult point to effectively eliminate the interference of sensing signal in communication receiver. In order to solve the above problems， a frame structure of ISAC signal based on NOMA system was proposed in this paper. In a frame of the ISAC signal， the orthogonal frequency division multiplexing （OFDM） communication signal and linear frequency modulation （LFM） sensing signal were Non-orthogonal superposition with power wight. Then a new interference cancellation algorithm was proposed， in this paper. In the communication receiver， according to the characteristics of LFM signal， short time Fourier transform （STFT） was used to analyze the time-frequency characteristics of the LFM signal and get the approximate range of LFM signal’s parameters. And then the convergence characteristic of LFM signal by specific order fractional Fourier transform （FRFT） was used to estimate the parameters of LFM signal precisely. And the maximum likelihood compensation technology was used to correct the LFM signal’s parameters estimation result， which could improved the accuracy of parameter estimation， so that the sensing signal was reconstructed accurately and then removed thoroughly in communication receiver. Simulation results showed that when the signal to noise radio （SNR） was greater than 15 dB and the power weight coefficient was less than 0.01， the mean square error （MSE） of the LFM signal’s parameters estimation by FRFT was less than 40 dB， and the MSE of the LFM signal’s parameters estimation was less than 70 dB by maximum likelihood compensation correcting. At the same time， the channel capacity achieved by the proposed interference cancellation algorithm， in this paper， was close to that of the ideal interference cancellation.