Abstract: For the problem of the performance of the bit error rate (BER) at the receiving end of the transform domain communication system (TDCS) under the spectrum sensing mismatch condition, a new optimization design of basic function is proposed. The method can dynamically design the available frequency of the basic function for different mismatch conditions, and achieve the optimization trade-off between system anti-interference performance and frequency band utilization under spectrum sensing mismatch, thereby improving the BER performance of system under spectrum sensing mismatch. Firstly, the model of long-distance TDCS is established, and the electromagnetic interference environment of transceiver and receiver is analyzed. Secondly, by analyzing the performance of the basic function under the spectrum sensing mismatch condition, deriving the system BER expression theoretically, the optimal model of the basic function amplitude spectrum is established and the steepest descent method is used to search the optimal solution. The simulation results show that compares with the existing methods, the proposed method makes a more reasonable design of the basic function amplitude spectrum, and the anti-interference performance is significantly improved under different mismatch ratios, and has better Signal-to-noise ratio BER performance.