Abstract: In order to improve the performances of channel estimation and multi-user detection in the grant-free non-orthogonal multiple access （NOMA） system based on the compressive sensing （CS） theory， a genetic algorithm （GA） based optimization method for spread spectrum matrix was proposed in this paper. In this method， a genetic algorithm was proposed to solve the combinatorial optimization problem to extract several rows from the Fourier transform matrix as the spreading spectrum matrix in NOMA system with the aim of minimizing the mutual coherence of the spreading spectrum matrix. As compared with the existing genetic algorithm that solved the similar problem， the genetic algorithm proposed in this paper was more novel in individual structure and could converge to a smaller mutual coherence of the spreading spectrum matrix. Simulation results show that in the grant-free NOMA system based on the multiple measurement vector-compressive Sensing model， as compared with using a Gaussian random matrix as the spreading spectrum matrix， using the optimized matrix obtained by our proposed method can reduce the symbol error rate of the system by 52.14% and increase the successful activity detection rate by 12.14% on average. Apart from that， the reduction of the channel estimation mean square error is about 10 dB.