Abstract: In this paper, the problem of average throughput maximization of the full-duplex relay-assisted cognitive MIMO system is considered. Compare to traditional relay-assisted cognitive radio system, the duplex relay of the proposed system model can not only cooperate with cognitive user source to perform multi-antenna spectrum sensing to enhance the performance of spectrum sensing, but also decode and forward the signal from cognitive user source to improve the system throughput. In order to maximize the system average throughput, the design optimization of frame structure is considered to obtain the optimal sensing time and maximum the average probability of spectrum holes discovery for cognitive user. And the transmit antennas are also optimized to select the best transmit antenna to send the signal. Then the optimal power allocation scheme is derived under the limitations of the total transmission power and the harmful interference to primary user. Simulation results show that the proposed system model and optimization scheme can achieve the better average throughput, compared to the traditional full duplex equal power allocation and half duplex power allocation scheme.