Abstract: By combining and demodulating signal copies from different paths, cooperative relay system achieves better bit error performance than non-cooperative system. Traditional schemes combine and demodulate signals separately. However it may lose some useful information in the separate processing procedures. So the system performance is bad. To overcome this problem, this paper presents a novel combining demodulation algorithm for multi-input multi-output (MIMO) amplify-and-forward (AF) cooperative relay system. The proposed algorithm is based on maximum likelihood (ML) rule. The process is, first, signals from source and cooperative relay are combined into traditional MIMO detected form, then ML detection is used to demodulate the combined signal. The analysis and simulation results demonstrate that compared with present combing demodulation algorithms, such as maximum ratio combing (MRC) demodulation algorithm, the ML combining demodulation algorithm has a significant performance gain for different modulations and channel conditions. The system complexity with this algorithm is lower than that with present algorithms for high-order modulation.