Abstract: ‍ ‍The large number of antennas in a massive multiple-input multiple-output （MIMO） radar system enables performance gains in detection and estimation tasks. An extended target detection in a massive MIMO radar system has been considered in this study. Assuming the presence of an extended target with numerous scatterers， the received signal model for the massive MIMO radar system was established. The spatial correlation of the target reflection coefficients was analyzed， where any two antennas with correlated target reflection coefficients were considered to fall within the same target beam； otherwise， they were considered to fall within different target beams. Antennas falling within the same target beam were treated as an antenna group. During detection， each antenna group first applied matched filtering to the received signal. The result of matched filtering was then processed by whitening and normalization filters. Based on the outputs of these two filters， the energy of all antennas within the group was summed， and the divergence was calculated to obtain an energy detector. Although obtaining a closed-form solution for the distribution of divergence was challenging， we proved that when the number of antennas was sufficiently large， the statistical effect of massive antennas could be used to derive the distribution of divergence. On this basis， the closed-form expression for the detection probability of extended target detection in a massive MIMO radar system was further derived. Moreover， it was proven that the energy detector can achieve optimal detection performance under the Neyman-Pearson criterion. Through simulations， the correctness of the theoretical results was verified， and the impact of different system parameters on the detection performance of the massive MIMO radar system was further analyzed.