Abstract: In speaker verification, Acoustic factor analysis uses MPPCA algorithm to derive a mixture dependent dimensionality reduction of the acoustic feature in every single component of Universal Background Model, which can eliminate channel mismatch and noise interference and use the enhanced speaker information to improve the performance of speaker verification. However, UBM is trained in an unsupervised method and each Gaussian has no defining acoustic meaning, which can’t distinguish between different speakers saying different types of phoneme. To address this, this paper replaced UBM with Deep Neural Network of ASR acoustic model in acoustic factor analysis and derived a phoneme dependent dimensionality reduction of the acoustic feature to extract speaker information which was used to extract the DNN i-vector for speaker verification. The experiment on RSR2015 PartIII showed that acoustic factor analysis based on the phoneme can achieve a valid reduction of 13.49% and 22.43% at the EER compared to acoustic factor analysis based UBM when evaluated on male and female test set separately.