Abstract: Learning face representation is a crucial problem of face recognition system. Recently deep face representation based on convolutional neural networks (CNN) have achieved state-of-the-art performance on public LFW (Labeled Faces in the Wild). However, the deficiency of generalization and robustness still exists in complex datasets, such as IARPA Janus Benchmark A (IJB-A). In this paper, a total variability modelling (TVM) method utilizing different output layers of CNN for face representation is proposed. The variations of local deep features can be modeled in the total variability subspace, which effectively aggregates the local deep features into a compact embedding feature (iVector). Evaluations on the IJB-A dataset show the proposed face representation learning method, compared with existed face deep representations, has achieved better performance and efficiency.