唐宏, 朱龙娇, 范森, 刘红梅. 基于光流法与伪三维残差网络的微表情识别[J]. 信号处理, 2022, 38(5): 1075-1087. DOI: 10.16798/j.issn.1003-0530.2022.05.020
引用本文: 唐宏, 朱龙娇, 范森, 刘红梅. 基于光流法与伪三维残差网络的微表情识别[J]. 信号处理, 2022, 38(5): 1075-1087. DOI: 10.16798/j.issn.1003-0530.2022.05.020
TANG Hong, ZHU Longjiao, FAN Sen, LIU Hongmei. Micro-expression Recognition Based on Optical Flow Method and Pseudo Three-dimensional Residual Network[J]. JOURNAL OF SIGNAL PROCESSING, 2022, 38(5): 1075-1087. DOI: 10.16798/j.issn.1003-0530.2022.05.020
Citation: TANG Hong, ZHU Longjiao, FAN Sen, LIU Hongmei. Micro-expression Recognition Based on Optical Flow Method and Pseudo Three-dimensional Residual Network[J]. JOURNAL OF SIGNAL PROCESSING, 2022, 38(5): 1075-1087. DOI: 10.16798/j.issn.1003-0530.2022.05.020

基于光流法与伪三维残差网络的微表情识别

Micro-expression Recognition Based on Optical Flow Method and Pseudo Three-dimensional Residual Network

  • 摘要: 微表情是一种动态变化的面部表情,具有复杂的时空特征,给其识别带来了极大的困难。本文提出一种基于光流法与伪三维残差网络(P3D ResNet)的微表情识别方法,通过光流法对微表情运动信息建模,为网络提供关键信息的同时丰富数据空间维度,采用伪三维残差网络进一步学习微表情的时间和空间特征。首先,将三个主流的微表情数据集进行融合,并对融合的数据集进行预处理;然后使用TVL1光流法提取表征微表情运动信息的光流特征序列,将得到的光流特征序列与微表情灰度图像序列进行通道连接,形成一个新的三通道微表情图像序列;最后将获得的微表情数据进行数据增强送入伪三维残差网络同时提取微表情的时空特征以实现微表情的识别。其中,P3D ResNet是在残差网络的框架中采用二维卷积滤波器提取微表情的空间特征,一维卷积滤波器提取微表情的时间特征来模拟三维卷积滤波器。在融合数据集上的实验表明,本文方法的性能相对基准方法有了显著的改进,UF1和UAR分别提高了14.71%、14.58%。本文提出的方法在融合数据集及三个独立数据集上的识别性能优于现有较先进的方法,从而证明了本文的微表情识别方法的先进性和鲁棒性。

     

    Abstract: ‍ ‍‍ ‍Micro-expression is a dynamically changing facial expression with complex temporal and spatial characteristics, which brings great difficulties to its recognition. A micro-expression recognition method based on optical flow method and pseudo three-dimensional residual network (P3D ResNet) was proposed. By means of optical flow, the movement information of micro-expression is modeled, which provides the network with key information and enriches the spatial dimension of data. The temporal and spatial characteristics of micro-expression were further studied by psedu-3D residual network. Firstly, the three mainstream micro-expression datasets were fused, and the fused datasets were preprocessed. Then, the TVL1 optical flow method was used to extract the optical flow feature sequence representing the micro-expression motion information, and the obtained optical flow feature sequence was connected with the micro-expression gray image sequence by channel to form a new three-channel micro-expression image sequence. Finally, the obtained micro-expression data was enhanced and fed into the psedu-3D residual network. Meanwhile, the spatial and temporal features of micro-expression are extracted to realize the recognition of micro-expression. Among them, P3D ResNet used two-dimensional convolution filters to extract spatial features of micro-expression in the framework of residual network, and one-dimensional convolution filters to extract temporal features of micro-expression to simulate three-dimensional convolution filters. Experiments on fused dataset showed that the performance of method in this paper had a significant improvement over the benchmark method, UF1 and UAR had increased by 14.71%, 14.58%, respectively. The recognition performances of the method proposed in this paper are better than the existing advanced methods on the fused datasets and three independent datasets, which proves the advancement and robustness of the micro-expression recognition method in this paper.

     

/

返回文章
返回