基于注意力机制的多模态人体行为识别算法

Multi-modal human behavior recognition algorithm based on attention mechanism

摘要:

提出了基于注意力机制的多模态人体行为识别算法。针对多模态特征的有效融合问题,设计基于注意力机制的双流特征融合卷积网络TAM3DNet(Two-stream Attention Mechanism 3D Network)。主干网络采用结合注意力机制的AM3DNet(Attention Mechanism 3D Network),将特征图与注意力图进行加权后得到加权行为特征,从而使网络聚焦于肢体运动区域的特征,减弱背景和肢体静止区域的影响。将RGB-D数据的颜色和深度两种模态数据分别作为双流网络的输入,从两条分支网络得到彩色和深度行为特征,然后将融合特征进行分类得到人体行为识别结果。

Abstract:

A multi-modal human behavior recognition algorithm based on attention mechanism is proposed. Aiming at the problem of effective fusion of multimodal features, a two-stream feature fusion convolutional network TAM3DNet (Two-stream Attention Mechanism 3D Network) based on attention mechanism is designed. The backbone network adopts AM3DNet (Attention Mechanism 3D Network), which combines the attention mechanism, and weights the feature map and the attention map to obtain the weighted behavior characteristics, so that the network focuses on the characteristics of the limb movement area and reduces the influence of the background and the static area of the limb. The color and depth modal data of the RGB-D data are respectively used as the input of the dual-stream network, and the color and depth behavior features are obtained from the two branch networks, and then the fusion features are classified to obtain the human behavior recognition results.