基于条件可逆神经网络的多模态医学图像融合

Multi-modal Medical Image Fusion based on Conditional Reversible Neural Networks

摘要:

为减少图像融合在深层网络前馈过程中的空间细节损耗，提出一种基于条件可逆神经网络（CINN）的医学图像融合方法；通过应用可逆的分析-综合架构实现空间细节与关键语义互补的多模态融合；在前向分析阶段，将多分辨率特征嵌入CINN作为条件向量实现多模态表示学习；在反向综合阶段，使用一个基于小波的条件融合（WCF）网络引导CINN完成反向重构；在特征融合中应用相关激活模板（RAM），聚焦多模态医学图像中的关键结构区域与纹理细节一致性信息；构造前向分析-反向重构联合损失高效地优化网络参数，以获得高质量的融合图像；实验测试了CT-MRI及MRI-PET场景，与现有融合基线相比，提出方法在SCD以及VIFF等客观融合指标上性能分别提升了15.16%和46.53%，并且在主观视觉质量上均取得了优越的结果。

Abstract:

In order to reduce the spatial detail loss of image fusion in the feedforward process of deep network, a medical image fusion method based on conditional reversible neural network (CINN) is proposed. Through the application of reversible analysis-synthesis architecture, the multi-modal fusion of spatial details and key semantic complementarity is realized. In the forward analysis phase, multi-resolution features are embedded into CINN as conditional vectors to realize multi-modal representation learning. In the reverse synthesis stage, a wavelet-based conditional fusion (WCF) network is used to guide CINN to complete the reverse reconstruction. In feature fusion, the correlation activation template (RAM) is applied to focus on the consistency information of key structural regions and texture details in multi-modal medical images. The combination of forward analysis and reverse reconstruction is constructed to optimize network parameters efficiently to obtain high quality fusion images. CT-MRI and MRI-PET scenes were tested. Compared with the existing fusion baseline, the proposed method improved the performance of objective fusion indicators such as SCD and VIFF by 15.16% and 46.53%, respectively, achieved superior results in subjective visual quality.