剪枝与蒸馏结合的高效神经网络压缩方法

An Efficient Neural Network Compression Method Based on Structured Pruning and Knowledge Distillation

摘要:

为应对深度神经网络在图像分类等任务中高计算成本与存储需求的问题，研究了一种结合结构化剪枝与知识蒸馏的模型压缩方法，旨在压缩模型的同时维持或提升其性能。采用基于L1正则化的结构化剪枝方法削减模型参数与计算量，保留主要特征提取能力；随后引入输出层引导的知识蒸馏技术，结合交叉熵损失与KL散度，将教师模型的logits分布传递给学生模型，以恢复其性能。进一步利用Optuna进行超参数自动调优，优化蒸馏温度、学习率与损失权重，提升蒸馏效率。实验结果表明，该方法在将VGG19_BN模型大小压缩55%–90%、计算量降低40%–74%的同时，精度提升3.82%，并具备良好的泛化能力。

Abstract:

To address the high computational cost and storage demands of deep neural networks in tasks such as image classification, a model compression method combining structured pruning and knowledge distillation is proposed to maintain or even enhance performance while compressing the model. The method employs L1-regularized structured pruning to reduce parameters and computational load while preserving key feature extraction capabilities. Subsequently, output-guided knowledge distillation is applied, integrating cross-entropy loss and KL divergence to transfer the teacher model’s logits distribution to the student model for performance recovery. Optuna is further utilized for automated hyperparameter tuning, optimizing distillation temperature, learning rate, and loss weights to enhance distillation efficiency. Experimental results show that this method reduces the size of the VGG19_BN model by 55%–90% and computational load by 40%–74%, while improving accuracy by 3.82% and maintaining strong generalization capability.