Page 113 - 《软件学报》2026年第6期

2432 软件学报 2026 年第 37 卷第 6 期

Adaptive Knowledge Distillation for Lightweight Large Code Models

3
1
1
2
1
SHU Shan-Fu , LIU Chao , SUN Yu-Zhong , ZHANG Hong-Yu , GAO Cui-Yun , ZHANG Xiao-Hong 1
1
(School of Big Data & Software Engineering, Chongqing University, Chongqing 401331, China)
2
(Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China)
3
(School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Shenzhen 518055, China)
Abstract: Software programming assistants based on large language models (LLMs), such as Copilot, significantly enhance programmer
productivity. However, LLMs have large computing and storage requirements and are difficult to deploy locally. Building a lightweight,
small LLM can meet computing, storage, and deployment requirements, but it leads to a greater accuracy loss in code generation compared
to large LLMs. Knowledge distillation (KD) techniques allow small LLMs (student models) to approximate the output distributions of
large LLMs (teacher models) on target training datasets, thus reducing accuracy loss in code generation. Cutting-edge KD techniques in
artificial intelligence are based on the Kullback-Leibler (KL) divergence loss function, which measures and reduces accuracy loss due to
discrepancies in the output distributions between student and teacher models. However, student models struggle to learn in the near-zero
distribution regions of teacher models. Consequently, researchers have employed the reverse KL (RKL) divergence loss function to address
this issue in near-zero distribution regions. This study finds that RKL faces learning challenges in high-probability distribution regions and
complements the KL divergence loss function. For some datasets, low-quality outputs from teacher models lead to poor learning outcomes
for the student models. This study proposes an adaptive knowledge distillation (AKD) method that uses prompts to enhance teacher model
output quality and constructs an adaptive loss function to adjust learning priorities based on the distributional differences between student
and teacher models. This ensures the student model effectively learns in both primary and near-zero probability regions. Using the AKD
method, this study trains a lightweight large code model based on StarCoder-1B/7B (student/teacher models) and the CodeAlpaca dataset,
evaluating accuracy loss and code quality issues. Experimental results show that the lightweight model size is reduced by 85.7%. On the
HumanEval and MBPP data sets, prompts with clear instructions improve teacher model code generation quality, reducing the average
accuracy loss of the trained student model by 6%. The AKD-trained model’s average accuracy loss compared to the teacher model
(StarCoder-7B) is 17.14%, a 30.6% reduction over the original student model. The AKD-trained model’s accuracy loss is reduced by an
average of 19.9% compared to state-of-the-art KD and RKD methods. Regarding inference memory requirements, the KD and RKD
methods require 54.7 GB, while the AKD method only adds 3 GB. In terms of training time, the AKD method incurs a 30% increase.
However, even when the KD and RKD methods are trained for the same duration, their average performance improves by only 3%, which
is 16.9% lower than that of the AKD method. Therefore, the additional training cost of the AKD method is justified. Moreover, applying
the AKD method to the Code Llama and CodeGen model series reduces accuracy loss by an average of 19.2% compared to state-of-the-art
KD and RKD methods, demonstrating the generalizability of the AKD method.
Key words: code generation; large language model (LLM); knowledge distillation (KD)
[1]
[2]
[3]
随着大语言模型 (large language model, LLM) 的发展, 如 GPT-4 、OPT 和 LLaMA , 代码生成任务在软件
工程领域取得了显著进展. 基于 LLM 的软件编程助手, 如 GitHub Copilot (https://copilot.microsoft.com/), 能够自动
补全代码、生成函数和类的实现, 并提供即时的编程建议, 提升程序员的开发效率. 然而, LLM 通常拥有至少数十
亿的参数, 导致其在资源受限的设备上无法运行 [4] . 研究表明在资源受限的用户端中实现高效部署和使用应小于
20 亿的参数 [5] , 10 亿及以下小参数模型是经常研究的对象 [6−8] . 但 1B (billion) 的小参数 LLM 相较于数十亿参数
的模型精度损失大 [9] , 知识蒸馏 (knowledge distillation, KD) 技术能够让小参数 LLM (学生模型) 在目标训练数据
集上拟合大参数 LLM (教师模型) 的生成分布, 降低代码生成的精度损失, 使小参数 LLM 能够达到大参数 LLM
的代码生成能力.
在人工智能领域, Hinton 等人 [10] 提出 KD 方法, 采用 KL 散度损失函数度量学生模型与教师模型之间的生成
分布差异, 指导学生模型学习. 学生模型与教师模型参数量差距较小, 生成分布差异小, KL 散度能够有效度量生成
分布差异, 引导学生模型学习 [11−16] . 教师模型与学生模型的参数量差距大、生成分布差异大. 基于 KL 散度的知识
蒸馏方法需要计算两个模型的分布差异. 由于教师模型的生成分布中存在较多趋零区域, 导致 KL 散度计算结果
无法正确引导学生模型学习教师模型的生成分布 [17] . 为解决上述问题, Malinin 等人 [17] 提出 RKL 散度损失函数,
让学生模型与教师模型生成分布中的趋零概率保持一致, 在此前提下降低其余分布区域的差异. Gu 等人 [9] 将
RKL 散度损失函数应用于通用大语言模型, 轻量化效果比 KL 散度好.
近年来, 研究人员提出了很多代码大模型, 如 Hugging Face 团队 [18] 研发的 StarCoder 支持多种编程语言, 能高