Page 253 - 《软件学报》2026年第5期

2132 软件学报 2026 年第 37 卷第 5 期

4
(State Key Laboratory of Computer Science (Institute of Software, Chinese Academy of Sciences), Beijing 100190, China)
5
(University of Chinese Academy of Sciences, Nanjing, Nanjing 211135, China)
Abstract: With the rapid development of large language model (LLM) technology, many Code LLMs have emerged to support tasks such
as code generation, code completion, code testing, and code refactoring. Different models may show significant performance differences
when processing the same task, and the decoding parameters at the inference stage will also have an important influence on model
performance. This study investigates how to efficiently select the best model and its optimal decoding parameters for a specific code
development task. Existing methods generally divide model selection and parameter tuning into two independent stages. Due to the
differences in sampling strategies at different stages, sample data cannot be shared, and the computational cost of sampling and evaluation
is high. Considering that the decoding parameter space of different Code LLMs is the same, this study proposes the utilization of the
propensity score matching (PSM) algorithm for conducting weighted adjustment and aligning sample data of different distributions to
improve the reuse efficiency of sample data and reduce computational costs. Therefore, this study proposes a framework CodeLLMTuner
for Code LLM selection and decoding parameter tuning based on sample reuse. The framework includes three stages, including the
independent sampling stage, which performs decoding parameter tuning (such as Bayesian optimization) on multiple Code LLMs in
parallel and conducts data sampling and evaluation to collect sample data. Additionally, the model selection stage adopts PSM technology
to align the sample data of different models and selects the model with the optimal performance expectations. The decoding parameter
tuning stage of the selected model reuses the sample data of the selected model and continues decoding parameter tuning based on it to
fully explore the performance space and significantly reduce sampling costs. Experimental results show that in the three tasks of code
generation, code summarization and test case generation, CodeLLMTuner improves performance by 10% to 15% at the same cost
compared to the baseline methods, or reduces the cost by more than 20% under the same performance.
Key words: Code LLM; automatic code generation; model selection; decoding parameter tuning; propensity score matching (PSM)
[3]
[4]
[2]
大语言模型技术的快速发展推动了 Codex 、StarCoder 、Code Llama 等代码大模型和 GitHub Copilot 等
[1]
智能开发助手的兴起 [5] . 这些专门针对编程语言和软件开发任务训练的代码大模型被广泛应用于自动代码生成 [6–9] 、
代码修复 [10,11] 、代码测试 [12] 和代码摘要 [13] 等任务. 当前代码大模型提供了多种可调的解码参数, 如温度和惩罚参数,
这些参数可以影响代码大模型输出的结果, 从而影响代码开发任务的性能. 本文中“性能”指代码大模型完成特定软
件开发任务时的质量, 如代码生成任务中代码的正确性和测试用例生成任务的代码覆盖率. 而不同的训练数据集和
模型结构导致不同的代码大模型在完成软件开发任务时的性能有明显差异. 这种模型与参数差异导致的性能差异直
接影响软件开发工作的效率与质量, 如表 1 所示, 同一模型的不同解码参数以及不同模型的同一解码参数所对应的
代码生成任务正确率可能存在极大差距, 因此用户需要针对特定软件开发需求选择合适的代码大模型与解码参数.

表 1 不同代码大模型在 HumanEval 上采用不同温度解码参数时的性能 (%)

代码大模型 评价指标 temperature=0.2 temperature=0.4 temperature=0.6 temperature=0.8 temperature=1.0
pass@1 48.18 45.65 44.90 43.31 36.59
Code Llama 34B pass@4 60.42 62.82 67.35 68.83 63.32
pass@10 66.34 70.10 77.92 81.02 77.66
pass@1 27.95 28.96 26.79 26.92 25.10
WizardCoder 34B pass@4 39.52 48.42 48.39 55.82 53.85
pass@10 44.71 58.05 60.01 72.47 69.82
pass@1 49.84 54.77 52.89 52.40 52.34
PhindCoder 34B pass@4 61.69 71.02 74.19 74.13 78.07
pass@10 66.71 77.43 81.74 81.77 87.17

通常来讲, 为充分探索解码参数性能空间, 代码大模型的解码参数调优需要对多组解码参数进行采样和评估,
即从解码参数空间选择某组配置, 并在测试基准 Benchmark 上评价所选择的配置对应的性能. 然而对于代码开发
任务, 这一过程通常会消耗大量时间和资源, 而庞大的参数空间与候选代码大模型的数量也使得采样与评估成本
成倍增加. 以表 1 为例, 该表仅包括 3 个候选模型与 1 维解码参数共计 15 个采样点, 但对于每个采样点均需要为
HumanEval 共 164 项任务中的每项任务重复生成 40 条以上目标代码并进行评估, 以计算生成代码通过率的期望
值, 若设最大长度为 400, 则该表需要消耗 15×164×40×400 = 39360000 个 token, 以 Together AI 平台的报价为例约
为 31 美元. 而代码大模型选择与参数调优通常需要对数十个模型与上百个采样点进行采样与评估. 因此, 代码大
模型选择与解码参数调优的主要挑战在于如何在有限的成本下高效地选择更优的模型和参数 [14] .