Page 318 - 《软件学报》2026年第4期

软件学报 ISSN 1000-9825, CODEN RUXUEW E-mail: jos@iscas.ac.cn
2026,37(4):1759−1776 [doi: 10.13328/j.cnki.jos.007460] [CSTR: 32375.14.jos.007460] http://www.jos.org.cn
©中国科学院软件研究所版权所有. Tel: +86-10-62562563

*
基于分类检索的操作规划方法

吴益露, 王瀚霖, 王利民

(南京大学 计算机学院, 江苏 南京 210023)
通信作者: 王利民, E-mail: lmwang@nju.edu.cn

摘 要: 聚焦于教学视频 (instructional video) 中的操作规划 (procedure planning) 问题, 探讨如何根据给定的开始和
结束视觉状态, 在教学视频提供的动作空间中规划出一条将开始状态转变为结束状态的动作序列. 教学视频以记
录和展示各种事件的操作过程为特点, 每个事件对应一组特定动作, 从而形成事件的动作空间. 多个事件的动作空
间共同构成了教学视频的整体动作空间. 传统方法未能充分挖掘事件的语义信息, 过于依赖强化学习等复杂训练
方法, 既增加了算法设计的复杂性, 又导致模型的可解释性较差. 针对这些问题, 结合教学视频的特点, 提出了一种
基于分类检索的操作规划器 CPP (classification-retrieval-based procedure planner), 分阶段解决操作规划任务. 具体
而言, 该方法首先通过视觉状态识别事件类别, 将动作空间限定在一个较小的子空间内, 显著降低规划的复杂性;
随后, 在该子空间中进行动作序列的规划. 此外, 提出了一种混合规划策略, 将动作序列的检索与预测相结合, 进一
步提升了规划性能. 实验结果表明, 该方法在 3 个不同规模的教学视频数据集上均取得了显著效果, 为操作规划任
务提供了一种简单而高效的基准方法.
关键词: 操作规划; 教学视频; 视觉推理
中图法分类号: TP18

中文引用格式: 吴益露, 王瀚霖, 王利民. 基于分类检索的操作规划方法. 软件学报, 2026, 37(4): 1759–1776. http://www.jos.org.cn/
1000-9825/7460.htm
英文引用格式: Wu YL, Wang HL, Wang LM. Classification-retrieval-based Procedure Planning Method. Ruan Jian Xue Bao/Journal
of Software, 2026, 37(4): 1759–1776 (in Chinese). http://www.jos.org.cn/1000-9825/7460.htm

Classification-retrieval-based Procedure Planning Method

WU Yi-Lu, WANG Han-Lin, WANG Li-Min
(School of Computer Science, Nanjing University, Nanjing 210023, China)
Abstract: This study focuses on the problem of procedure planning in instructional videos. Given the start and end observations, the task
is to plan an action sequence that transforms the start state into the end state within the action space provided by the instructional videos.
Instructional videos record and demonstrate the operational processes of various events. Each event includes a specific set of actions,
forming the action space for that event. Therefore, the action space is composed of various subspaces corresponding to different events in
instructional videos. Previous methods fail to effectively utilize the semantic information of events and overly rely on techniques such as
reinforcement learning, resulting in complex training schemes and poorly explainable approaches. In contrast, this study considers the
characteristics of instructional videos and proposes the classification-retrieval-based procedure planner (CPP), a pipeline that addresses
procedure planning from coarse to fine. Specifically, the planner first identifies the event category based on the given observations,
narrowing the action space to a smaller subspace. Then, action planning is performed within the selected subspace, which is significantly
easier than planning in the entire action space. Moreover, this study introduces a hybrid planning method that combines retrieval and
prediction approaches to generate the action sequence. The proposed method achieves competitive results on three popular procedure
planning datasets of varying scales, establishing itself as a simple yet robust baseline for procedure planning.
Key words: procedure planning; instructional video; visual reasoning

* 基金项目: 科技创新 2030—“新一代人工智能”重大项目 (2022ZD0160900)
收稿时间: 2025-01-07; 修改时间: 2025-02-10; 采用时间: 2025-04-29; jos 在线出版时间: 2025-11-05
CNKI 网络首发时间: 2025-11-06