面向多目标动态作业车间调度的强化学习决策算法研究*

doi:10.16731/j.cnki.1671-3133.2025.07.003

现代制造工程 ›› 2025, Vol. 538 ›› Issue (7): 20-30.doi: 10.16731/j.cnki.1671-3133.2025.07.003

• 先进制造系统管理运作 • 上一篇下一篇

面向多目标动态作业车间调度的强化学习决策算法研究^*

张宁宁¹, 万卫兵¹, 张梦晓¹, 赵宇明²

1 上海工程技术大学电子电气工程学院,上海 201620;
2 上海交通大学自动化系,上海 201100

收稿日期:2024-06-11 出版日期:2025-07-18 发布日期:2025-08-04
通讯作者: 万卫兵,副教授,硕士生导师,博士,主要研究方向为人工智能。E-mail:wbwan@sues.edu.cn
作者简介:张宁宁,硕士研究生,主要研究方向为智能制造、深度强化学习。张梦晓,硕士研究生,主要研究方向为人工智能。E-mail:m020220118@sues.edu.cn; 赵宇明,副教授,硕士生导师,博士,主要研究方向为图像处理、模式识别和计算机视觉。E-mail:mx_zhang@ sues.edu.cn
基金资助:
^*科技部科技创新2030——“新一代人工智能”重大项目(2020AAA0109300)

Research on reinforcement learning decision algorithm for multi-objective dynamic job shop scheduling

ZHANG Ningning¹, WAN Weibing¹, ZHANG Mengxiao¹, ZHAO Yuming²

1 School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China;
2 The Department of Automation,Shanghai Jiao Tong University, Shanghai 201100, China

Received:2024-06-11 Online:2025-07-18 Published:2025-08-04

摘要/Abstract

摘要： 为求解多目标动态作业车间调度问题,在调度环境规模可变的情况下满足制造车间的实时调度需求,提出一种将近端策略优化(Proximal Policy Optimization,PPO)与GoogLeNet结合的方法即GLN-PPO。使用多维矩阵构造调度问题的状态空间,设计基于多种优先级规则的动作空间以及多目标奖励函数。为验证所提算法的有效性,分别在基于公共算例的静态公共环境、基于实际算例的静态实际环境和动态实际环境中训练并测试算法的性能。实验结果表明,与遗传算法相比,GLN-PPO能够提供高质量调度结果,满足企业的实时调度要求并且能够灵活应对规模可变的调度环境。

关键词: 深度强化学习, 作业车间调度, GoogLeNet, 近端策略优化

Abstract: To address the multi-objective dynamic job shop scheduling problem and meet the real-time scheduling needs of manufacturing workshops in environments with variable scales, a method combining Proximal Policy Optimization (PPO) with GoogLeNet, named GLN-PPO, is proposed. This method constructs the state space of the scheduling problem using multidimensional matrices, designs an action space based on various priority rules, and devises a multi-objective reward function. To verify the effectiveness of the proposed algorithm, it is trained and tested in three environments: a static public environment based on common benchmark problems, a static real environment based on actual cases, and a dynamic real environment. Experimental results show that compared to genetic algorithms, GLN-PPO can provide high-quality scheduling results, meet the real-time scheduling requirements of enterprises, and adapt flexibly to environments with variable scales.

Key words: deep reinforcement learning, job shop scheduling, GoogLeNet, Proximal Policy Optimization (PPO)

中图分类号:

TP18
TH164

张宁宁, 万卫兵, 张梦晓, 赵宇明. 面向多目标动态作业车间调度的强化学习决策算法研究^*[J]. 现代制造工程, 2025, 538(7): 20-30.

ZHANG Ningning, WAN Weibing, ZHANG Mengxiao, ZHAO Yuming. Research on reinforcement learning decision algorithm for multi-objective dynamic job shop scheduling[J]. Modern Manufacturing Engineering, 2025, 538(7): 20-30.

参考文献

[1] LI J, PAN Q, LIANG Y C.An effective hybrid tabu search algorithm for multi-objective flexible job-shop scheduling problems[J]. Computers & Industrial Engineering, 2010,59(4):647-662.
[2] 胡蓉,伍星,毛剑琳,等.融入概率学习的混合DE求解绿色分布式可重入作业车间调度[J/OL]. 控制理论与应用:1-10[2024-04-21]. http://kns.cnki.net/kcms/detail/44.1240.TP.20240301.0841.006.html.
[3] 罗哲,朱光宇,杨志锋,等.多策略相结合粒子群算法求解作业车间调度问题[J/OL]. 计算机集成制造系统:1-24[2024-04-21]. https://doi.org/10.13196/j.cims.2023.0611.
[4] 黄洋鹏,李玲玲,李丽.基于改进双档案多目标进化算法的柔性作业车间批量流混排调度[J/OL]. 计算机应用研究:1-11[2024-04-21]. https://doi.org/10.19734/j.issn.1001-3695.2023.09.0499.
[5] 史双元,熊禾根.考虑外协的作业车间无拖期调度问题多目标差分进化算法[J/OL]. 计算机集成制造系统:1-27[2024-04-21]. https://doi.org/10.13196/j.cims.2023.0550.
[6] 常大亮,史海波,刘昶.具有紧时高能耗工序特征的多目标调度优化问题求解[J/OL]. 中国机械工程:1-12[2024-04-25]. http://kns.cnki.net/kcms/detail/42.1294.th.20240308.1647.008.html.
[7] 轩华,蔡舒跃,李冰.改进遗传禁忌算法求解含恶化效应和多时间约束的柔性作业车间调度[J/OL]. 工业工程与管理:1-19[2024-04-25]. http://kns.cnki.net/kcms/detail/31.1738.T.20240226.1806.013.html.
[8] WEMELSFELDER M.Approximating optimal solutions for Job Shop Scheduling Problems with unrelated machines in parallel using generalizable deep Multi-Agent Reinforcement Learning[D]. [S.l.] :University of Amsterdam, 2020.
[9] WU Z, FAN H, SUN Y, et al.Efficient multi-objective optimization on dynamic flexible job shop scheduling using deep reinforcement learning approach[J]. Processes, 2023, 11(7): 2018.
[10] WANG H, CHENG J, LIU C, et al.Multi-objective reinforcement learning framework for dynamic flexible job shop scheduling problem with uncertain events[J]. Applied Soft Computing, 2022,131:109717.
[11] CHANG J, YU D, HU Y, et al.Deep reinforcement learning for dynamic flexible job shop scheduling with random job arrival[J]. Processes,2022,10(4):760.
[12] CHANG J, YU D, ZHOU Z, et al.Hierarchical reinforcement learning for multi-objective real-time flexible scheduling in a smart shop floor[J]. Machines, 2022,10(12):1195.
[13] 陈勇,王昊天,易文超,等.基于元胞机与强化学习的多扰动车间调度算法[J]. 计算机集成制造系统, 2021,27(12):3536-3549.DOI:10.13196/j.cims.2021.12.015.
[14] XU K, YE C, GONG H, et al.Reinforcement Learning-Based Multi-Objective of Two-Stage Blocking Hybrid Flow Shop Scheduling Problem[J]. Processes,2023,12(1):51.
[15] LENG J, WANG X, WU S, et al.A multi-objective reinforcement learning approach for resequencing scheduling problems in automotive manufacturing systems[J]. International Journal of Production Research,2023,61(15):5156-5175.
[16] ZENG Z, LI X, BAI C.A Deep Reinforcement Learning Approach to Flexible Job Shop Scheduling[C]//2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC).[S.l.] : IEEE, 2022:884-890.
[17] WU X, YAN X.A spatial pyramid pooling-based deep reinforcement learning model for dynamic job-shop scheduling problem[J]. Computers & Operations Research, 2023,160:106401.
[18] PALOMBARINI J A, MARTÍNEZ E C.End-to-end on-line rescheduling from Gantt chart images using deep reinforcement learning[J]. International Journal of Production Research, 2022,60(14):4434-4463.
[19] SZEGEDY C, LIU W, JIA Y, et al.Going deeper with convolutions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.[S.l.] :[s.n.] ,2015:1-9.
[20] WU X, YAN X, GUAN D, et al.A deep reinforcement learning model for dynamic job-shop scheduling problem with uncertain processing time[J]. Engineering Applications of Artificial Intelligence, 2024,131:107790.
[21] ZHANG C, SONG W, CAO Z, et al.Learning to dispatch for job shop scheduling via deep reinforcement learning[J]. Advances in Neural Information Processing Systems, 2020,33:1621-1632.
[22] HAN B A, YANG J J.Research on adaptive job shop scheduling problems based on dueling double DQN[J]. Ieee Access, 2020,8:186474-186495.
[23] LUO Y, DONG K, ZHAO L, et al.Balance between efficient and effective learning: Dense2sparse reward shaping for robot manipulation with environment uncertainty[J]. arXiv preprint arXiv,2020,2003:02740.
[24] LIN M, CHEN Q, YAN S.Network in network[J]. arXiv preprint arXiv,2013,1312:4400.
[25] CHEN L, LU K, RAJESWARAN A, et al.Decision transformer: Reinforcement learning via sequence modeling[J]. Advances in neural information processing systems,2021,34:15084-15097.
[26] SHANG J, KAHATAPITIYA K, LI X, et al.Starformer: Transformer with state-action-reward representations for visual reinforcement learning[C]//European conference on computer vision.Cham: Springer Nature Switzerland,2022:462-479.

面向多目标动态作业车间调度的强化学习决策算法研究^*

Research on reinforcement learning decision algorithm for multi-objective dynamic job shop scheduling

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	夏良才, 陈仕军. 基于Dueling-DQN的协同进化算法求解绿色分布式异构柔性作业车间调度问题^*[J]. 现代制造工程, 2025, 538(7): 8-19.
[2]	廖雪超, 向桂宏, 阮兵, 田芮利, 钟实. 带AGV数量约束的柔性作业车间调度问题研究^*[J]. 现代制造工程, 2025, 537(6): 11-21.
[3]	金桥, 杨光锐, 王霄, 徐凌桦, 张芳. 基于A-TD3的码垛机器人轨迹规划^*[J]. 现代制造工程, 2025, 536(5): 42-52.
[4]	徐帅, 李艳武, 谢辉, 牛晓伟. 基于卷积金字塔网络的PPO算法求解作业车间调度问题^*[J]. 现代制造工程, 2025, 534(3): 19-30.
[5]	刘亮, 贺禹铭, 祁思远. 基于数字孪生仿真的柔性作业车间调度优化研究^*[J]. 现代制造工程, 2025, 534(3): 41-51.
[6]	杨丹, 舒先涛, 余震, 鲁光涛, 纪松霖, 王家兵. 深度强化学习求解动态柔性作业车间调度问题^*[J]. 现代制造工程, 2025, 533(2): 10-16.
[7]	杨逢海, 杨晓英, 裴志杰, 武亚琪, 张志伟. 基于深度强化学习的风电拉挤板生产智能排程^*[J]. 现代制造工程, 2025, 532(1): 23-32.
[8]	谢子健, 秦建军, 曹钰. 基于改进TD3的四足机器人非结构化地形运动控制^*[J]. 现代制造工程, 2025, 532(1): 33-41.
[9]	闫富乾, 石致远, 王立闻. 基于改进灰狼算法的柔性作业车间动态节能分批调度问题^*[J]. 现代制造工程, 2024, 520(1): 24-32.
[10]	李峥峰;丁其聪;张东方;张国辉. 改进离散麻雀搜索算法求解柔性作业车间调度问题[J]. 现代制造工程, 2023, 516(9): 18-27.
[11]	陆心屹;韩晓龙. 基于强化学习的改进NSGA-Ⅱ求解柔性作业车间节能调度问题[J]. 现代制造工程, 2023, 515(8): 22-35.
[12]	吴迎晨，肖彪，赵正彩，彭仕鑫，苏宏华，朱夏林. 柔性作业车间调度多策略果蝇优化算法研究[J]. 现代制造工程, 2023, 512(5): 22-30.
[13]	秦红斌;常永顺;唐红涛;张峰;王玲军. 混合麻雀算法求解带准备时间的分布式柔性作业车间调度问题[J]. 现代制造工程, 2023, 518(11): 1-11.
[14]	唐艺军;李雪. 基于改进混合遗传算法的柔性车间调度问题研究[J]. 现代制造工程, 2023, 517(10): 8-14.
[15]	陶鑫钰;王艳;纪志成. 基于A3C的特征重构工艺路线规划方法[J]. 现代制造工程, 2023, 517(10): 15-26.