基于深度强化学习的晶圆分拣系统实时调度研究

doi:10.16731/j.cnki.1671-3133.2025.10.003

摘要/Abstract

摘要： 调度问题常见于各类系统中,然而其求解过程依赖于先验知识与人为设定的规则。为降低调度问题求解成本并保证调度效率,针对具有共享资源的晶圆分拣系统实时系统调度问题,定义了并行控制 Petri 网(PcPN)进行系统建模,并基于PcPN开展了强化学习(DRL)训练,成功获得了可根据PcPN状态实时生成有效调度动作的调度智能体。此外,为提升DRL训练的稳定性,构建了PcPN深度策略网络并验证了其有效性。在实机实验阶段,PcPN实时调度算法被部署至晶圆分拣系统生产实例中,实现了PcPN驱动的晶圆分拣实时调度,有力验证了该算法的可行性与高效性。

关键词: 深度强化学习, Petri网, 调度, 实时系统

Abstract: Scheduling problems are widespread in various systems. However,the solution of scheduling problems is costly. Taking the scheduling problem with shared resources of wafer sorting system as an object,the Parallel control Petri Net (PcPN) for system modeling was defined in order to reduce the cost of scheduling problem solving and ensure scheduling efficiency.Based on PcPN,Deep Reinforcement Learning (DRL) training was carried out to successfully construct a scheduling agent that can generate effective scheduling actions according to the state of PcPN in real-time,and the PcPN real-time scheduling algorithm was proposed accordingly. To further enhance the stability of DRL training, a deep policy network for PcPN was proposed and its validity was verified. In actual machine experiment stage,the scheduling algorithm was deployed in the wafer sorting system,realizing the real-time scheduling of wafer sorting driven by PcPN. The experimental result proves the feasibility and efficiency of the algorithm.

Key words: Deep Reinforcement Learning(DRL), Petri Net(PN), scheduling, real-time system

中图分类号:

TP273

刘嘉成. 基于深度强化学习的晶圆分拣系统实时调度研究[J]. 现代制造工程, 2025, 541(10): 26-34.

LIU Jiacheng. Deep reinforcement learning based wafer sorting system real-time scheduling[J]. Modern Manufacturing Engineering, 2025, 541(10): 26-34.

参考文献

[1] DESTOUET C,TLAHIG H,BETTAYEB B,et al.Flexible job shop scheduling problem under Industry 5.0:A survey on human reintegration,environmental consideration and resilience improvement[J].Journal of Manufacturing Systems,2023,1(67):155-173.
[2] KAYHAN B M,YILDIZ G.Reinforcement learning applications to machine scheduling problems:a comprehensive literature review[J].Journal of Intelligent Manufacturing,2023,34(3):905-929.
[3] WANG Ling,PAN Zixiao,WANG Jingjing.A Review of Reinforcement Learning Based Intelligent Optimization for Manufacturing Scheduling[J]. Complex System Modeling and Simulation,2021,1(4):257-270.
[4] HAN Wei,GUO Fang,SU Xichao.A reinforcement learning method for a hybrid flow-shop scheduling problem[J].Algorithms,2019,12(11):222.
[5] CAI Jingcao,LEI Deming,WANG Jing,et al.A novel shuffled frog-leaping algorithm with reinforcement learning for distributed assembly hybrid flow shop scheduling[J].International Journal of Production Research,2023,61(4):1233-1251.
[6] ZHAO Yejian,LUO Xiaochuan,ZHANG Yulin.The application of heterogeneous graph neural network and deep reinforcement learning in hybrid flow shop scheduling problem[J].Computers & Industrial Engineering,2024,187:109802.
[7] ZHOU L,ZHANG L,BERTHOLD K P H. Deep reinforcement learning-based dynamic scheduling in smart manufacturing[J].Procedia Cirp,2020,93:383-388.
[8] LI Chengxi,PAI Zheng,YIN Yue,et al.Deep reinforcement learning in smart manufacturing:A review and prospects[J].CIRP Journal of Manufacturing Science and Tech-nology,2023,1(40):75-101.
[9] LIU RENKE,RAJESH PIPLANI,CARLOS TORO.Deep reinforcement learning for dynamic scheduling of a flexible job shop[J].International Journal of Production Research,2022,60(13):4049-4069.
[10] ZHANG Yi,ZHU Haihua,TANG Dunbing,et al.Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems[J].Robotics and Computer-Integrated Manufacturing,2022,78:102412.
[11] BO Zitong,YING Qiao,CHANG Leng,et al.Developing real-time scheduling policy by deep reinforcement learning[C]//2021 IEEE 27th Real-Time and Embedded Technology and Applications Symposium(RTAS). Nashville:IEEE,2021:131-142.
[12] 李诚,李爽,冯毅萍,等.基于时间Petri网和启发式搜索的柔性制造系统调度算法[J].上海交通大学学报,2015,49(5):6.
[13] LI Cheng,LI Shuang,FENG Yiping,et al.Scheduling flexible manufacturing system based on timed Petri net and heuristic search[J],Journal of Shanghai Jiao Tong University,2015,49(5):6.
[14] ZUBEREK W M.Timed Petri nets definitions,properties,and applications[J].Microelectronics Reliability,1991,31(4):627-644.
[15] LEE JUN-HO,KIM HYUN-JUNG.Reinforcement learning for robotic flow shop scheduling with processing time variations[J].International Journal of Production Research,2022,60(7):2346-2368.
[16] LUO Jiliang,YI Sijia,LIN Zexuan,et al.Petri-net-based deep reinforcement learning for real-time scheduling of automated manufacturing systems[J].Journal of Manufacturing Systems,2024,74:995-1008.
[17] HU Liang,LIU Zhenyu,HU Weifei,et al,.Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network[J]. Journal of Manufacturing Systems,2020,55:1-14.
[18] WU Zonghan,PAN Shirui,CHEN Fengwen,et al.A comprehensive survey on graph neural networks[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(1):4-24.
[19] YU CHAO,AKASH VELU,EUGENE VINITSKY,et al.The surprising effectiveness of ppo in cooperative multi-agent games[J].Advances in Neural Information Processing Systems,2022 35:24611-24624.
[20] ADAM Paszke,SAM Gross,FRANCISCO Massa,et al. Pytorch:An imperative style,high-performance deep learning library[C]//Advances in Neural Information Processing Systems.Vancouver:[s.n.],2019:32.