Unstructured terrain motion control for quadruped robot based on improved TD3

doi:10.16731/j.cnki.1671-3133.2025.01.004

Abstract

Abstract: The motion control of quadruped robots in unstructured terrains heavily relies on complex dynamics models and controller designs. Utilizing Deep Reinforcement Learning (DRL) methods to design controllers for quadruped robots has become a trend. To address issues such as slow convergence, susceptibility to local optima, and high computational resource consumption during the training process of DRL, an improved algorithm called Memory-integrated Twin Delayed Deep Deterministic policy gradient (M-TD3) was proposed. Firstly, models for both the quadruped robot and unstructured terrains were established. Then, the convergence state and learning efficiency of M-TD3 algorithm were analyzed. Finally, motion control simulation comparisons for various terrains were carried out, and production of prototype was tested to verify controller performance. Simulation results show that compared with traditional TD3 algorithm, M-TD3 algorithm has faster convergence, higher efficiency, and significantly improves motion control performance. Prototype test results show that the controller designed based on the improved TD3 algorithm can make quadruped robot effectively move over obstacles in unstructured terrain.

Key words: quadruped robot, unstructured terrains, Deep Reinforcement Learning(DRL), TD3 algorithm

CLC Number:

TP242.6

XIE Zijian, QIN Jianjun, CAO Yu. Unstructured terrain motion control for quadruped robot based on improved TD3[J]. Modern Manufacturing Engineering, 2025, 532(1): 33-41.

References

[1] SEHOON H .Quadrupedal robots trot into the wild[J].Science Robotics,2020,5(47):eabe5218.
[2] 谢惠祥. 四足机器人对角小跑步态虚拟模型直觉控制方法研究[D].长沙:国防科技大学,2015.
[3] CHAI H, LI Y, SONG R, et al. A survey of the development of quadruped robots: joint configuration, dynamic locomotion control method and mobile manipulation approach[J].Biomimetic Intelligence and Robotics,2022,2(1):100029.
[4] CARLO J D, WENSING P M, KATZ B, et al. Dynamic locomotion in the MIT cheetah 3 through convex model-predictive control[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid:IEEE, 2018.DOI:10.1109/IROS.2018.8594448.
[5] ZHAO J, MA S, NIU S, et al. Fractional-order virtual model control for trotting motion of quadruped robot[C]//2020 Chinese Control and Decision Conference (CCDC).Hefei:[s.n.],2020.DOI:10.1109/CCDC49329.2020.9164655.
[6] ZHAO W, QUERALTA J P, WESTERLUND T. Sim-to-real transfer in deep reinforcement learning for robotics: a survey[C]//2020 IEEE Symposium Series on Computational Intelligence (SSCI). Canberra:[s.n.],2020.DOI:10.1109/SSCI47803.2020.9308468.
[7] 多南讯,吕强,林辉灿,等.迈进高维连续空间:深度强化学习在机器人领域中的应用[J].机器人,2019,41(2):276-288.
[8] BELLEGARDA G , NGUYEN Q .Robust quadruped jumping via deep reinforcement learning[EB/OL]. (2020-11-13)[2023-11-06]. http://researchgate.net/pubtication/345971287. DOI:10.48550/arXiv.2011.07089.
[9] JI Q, FU S, TAN K, et al. Synthesizing the optimal gait of a quadruped robot with soft actuators using deep reinforcement learning[J]. Robotics and Computer-Integrated Manufacturing,2022,78:102382.
[10] ZHU K, ZHANG T. Deep reinforcement learning based mobile robot navigation: a review[J]. Tsinghua Science and Technology, 2021,26(5):674-691.
[11] RUDIN N, HOELLER D,REIST P,et al. Learning to walk in minutes using massively parallel deep reinforcement learning[C]//5th Annual Conference on Robot Learning.[S.l.]:[s.n.],2022:91-100.
[12] LI Y, HAO X, SHE Y, et al. Constrained motion planning of free-float dual-arm space manipulator via deep reinforcement learning[J]. Aerospace Science and Technology,2021,109:106446.
[13] LEE J, HWANGBO J, WELLHAUSEN L, et al. Learning quadrupedal locomotion over challenging terrain[J]. Science Robotics,2020,5(47):5986.
[14] MOCK J W, MUKNAHALLIPATNA S S. A comparison of ppo, TD3 and sac reinforcement algorithms for quadruped walking gait generation[J]. Journal of Intelligent Learning Systems and Applications, 2023,15(1):36-56.
[15] 陈恺丰,田博睿,李和清,等.基于DDPG算法的双轮腿机器人运动控制研究[J].系统工程与电子技术,2023,45(4):1144-1151.