Cooperative Highway Work Zone Merge Control Based on Reinforcement Learning in a Connected and Automated Environment
Abstract
Given the aging infrastructure and the anticipated growing number of highway work zones in the U.S.A., it is important to investigate work zone merge control, which is critical for improving work zone safety and capacity. This paper proposes and evaluates a novel highway work zone merge control strategy based on cooperative driving behavior enabled by artificial intelligence. The proposed method assumes that all vehicles are fully automated, connected, and cooperative. It inserts two metering zones in the open lane to make space for merging vehicles in the closed lane. In addition, each vehicle in the closed lane learns how to adjust its longitudinal position optimally to find a safe gap in the open lane using an off-policy soft actor critic reinforcement learning (RL) algorithm, considering its surrounding traffic conditions. The learning results are captured in convolutional neural networks and used to control individual vehicles in the testing phase. By adding the metering zones and taking the locations, speeds, and accelerations of surrounding vehicles into account, cooperation among vehicles is implicitly considered. This RL-based model is trained and evaluated using a microscopic traffic simulator. The results show that this cooperative RL-based merge control significantly outperforms popular strategies such as late merge and early merge in terms of both mobility and safety measures. It also performs better than a strategy assuming all vehicles are equipped with cooperative adaptive cruise control.
Get full access to this article
View all access and purchase options for this article.
References
1. Tarko A. P., Kanipakapatnam S. R., Wasson J. S. Modeling and Optimization of the Indiana Lane Merge Control System on Approaches to Freeway Work Zones, Part I. Publication FHWA/IN/JTRP-97/12-1. Joint Transportation Research Program, Indiana Department of Transportation and Purdue University, West Lafayette, IN, 1998, p. 345.
2. Beacher A. G., Fontaine M. D., Garber N. J. Evaluation of the Late Merge Work Zone Traffic Control Strategy. Publication FHWA/VTRC 05-R6. Virginia Transportation Research Council, Virginia Department of Transportation, Federal Highway Administration, Washington, D.C., 2004.
3. Marsden G., McDonald M., Brackstone M. Towards an Understanding of Adaptive Cruise Control. Transportation Research Part C: Emerging Technologies, Vol. 91, 2001, pp. 33–51.
4. Chen X., Jin M., Chan C., Mao Y., Gong W. Bionic Decision-Making Analysis during Urban Expressway Ramp Merge for Autonomous Vehicle. Presented at 96th Annual Meeting of the Transportation Research Board, Washington, D.C., 2017.
5. Urmson C., Anhalt J., Bagnell D., Baker C., Bittner R., Clark M., Dolan J., Duggins D., Galatali T., Geyer C., Grittleman M. Autonomous Driving in Urban Environments: Boss and the Urban Challenge. Journal of Field Robotics, Vol. 25, No. 8, 2008, pp. 425–466.
6. Van Arem B., Van Driel C. J., Visser R. The Impact of Cooperative Adaptive Cruise Control on Traffic-Flow Characteristics. IEEE Transactions on Intelligent Transportation Systems, Vol. 7, No. 4, 2006, pp. 429–436.
7. Bojarski M., Del Tesla D., Zhang D., Dworakowski B., Firner B., Flepp P., Goyal L. D., Monfort U., Muller J., Zhang X. End to End Learning for Self-Driving Cars. arXiv Preprint arXiv:1604.07316, 2016.
8. Endsley M. R. Autonomous Driving Systems: A Preliminary Naturalistic Study of the Tesla Model S. Journal of Cognitive Engineering and Decision Making, Vol. 11, No. 3, 2017, pp. 225–238.
9. Desjardins C., Chaib-Draa B. Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach. IEEE Transactions on Intelligent Transportation Systems, Vol. 12, No. 4, 2011, pp. 1248–1260.
10. Chu T., Kalabić U. Model-Based Deep Reinforcement Learning for CACC in Mixed-Autonomy Vehicle Platoon. Proc., 58th IEEE Conference on Decision and Control, Nice, France. IEEE, New York, 2019.
11. Wei S., Zou Y., Zhang T., Zhang X., Wang W. Design and Experimental Validation of a Cooperative Adaptive Cruise Control System Based on Supervised Reinforcement Learning. Applied Sciences, Vol. 8, No. 7, 2018, p. 1014.
12. Wang P., Chan C. Y. Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge. Proc., 20th International Conference on Intelligent Transportation Systems, Yokohama, Japan, IEEE, New York, 2017.
13. Bouton M., Nakhaei A., Fujimura K., Kochenderfer M. J. Cooperation-Aware Reinforcement Learning for Merging in Dense Traffic. 2019 IEEE Intelligent Transportation Systems Conference, Auckland, New Zealand, IEEE, New York, 2019, pp. 3441–3447.
14. Fares A., Gomaa W. Freeway Ramp-Metering Control Based on Reinforcement Learning. Proc., 11th IEEE International Conference on Control & Automation, Taiwan, 2014.
15. Yang H., Rakha H. Reinforcement Learning Ramp Metering Control for Weaving Sections in a Connected Vehicle Environment. Presented at 96th Annual Meeting of the Transportation Research Board, Washington, D.C., 2017.
16. Howard R. A. Dynamic Programming and Markov Processes. Technology Press of the Massachusetts Institute of Technology, Cambridge, MA, 1960.
17. Watkins C. J., Dayan P. Q-Learning. Machine Learning, Vol. 8, No. 3–4, 1992, pp. 279–292.
18. Mnih V., Kavukcuoglu K., Silver D., Rusu A. A., Veness J., Bellemare M. G., Graves A., Riedmiller M., Fidjeland A. K., Ostrovski G., Petersen S. Human-Level Control Through Deep Reinforcement Learning. Nature, Vol. 518, No. 7540, 2015, pp. 529–533.
19. Yu A., Palefsky-Smith R., Bedi R. Deep Reinforcement Learning for Simulated Autonomous Vehicle Control. Course Project Reports: Winter, Stanford University, Stanford, CA, 2016, pp. 1–7.
20. Sallab A., Abdou M., Perot E., Yogamani S. End-to-End Deep Reinforcement Learning for Lane Keeping Assistance. Proc., 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016.
21. Lillicrap P. T., Hunt J. J., Pritzel A., Heess N., Erez T., Tassa Y., Silver D., Wierstra D. Continuous Control with Deep Reinforcement Learning. arXiv Preprint arXiv:1509.02971, 2015.
22. Haarnoja T., Zhou A., Abbeel P., Levine S. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv Preprint arXiv:1801.01290, 2018.
23. Browne C. B., Powley E., Whitehouse D., Lucas S. M., Cowling P. I., Rohlfshagen P., Tavener S., Perez D., Samothrakis S., Colton S. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligence and AI in Games, Vol. 4, No. 1, 2012, pp. 1–43.
24. Silver D., Schrittwieser J., Simonyan K., Antonoglou I., Huang A., Guez A., Hubert T., Baker L., Lai M., Bolton A., Chen Y. Mastering the Game of Go Without Human Knowledge. Nature, Vol. 550, No. 7676, 2017, pp. 354–359.
25. Krizhevsky A., Sutskever I., Hinton G. E. Imagenet Classification with Deep Convolutional Neural Networks. Communications of the ACM, Vol. 60, 2012, pp. 84–90.
26. Szegedy C., Ioffe S., Vanhoucke V., Alemi A. A. Inception-V4, Inception-Resnet and the Impact of Residual Connections on Learning. Proc., 31st AAAI Conference on Artificial Intelligence, San Francisco, CA, 2017.
27. Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O. Proximal Policy Optimization Algorithms. arXiv Preprint arXiv:1707.06347, 2017.
28. Mnih V., Badia A. P., Mirza M., Graves A., Lillicrap T., Harley T., Silver D., Kavukcuoglu K. Asynchronous Methods for Deep Reinforcement Learning. Proc., 33rd International Conference on Machine Learning, New York City, NY, 2016, pp. 1928–1937.
29. Schulman J., Moritz P., Levine S., Jordan M., Abbeel P. High-Dimensional Continuous Control using Generalized Advantage Estimation. arXiv Preprint arXiv:1506.02438, 2015.
30. Schulman J., Levine S., Abbeel P., Jordan M., Moritz P. Trust Region Policy Optimization. Proc., 37th International Conference on Machine Learning, Lille, France, 2015.
31. Hershey J. R., Peder A. O. Approximating the Kullback Leibler Divergence between Gaussian Mixture Models. Proc., 2007 IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, Hawaii. IEEE, New York, Vol. 4, 2007.
32. Yang N., Chang G. L., Kang K. P. Simulation-Based Study on a Lane-Based Signal System for Merge Control at Freeway Work Zones. Journal of Transportation Engineering, Vol. 135, No. 1, 2009, pp. 9–17.
33. Dudek C. L. Notes on Work Zone Capacity and Level of Service. Texas Transportation Institute, Texas A&M University, College Station, TX, 1984.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: July 17, 2020
Issue published: October 2020
Authors
Author Contributions
The authors confirm contribution to the paper as follows: study conception and design: Tianzhu Ren, Yuanchang Xie; data collection: Tianzhu Ren; analysis and interpretation of results: Tianzhu Ren, Yuanchang Xie, Liming Jiang; draft manuscript preparation: Tianzhu Ren, Yuanchang Xie, Liming Jiang. All authors reviewed the results and approved the final version of the manuscript.
Metrics and citations
Metrics
Journals metrics
This article was published in Transportation Research Record: Journal of the Transportation Research Board.
View All Journal MetricsArticle usage*
Total views and downloads: 523
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 0
Crossref: 25
- A review of car-following and lane-changing models under heterogeneous environments
- Artificial intelligence for geoscience: Progress, challenges, and perspectives
- Evaluating the performance of traffic conflict measures in real-time crash risk prediction using pre-crash vehicle trajectories
- Gap setting control strategy for connected and automated vehicles in freeway lane‐drop bottlenecks
- PID‐Based Freeway Work Zone Merge Control with Traffic State Prediction under Mixed Traffic Flow of Connected Automated Vehicles and Manual Vehicles
- Graph-based multi agent reinforcement learning for on-ramp merging in mixed traffic
- Distance headway-based safety evaluation of emerging mixed traffic flow under snowy weather
- Investigating the Effects of Human–Machine Interface on Cooperative Driving Using a Multi-Driver Co-Simulation Platform
- A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario
- Assessing the Impacts of Merge and Speed Control Strategies on Highway Work Zone Safety and Operations Using Artificial Intelligence and Advanced Sensors
- View More
Figures and tables
Figures & Media
Tables
View Options
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
loading institutional access options
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
