Urban Traffic Management Using Deep Reinforcement Learning in Connected Vehicle Environments

Alleviating traffic congestion is a primary concern of urban traffic management, as citizens often complain about the frustrating time wasted on the road. Traffic congestion causes enormous economic losses. Taking the European Union as an example, the annual economic losses from congestion exceed 1% of GDP. Urban intersections are often traffic bottlenecks, thus, improving intersection throughput can reduce travel time. However, increasing traffic efficiency is not the only target. The next generation of urban traffic management should align with the United Nations Sustainable Development Goals (UN-SDGs), addressing climate change and safety issues, i.e., reducing greenhouse gas emissions and collision risks. Additionally, the ease and cost of infrastructure deployment and maintenance are critical considerations in practice. At most urban intersections, traffic lights coordinate conflicting traffic from different directions and influence intersection throughput. Many cities have adopted adaptive systems (such as SCATS and SCOOT) that dynamically adjust traffic lights based on real-time traffic volumes estimated by induction loops. However, these traditional methods rely on deterministic, rule-based adjustments, which often struggle to improve fast-changing urban traffic. In contrast, Deep Reinforcement Learning (DRL) has shown great potential in handling diverse traffic scenarios. With trial-and-error learning, DRL-based systems improve control strategies for long-term traffic performance goals. These systems can process extensive data from sensors and connected vehicles while quickly responding to complex urban traffic conditions. Advanced Vehicle-To-Everything (V2X) communication allows vehicles to exchange information with each other and with road infrastructure, providing DRL-based systems with more accurate real-time traffic data. Although DRL-based traffic control systems have advantages in improving urban traffic and reducing congestion, they have not yet been applied in real-world applications due to challenges in system efficiency, scalability, practicality, robustness, and interpretability. This thesis aims to address these challenges and advance the potential implementation and deployment of DRL-based systems for sustainable urban traffic management. This thesis first introduces an innovative joint DRL method for controlling both traffic lights and CAV speeds simultaneously. The joint system can achieve sustainable traffic more comprehensively by reducing travel time, fuel consumption and emissions, and enhancing traffic safety. Secondly, to address scalability issues in large urban environments and consider practical transitions to mixed-autonomy traffic, the joint system reduces excessive Connected Autonomous Vehicle (CAV) agents while ensuring that V2X communication demands are met without overloading existing infrastructure. Thirdly, more real-world urban scenarios are considered, including imperfect communication, i.e., delays and data loss, and road emergencies. Ontology-based models of traffic-related knowledge can enhance DRL traffic light systems by augmenting state input under imperfect communication and providing targeted control for situations such as prioritizing emergency vehicles or managing congestion from unplanned road closures. To specifically address non-recurrent congestion, a novel UAV-assisted framework is proposed, offering a rapidly deployed and cost-effective solution. Unmanned Aerial Vehicle (UAV) embedded with DRL model can temporarily take control of traffic lights at critical intersections, efficiently solving unexpected congestion that existing systems cannot address promptly. Lastly, eXplainable Artificial Intelligence (XAI) techniques are used to enhance transparency in DRL decision-making, help traffic light model development to minimize communication overhead and build trust among various stakeholders.