CN114550456A - Urban traffic jam scheduling method based on reinforcement learning - Google Patents
Urban traffic jam scheduling method based on reinforcement learning Download PDFInfo
- Publication number
- CN114550456A CN114550456A CN202210188427.3A CN202210188427A CN114550456A CN 114550456 A CN114550456 A CN 114550456A CN 202210188427 A CN202210188427 A CN 202210188427A CN 114550456 A CN114550456 A CN 114550456A
- Authority
- CN
- China
- Prior art keywords
- intersection
- traffic
- reinforcement learning
- vehicle
- traffic light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/048—Detecting movement of traffic to be counted or controlled with provision for compensation of environmental or other condition, e.g. snow, vehicle stopped at detector
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/07—Controlling traffic signals
- G08G1/08—Controlling traffic signals according to detected number or speed of vehicles
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/095—Traffic lights
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2119/00—Details relating to the type or aim of the analysis or the optimisation
- G06F2119/02—Reliability analysis or reliability optimisation; Failure analysis, e.g. worst case scenario performance, failure mode and effects analysis [FMEA]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Chemical & Material Sciences (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Geometry (AREA)
- General Engineering & Computer Science (AREA)
- Analytical Chemistry (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention discloses an urban traffic jam scheduling method based on reinforcement learning, which comprises the steps of acquiring vehicle quantity information, vehicle queuing information and real-time data of traffic light states of urban road intersections through an image sensor and an inductance sensor; then, according to the vehicle quantity information, the vehicle queuing information and the real-time data of the traffic light state, and by combining the intersection priori knowledge of the road section limit and the lane information acquired from the image information and the storage structured data, forming intersection road condition state data as scheduling model training data by using a machine learning algorithm; the dispatching model calculates reward signals according to the traffic effect and the reward function of each lane of the intersection fed back by the environment, and therefore the dispatching model is trained; training a scheduling model based on the intersection road condition state data and the intersection traffic safety criterion by using a reinforcement learning algorithm; and taking the road condition state data of the intersection as input, and outputting a traffic light state instruction and a corresponding traffic light control signal through the trained scheduling model.
Description
Technical Field
The invention relates to the field of intelligent traffic, in particular to an urban traffic jam scheduling method based on reinforcement learning.
Background
With the continuous improvement of the economic level of people and the promotion of the urbanization process, automobiles as the most main transportation means walk into thousands of households, and the problem of urban traffic jam is more serious. Traffic congestion on the one hand reduces social productivity, causes a great amount of economic loss, consumes fuel resources, and causes serious carbon dioxide emission problems. Therefore, the urban traffic efficiency is improved, and the traffic scheduling method is optimized to occupy an important position in the field of modern traffic, wherein traffic light intersection traffic is the most common traffic efficiency bottleneck of urban road sections.
The existing traffic light control methods are mainly divided into two categories, one category is a traditional signal light control algorithm based on rules, such as fixed time, traffic flow, lane occupancy ratio and other algorithms, the method is relatively comprehensive in cognition of scenes, vehicle flow scheduling is difficult to deal with in complex scenes, and vehicle passing efficiency is low. The other type is a self-adaptive control algorithm based on machine learning, such as a traffic light scheduling algorithm based on reinforcement learning, the reinforcement learning has achieved good performance in the fields of game playing, optimized scheduling and the like, and due to the characteristic that the reinforcement learning can learn by itself and improve decision-making capability, attention is drawn in the field of traffic light control in recent years.
Disclosure of Invention
Aiming at the defects of the prior art, the technical problems to be solved by the invention are as follows: how to provide an urban traffic jam scheduling method based on reinforcement learning, which improves the urban vehicle passing efficiency and relieves the traffic jam condition.
In order to solve the technical problems, the invention adopts the following technical scheme:
a city traffic jam scheduling method based on reinforcement learning comprises the following steps:
(1) acquiring vehicle quantity information, vehicle queuing information and real-time data of traffic light states at urban road intersections by using an image sensor and an inductance sensor;
(2) by utilizing a machine learning algorithm, according to the quantity information of vehicles, the queuing information of the vehicles and the real-time data of the states of traffic lights, and by combining with intersection priori knowledge of road section limitation and lane information acquired from image information and storage structured data, intersection road condition state data are jointly formed to serve as scheduling model training data;
(3) adopting a reinforcement learning algorithm, selecting a traffic light state switching action in an action space for switching the traffic light states by a scheduling model according to intersection road condition state data and intersection traffic safety criteria at a given moment, calculating reward signals according to the traffic effect and reward functions of each lane of the intersection fed back by the environment, and maximizing the reward signals by the action selected by the model after multiple iterations so as to train the scheduling model;
(4) and taking the road condition state data of the intersection as input, and outputting a traffic light state instruction and a corresponding traffic light control signal through the trained scheduling model.
As optimization, in step (1), the running speed of the vehicle approaching the intersection is also acquired through the laser radar, and the environmental state information of the intersection is also acquired through the temperature sensor and the humidity sensor.
As optimization, in the step (2), data preprocessing work of data cleaning and feature construction is firstly carried out on the real-time data of the vehicle quantity information, the vehicle queuing information and the traffic light state, and then structured real-time road condition features input as a scheduling model are extracted by utilizing any one machine learning algorithm of CNN, MLP, GBDT and SVM.
As an optimization, in step (2), the intersection prior knowledge includes a road section speed limit, a steering limit, the number of lanes, a lane category and a traffic light switching time length.
As optimization, in the step (3), the reinforcement learning algorithm comprises a Q-learning or time difference algorithm, the input features of the reinforcement learning algorithm and the variables of the reward function are obtained from the intersection road condition state data in the step (2), the input features of the reinforcement learning algorithm comprise the vehicle average speed, the vehicle number, the vehicle position, the lane number, the lane category, the weather state, the accident state and the traffic efficiency of each lane of the intersection, wherein the traffic efficiency is calculated by a formula I, and the variables of the reward function comprise the traffic number, the vehicle waiting time, the vehicle average speed difference before and after traffic and whether traffic lights are switched;
wherein efficiency is the overall traffic efficiency of the vehicle, vcar_avgAverage speed of vehicles at the intersection, vlane_speed_limitIs the intersection upper limit speed.
As an optimization, in the step (3), the intersection passage safety criterion is a basic constraint on the safe passage of the intersection so as to ensure that the traffic flow of each lane cannot collide.
And (4) inputting the intersection road condition state data and the intersection prior knowledge into the scheduling model to obtain the traffic light target state, if the current traffic light state is consistent with the target state, not performing traffic light switching action, and otherwise switching the traffic light to the target state.
In conclusion, the beneficial effects of the invention are as follows: according to the invention, through the road condition information of the current intersection and the combination of a reinforcement learning algorithm, the problems of incomplete strategy input and inflexible control strategy of the traditional scheduling algorithm are solved, a solution is provided for the scheduling of the urban complex traffic network, the traffic jam condition is effectively relieved, and the traffic efficiency of urban vehicles is improved.
Drawings
For purposes of promoting a better understanding of the objects, aspects and advantages of the invention, reference will now be made in detail to the present invention as illustrated in the accompanying drawings, in which:
FIG. 1 is an overall flow chart of the vehicle dispatching control at an intersection according to the present invention;
FIG. 2 is a flow chart of reinforcement learning model information in the present invention;
fig. 3 is a space diagram of the effective state of the traffic light of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1 and fig. 2, the city traffic jam scheduling method based on reinforcement learning in the present embodiment includes the following steps:
(1) acquiring vehicle quantity information, vehicle queuing information and real-time data of traffic light states at urban road intersections by using an image sensor and an inductance sensor;
(2) by utilizing a machine learning algorithm, according to the quantity information of vehicles, the queuing information of the vehicles and the real-time data of the states of traffic lights, and by combining with intersection priori knowledge of road section limitation and lane information acquired from image information and storage structured data, intersection road condition state data are jointly formed to serve as scheduling model training data;
(3) adopting a reinforcement learning algorithm, selecting a traffic light state switching action in an action space for switching the traffic light states by a scheduling model according to intersection road condition state data and intersection traffic safety criteria at a given moment, calculating reward signals according to the traffic effect and reward functions of each lane of the intersection fed back by the environment, and maximizing the reward signals by the action selected by the model after multiple iterations so as to train the scheduling model;
(4) and taking the road condition state data of the intersection as input, and outputting a traffic light state instruction and a corresponding traffic light control signal through the trained scheduling model.
In the specific embodiment, in step (1), the running speed of the vehicle approaching the intersection is also acquired by the laser radar, and the environmental state information of the intersection is also acquired by the temperature sensor and the humidity sensor.
In the specific embodiment, in the step (2), data preprocessing work of data cleaning and feature construction is performed on the real-time data of the vehicle quantity information, the vehicle queuing information and the traffic light state, and then the structured real-time road condition features input as the scheduling model are extracted by using any one of the machine learning algorithms of CNN, MLP, GBDT and SVM.
In this specific embodiment, in step (2), the intersection priori knowledge includes a road speed limit, a steering limit, a number of lanes, a lane category, and a traffic light switching duration.
In the specific embodiment, in the step (3), the reinforcement learning algorithm comprises a Q-learning or time difference algorithm, the input features of the reinforcement learning algorithm and the variables of the reward function are obtained from the intersection road condition state data in the step (2), the input features of the reinforcement learning algorithm comprise the vehicle average speed, the vehicle number, the vehicle position, the lane number, the lane category, the weather state, the accident state and the traffic efficiency of each lane of the intersection, wherein the traffic efficiency is calculated by a formula (i), and the variables of the reward function comprise the traffic number, the vehicle waiting time, the vehicle average speed difference before and after traffic and whether traffic lights are switched;
wherein efficiency is the overall traffic efficiency of the vehicle, vcar_avgAverage speed of vehicles at the intersection, vlane_speed_limitIs the upper limit speed of the intersection.
In this embodiment, in step (3), the intersection passage safety criterion is a basic constraint on the intersection safety passage, so as to ensure that the traffic flow of each lane does not collide. In the dispatching model, the safety criterion can be combined with the traffic light state space, for example, for a standard intersection, the effective state space of the traffic light can be considered to have 8 states, as shown in fig. 3, so that the most excited state of the 8 states can be selected according to the model as the target state of the traffic light according to the input of the road condition state
In the specific embodiment, in the step (4), the intersection road condition state data and the intersection prior knowledge are input into the scheduling model to obtain the traffic light target state, if the current traffic light state is consistent with the target state, the traffic light switching action is not performed, otherwise, the traffic light is switched to the target state.
Finally, it is noted that the above-mentioned embodiments illustrate rather than limit the invention, and that, while the invention has been described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (7)
1. A city traffic jam scheduling method based on reinforcement learning is characterized in that: the method comprises the following steps:
(1) acquiring vehicle quantity information, vehicle queuing information and real-time data of traffic light states at urban road intersections by using an image sensor and an inductance sensor;
(2) by utilizing a machine learning algorithm, according to the quantity information of vehicles, the queuing information of the vehicles and the real-time data of the states of traffic lights, and by combining with intersection priori knowledge of road section limitation and lane information acquired from image information and storage structured data, intersection road condition state data are jointly formed to serve as scheduling model training data;
(3) adopting a reinforcement learning algorithm, selecting a traffic light state switching action in an action space for switching the traffic light states by a scheduling model according to intersection road condition state data and intersection traffic safety criteria at a given moment, calculating reward signals according to the traffic effect and reward functions of each lane of the intersection fed back by the environment, and maximizing the reward signals by the action selected by the model after multiple iterations so as to train the scheduling model;
(4) and taking the road condition state data of the intersection as input, and outputting a traffic light state instruction and a corresponding traffic light control signal through the trained scheduling model.
2. The reinforcement learning-based urban traffic jam scheduling method according to claim 1, wherein: in the step (1), the running speed of a vehicle approaching the intersection is also acquired through a laser radar, and the environmental state information of the intersection is also acquired through a temperature sensor and a humidity sensor.
3. The reinforcement learning-based urban traffic jam scheduling method according to claim 1, wherein: in the step (2), data preprocessing work of data cleaning and feature construction is firstly carried out on the real-time data of the vehicle quantity information, the vehicle queuing information and the traffic light state, and then structured real-time road condition features input as a scheduling model are extracted by using any one machine learning algorithm of CNN, MLP, GBDT and SVM.
4. The reinforcement learning-based urban traffic congestion scheduling method according to claim 1, wherein: in step (2), the intersection priori knowledge comprises road section speed limit, steering limit, lane number, lane category and traffic light switching time length.
5. The reinforcement learning-based urban traffic congestion scheduling method according to claim 1, wherein: in the step (3), the reinforcement learning algorithm comprises a Q-learning or time difference algorithm, the input characteristics of the reinforcement learning algorithm and the variables of the reward function are obtained from the intersection road condition state data in the step (2), the input characteristics of the reinforcement learning algorithm comprise the vehicle average speed, the vehicle number, the vehicle position, the lane number, the lane category, the weather state, the accident state and the traffic efficiency of each lane of the intersection, wherein the traffic efficiency is calculated through a formula I, and the variables of the reward function comprise the traffic number, the vehicle waiting time, the vehicle average speed difference before and after traffic and whether traffic lights are switched;
wherein efficiency is the overall traffic efficiency of the vehicle, vcar_avgAverage speed of vehicles at the intersection, vlane_speed_limitIs the upper limit speed of the intersection.
6. The reinforcement learning-based urban traffic congestion scheduling method according to claim 1, wherein: in the step (3), the intersection passage safety criterion is a basic constraint on the intersection safety passage so as to ensure that the traffic flow of each lane does not collide.
7. The reinforcement learning-based urban traffic congestion scheduling method according to claim 1, wherein: and (4) inputting the intersection road condition state data and the intersection priori knowledge into a scheduling model to obtain a traffic light target state, if the current traffic light state is consistent with the target state, not performing traffic light switching action, and otherwise switching the traffic light to the target state.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210188427.3A CN114550456B (en) | 2022-02-28 | 2022-02-28 | Urban traffic jam scheduling method based on reinforcement learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210188427.3A CN114550456B (en) | 2022-02-28 | 2022-02-28 | Urban traffic jam scheduling method based on reinforcement learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114550456A true CN114550456A (en) | 2022-05-27 |
| CN114550456B CN114550456B (en) | 2023-07-04 |
Family
ID=81678879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210188427.3A Active CN114550456B (en) | 2022-02-28 | 2022-02-28 | Urban traffic jam scheduling method based on reinforcement learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114550456B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109493617A (en) * | 2018-10-29 | 2019-03-19 | 沈阳天久信息技术工程有限公司 | A kind of traffic signal optimization control method and device |
| CN109544913A (en) * | 2018-11-07 | 2019-03-29 | 南京邮电大学 | A kind of traffic lights dynamic timing algorithm based on depth Q e-learning |
| CN110114806A (en) * | 2018-02-28 | 2019-08-09 | 华为技术有限公司 | Signalized control method, relevant device and system |
| CN110164150A (en) * | 2019-06-10 | 2019-08-23 | 浙江大学 | A kind of method for controlling traffic signal lights based on time distribution and intensified learning |
| US20190347933A1 (en) * | 2018-05-11 | 2019-11-14 | Virtual Traffic Lights, LLC | Method of implementing an intelligent traffic control apparatus having a reinforcement learning based partial traffic detection control system, and an intelligent traffic control apparatus implemented thereby |
| CN111047884A (en) * | 2019-12-30 | 2020-04-21 | 西安理工大学 | Traffic light control method based on fog calculation and reinforcement learning |
| CN111915894A (en) * | 2020-08-06 | 2020-11-10 | 北京航空航天大学 | Variable lane and traffic signal cooperative control method based on deep reinforcement learning |
| CN113287156A (en) * | 2019-10-28 | 2021-08-20 | 乐人株式会社 | Signal control device and signal control method based on reinforcement learning |
| CN113643528A (en) * | 2021-07-01 | 2021-11-12 | 腾讯科技（深圳）有限公司 | Signal lamp control method, model training method, system, device and storage medium |
-
2022
- 2022-02-28 CN CN202210188427.3A patent/CN114550456B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110114806A (en) * | 2018-02-28 | 2019-08-09 | 华为技术有限公司 | Signalized control method, relevant device and system |
| US20190347933A1 (en) * | 2018-05-11 | 2019-11-14 | Virtual Traffic Lights, LLC | Method of implementing an intelligent traffic control apparatus having a reinforcement learning based partial traffic detection control system, and an intelligent traffic control apparatus implemented thereby |
| CN109493617A (en) * | 2018-10-29 | 2019-03-19 | 沈阳天久信息技术工程有限公司 | A kind of traffic signal optimization control method and device |
| CN109544913A (en) * | 2018-11-07 | 2019-03-29 | 南京邮电大学 | A kind of traffic lights dynamic timing algorithm based on depth Q e-learning |
| CN110164150A (en) * | 2019-06-10 | 2019-08-23 | 浙江大学 | A kind of method for controlling traffic signal lights based on time distribution and intensified learning |
| CN113287156A (en) * | 2019-10-28 | 2021-08-20 | 乐人株式会社 | Signal control device and signal control method based on reinforcement learning |
| CN111047884A (en) * | 2019-12-30 | 2020-04-21 | 西安理工大学 | Traffic light control method based on fog calculation and reinforcement learning |
| CN111915894A (en) * | 2020-08-06 | 2020-11-10 | 北京航空航天大学 | Variable lane and traffic signal cooperative control method based on deep reinforcement learning |
| CN113643528A (en) * | 2021-07-01 | 2021-11-12 | 腾讯科技（深圳）有限公司 | Signal lamp control method, model training method, system, device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 武强: "多智能体强化学习在城市交通信号控制中的研究与应用", 《中国博士学位论文全文数据库 工程科技II辑》, vol. 2021, no. 4, pages 034 - 14 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114550456B (en) | 2023-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021227502A1 (en) | Method for traffic light and vehicle track control at signalized intersection | |
| CN111445692B (en) | Speed collaborative optimization method for intelligent networked automobile at signal-lamp-free intersection | |
| CN109410561B (en) | Uniform and heterogeneous formation driving control method for vehicles on highway | |
| CN114973733B (en) | Network-connected automatic vehicle track optimization control method under mixed flow at signal intersection | |
| CN107507430A (en) | A kind of urban road crossing traffic control method and system | |
| CN113312752B (en) | Traffic simulation method and device for main road priority control intersection | |
| CN110444015B (en) | Intelligent network-connected automobile speed decision method based on no-signal intersection partition | |
| CN113570875B (en) | Green wave vehicle speed calculation method, device, equipment and storage medium | |
| CN108171979A (en) | A kind of tramcar whole day runs time optimization method and system | |
| CN206194129U (en) | Device based on induced tactful guide car of vehicle velocity does not stop and to pass through crossing | |
| CN116740945B (en) | Method and system for multi-vehicle collaborative grouping intersection of expressway confluence region in mixed running environment | |
| WO2023035666A1 (en) | Urban road network traffic light control method based on expected reward estimation | |
| CN116895163A (en) | Multi-vehicle cooperative control method and system based on signal lamp active communication | |
| CN114495547A (en) | Signal intersection cooperative traffic strategy for automatically driven automobile | |
| CN113628443A (en) | Signal intersection speed guiding method based on driver finiteness | |
| CN115565390A (en) | Intelligent internet automobile multi-lane queue traffic control method and system and computer readable storage medium | |
| CN112614357B (en) | Intelligent vehicle intersection left-turn phase signal optimization method and device | |
| CN105741585A (en) | Fuel-economizing-oriented vehicle track smoothing control method based on Internet-of-vehicles | |
| CN114550456A (en) | Urban traffic jam scheduling method based on reinforcement learning | |
| CN116935673A (en) | Signal intersection vehicle passing method considering pedestrian crossing under network environment | |
| CN115083174B (en) | Traffic signal lamp control method based on cooperative multi-agent reinforcement learning | |
| CN116524745A (en) | Cloud edge cooperative area traffic signal dynamic timing system and method | |
| CN116189454A (en) | Traffic signal control method, device, electronic equipment and storage medium | |
| Zou et al. | Vehicle trajectory control and signal timing optimization of isolated intersection under V2X environment | |
| CN117496689B (en) | Continuous signal intersection fleet cooperative control method and system in mixed traffic environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |