CN117875674A - A bus dispatching method based on Q-learning - Google Patents

Abstract

The present invention discloses a bus dispatching method based on Q-learning, comprising the following steps: step 1, obtaining historical operation data of a bus system; step 2, obtaining expected passenger flow data within a preset time according to the historical operation data; step 3, using a Q-learning algorithm to construct a dispatching model according to the expected passenger flow data; step 4, applying the dispatching model to the actual operation of the bus system. The bus dispatching method based on Q-learning proposed in the present application predicts the expected passenger flow data within a preset time through the historical operation data of the bus system, and then uses a Q-learning algorithm to construct a dispatching model according to the expected passenger flow data, thereby improving the correlation between the collected data and the dispatching decision, and improving the accuracy of bus dispatching.

Description

A bus dispatching method based on Q-learning

Technical Field

The present invention relates to the technical field of traffic dispatching, and in particular to a public transportation dispatching method based on Q-learning.

Background technique

With the acceleration of my country's urbanization process, public transportation systems play an increasingly important role in solving urban traffic congestion problems. Bus Rapid Transit (BRT), as a large-capacity and high-efficiency public transportation mode, has been widely used around the world. How to effectively dispatch BRT to improve operating efficiency and meet passenger needs has become an urgent problem to be solved.

In the prior art, the patent publication number CN116895144A, "A dynamic dispatching system and method for electric buses based on deep reinforcement learning", adopts the deep reinforcement learning DQN algorithm to design a deep reinforcement learning program for electric bus dispatching. The design of the cost function takes into account service efficiency, service reliability, and operating costs, and trains them so that the intelligent agent can make dispatching decisions based on the trained neural network; considering the problems of short cruising range and long charging time faced by electric buses, which are different from fuel buses, this method also takes into account the passenger's request, the power of the electric bus itself, and the location and status of the charging station, and can provide the best charging plan and dispatching decision.

When conducting bus dispatch, in addition to considering the impact of direct factors such as operating income and operating costs, the impact of indirect factors such as expected passenger flow cannot be ignored. The expected passenger flow can reflect the number of passengers arriving at the station within the expected time, which in turn affects the dispatch decision. However, the above-mentioned prior art does not consider the impact of expected passenger flow, and the correlation between the collected data and the dispatch decision is poor. In addition, in the above-mentioned prior art, the DQN algorithm has the problem of not being able to guarantee convergence. When conducting reinforcement learning research on bus dispatch, it is easy to fall into suboptimal strategies, and the accuracy of bus dispatch is poor.

Summary of the invention

The present invention provides a bus dispatching method based on Q-learning, which is used to solve the problem that there is no relatively reliable bus dispatching collection data and the correlation between the dispatching decision is poor, and the bus dispatching accuracy is poor in the prior art.

On the one hand, the present invention provides a bus scheduling method based on Q-learning, comprising the following steps:

Step 1: Obtain historical operation data of the public transportation system.

Step 2: Obtain expected passenger flow data within a preset time based on the historical operation data.

Step three: Use the Q-learning algorithm to construct a scheduling model based on the expected passenger flow data.

Step 4: Apply the scheduling model to the actual operation of the public transportation system.

In a possible implementation, in step one, the historical operation data includes: departure time data, route data, arrival time data, GPS track data and card swiping data.

In a possible implementation, step 2 includes:

The historical passenger flow data is obtained according to the historical operation data.

The expected passenger flow data is obtained according to the historical passenger flow data using a spatiotemporal graph convolutional network.

In a possible implementation, in step 3, the step of using the Q-learning algorithm to construct a scheduling model according to the expected passenger flow data includes:

Create a Q matrix with rows representing states and columns representing actions.

The Q matrix is trained using the expected passenger flow data to obtain a trained Q matrix, namely the scheduling model.

In a possible implementation, in step 3, the using the expected passenger flow data to train the Q matrix includes:

A, initialize the current state to the starting state.

B1, according to the current state and Q matrix, use the ε-greedy strategy to select the decision action.

B2, execute the decision action to obtain a new state.

B3, observe the new state and immediate reward.

B4, updates the new Q value into the Q matrix.

B5, sets the new state as the current state.

B6, if the preset number of training steps is reached or the end state is reached, proceed to the next step, otherwise return to step B1.

C. If the preset number of training times is reached, the training is completed, otherwise return to step A.

In a possible implementation, in step B3, the instant reward is obtained through a pre-designed reward function.

The reward function includes: an operating income reward function, an operating cost reward function and a passenger time cost reward function.

In a possible implementation, in step B4, the Q-value is updated using a Q-learning update strategy.

In a possible implementation, in step three, after obtaining the scheduling model, the scheduling model is tested using the expected passenger flow data.

In a possible implementation, step 4 includes:

Get real-time operation data of the public transportation system.

The expected passenger flow data corresponding to the real-time operation data is obtained and input into the scheduling model, and a scheduling decision is output.

Actual scheduling is performed according to the scheduling decision.

In a possible implementation manner, after step 4, the method further includes: performing a performance evaluation on the scheduling model.

When the performance evaluation result is lower than the preset expectation, return to step 1 and repeat until the performance evaluation result is higher than or equal to the preset expectation.

A bus dispatching method based on Q-learning in the present invention has the following advantages:

The expected passenger flow data within a preset time is predicted by using the historical operation data of the public transportation system, and then the scheduling model is constructed based on the expected passenger flow data using the Q-learning algorithm, which improves the correlation between the collected data and the scheduling decision, and improves the accuracy of public transportation scheduling; the proposed spatiotemporal graph convolutional network is used to obtain the expected passenger flow data based on the historical passenger flow data, which improves the accuracy of the expected passenger flow data; the proposed reward functions include: operating income reward function, operating cost reward function and passenger time cost reward function, which improves the accuracy of public transportation scheduling by considering the impact of passenger time cost; the proposed scheduling model is subjected to performance evaluation, and when the performance evaluation result is lower than the preset expectation, it returns to step one and loops until the performance evaluation result is higher than or equal to the preset expectation, thereby improving the optimizability of the scheduling model.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a process flow of a bus dispatching method based on Q-learning provided by an embodiment of the present invention;

FIG. 2 is another schematic flow chart of a bus dispatching method based on Q-learning provided in an embodiment of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

As shown in FIG1 , an embodiment of the present invention provides a bus scheduling method based on Q-learning, comprising the following steps:

Step 1: Obtain historical operation data of the public transportation system.

Step 2: Obtain expected passenger flow data within a preset time based on the historical operation data.

Step three: Use the Q-learning algorithm to construct a scheduling model based on the expected passenger flow data.

Step 4: Apply the scheduling model to the actual operation of the public transportation system.

Exemplarily, in step one, the historical operation data includes: departure time data, route data, arrival time data, GPS track data and card swiping data.

Exemplarily, step two includes:

The historical passenger flow data is obtained according to the historical operation data.

The expected passenger flow data is obtained according to the historical passenger flow data using a spatiotemporal graph convolutional network.

Specifically, in this embodiment, historical operation data is counted at 5-minute time intervals, and historical passenger flow data every 5 minutes is obtained based on a comprehensive analysis of departure time data, route data, arrival time data, GPS trajectory data and card swiping data, including passenger flow data for each station at a certain historical moment.

The trained spatiotemporal graph convolutional network is used to analyze and predict the historical passenger flow data to obtain the expected passenger flow data. The training process of the spatiotemporal graph convolutional network includes: taking the data in the historical passenger flow training set (i.e., the training data set established based on the historical passenger flow data) as the input of the spatiotemporal graph convolutional network, and outputting the expected passenger flow training data; taking the actual passenger flow data after the expected time corresponding to the input data as the target value, and after several training iterations, the loss function is minimized to obtain the trained spatiotemporal graph convolutional network.

Exemplarily, in step 3, the step of constructing a scheduling model according to the expected passenger flow data using a Q-learning algorithm includes:

Create a Q matrix with rows representing states and columns representing actions.

The Q matrix is trained using the expected passenger flow data to obtain a trained Q matrix, namely the scheduling model.

Specifically, in this embodiment, the agent of the Q matrix is represented as a bus, the state is represented as the passenger flow of each bus line, and the action is represented as selecting a certain time and a certain bus line for departure. The agent will traverse all lines, obtain the maximum Q value of the action combination in the current state, obtain the action corresponding to the maximum Q value and execute it, and then transfer to the next state.

In this embodiment, all initial Q values are set to 0. Set the learning rate (alpha) to 0.5, the discount factor (gamma) to 0.9, set the number of training times to 100, the maximum number of training steps to 20, and the ε value in the ε-greedy strategy to 0.2. In other possible embodiments, the above parameters can also be adjusted according to actual conditions.

Exemplarily, in step 3, the using the expected passenger flow data to train the Q matrix includes:

A, initialize the current state to the starting state.

B1, according to the current state and Q matrix, use the ε-greedy strategy to select the decision action.

B2, execute the decision action to obtain a new state.

B3, observe the new state and immediate reward.

B4, updates the new Q value into the Q matrix.

B5, sets the new state as the current state.

B6, if the preset number of training steps is reached or the end state is reached, proceed to the next step, otherwise return to step B1.

C. If the preset number of training times is reached, the training is completed, otherwise return to step A.

Exemplarily, in step B3, the instant reward is obtained through a pre-designed reward function.

The reward function includes: an operating income reward function, an operating cost reward function and a passenger time cost reward function.

Specifically, considering the operating income, operating costs, and passenger time costs brought by passengers, the reward function R can be expressed as , where _RI represents operating revenue, _RO represents operating costs, and _RP represents passenger time cost.

The operating income reward function is as follows:

Where k is the number of passengers at station j and s is the fare.

The operating costs of bus companies include fixed costs and vehicle operating costs. The vehicle operating costs and operating mileage are positively correlated. The vehicle operating costs are directly used to represent the operating costs. The operating cost reward function is as follows:

in represents the current operating cost between site i and site j,/> represents the distance between station i and station j, p represents the unit fuel consumption cost, taking the actual price, n represents the number of stations, the departure point is recorded as the 0th station, and the parking lot is recorded as the n+1th station.

Assume that all passengers arrive at the station on time within the bus arrival time, and the passenger's time cost is the time cost of the passenger waiting caused by the late arrival of the bus at the station. The passenger time cost reward function is as follows:

in represents the time cost of passengers at station j; k is the number of passengers at station j; /> Indicates the actual time when the bus arrives at station j; /> is the latest time of the time window of station j;/> The time value of the passenger is the preset value of the time saved by the passenger when taking the bus. In this embodiment, the time value of the passenger is set to 50 yuan per hour; It is a positive number close to 0, and in this embodiment is taken as 0.0001 to avoid the denominator being 0.

In summary, the reward function is as follows:

Exemplarily, in step B4, the Q-value is updated using a Q-learning update strategy.

Specifically, first define the memory matrix To record all the states st and corresponding actions at experienced by the agent in sequence; let the memory matrix be a matrix with h rows and 2 columns, where h represents the number of states experienced from the initial moment to the current moment; with the memory matrix in/> Find the Q value corresponding to the previous "state-action" for the index and update it; then subtract 1 from t and determine whether t-1 is 0. If it is 0, it means that the Q values of all the "state-actions" that the state st has experienced before have been updated; if it is not 0, find the Q value of the next "state-action" and update it until all Q values are updated. The Q-learning update strategy is as follows:

in Display status/> Take Action/> The updated Q value, /> Indicates the state at time g, /> Display status/> Actions taken, /> Display status/> Take Action/> Instant rewards received, /> is the discount factor, /> Display status/> The maximum Q value that can be obtained by taking action a is g=t-1,t-2,… ,2,1.

Exemplarily, in step three, after obtaining the scheduling model, the scheduling model is tested using the expected passenger flow data.

Specifically, the testing steps include:

A`, initialize the current state to the starting state.

B1`, according to the current state and the trained Q matrix, select the action with the maximum Q value as the decision action.

B2`, execute the decision action to obtain a new state.

B3`, observe the new state and immediate reward.

B4`, set the new state as the current state.

B5`, if the end state is reached, end the test, otherwise return to step B1`.

Exemplarily, step four includes:

Get real-time operation data of the public transportation system.

The expected passenger flow data corresponding to the real-time operation data is obtained and input into the scheduling model, and a scheduling decision is output.

Actual scheduling is performed according to the scheduling decision.

Specifically, the expected passenger flow data corresponding to the real-time operation data is used as the current state of the scheduling model, and the action with the maximum Q value is selected as the decision action, that is, the scheduling decision; and a bus departure timetable or route planning is generated according to the scheduling decision.

As shown in FIG. 2 , exemplarily, step 4 further includes: performing performance evaluation on the scheduling model.

When the performance evaluation result is lower than the preset expectation, return to step 1 and repeat until the performance evaluation result is higher than or equal to the preset expectation.

Specifically, the generated bus departure timetable or route planning is used for simulation. Based on the results of the simulation, the performance indicators of the bus system are evaluated, including passenger waiting time and vehicle utilization. When the performance evaluation result is lower than the preset expectation, return to step 1 for a loop, adjust the hyperparameters such as learning rate and discount factor, and retrain and test until the performance evaluation result is higher than or equal to the preset expectation.

The embodiment of the present invention predicts the expected passenger flow data within a preset time through the historical operation data of the public transportation system, and then uses the Q-learning algorithm to construct a scheduling model according to the expected passenger flow data, thereby improving the correlation between the collected data and the scheduling decision, and improving the accuracy of public transportation scheduling; the proposed spatiotemporal graph convolutional network is used to obtain the expected passenger flow data according to the historical passenger flow data, thereby improving the accuracy of the expected passenger flow data; the proposed reward function includes: an operating income reward function, an operating cost reward function and a passenger time cost reward function, and improves the accuracy of public transportation scheduling by considering the influence of the passenger time cost; the proposed scheduling model is subjected to performance evaluation, and when the performance evaluation result is lower than the preset expectation, it returns to step one and loops until the performance evaluation result is higher than or equal to the preset expectation, thereby improving the optimizability of the scheduling model.

In a possible embodiment, the research principle of this application is as follows:

Depending on whether the bus company adjusts the line capacity configuration, the dispatching operation mode mainly includes two types: capacity unchanged type and capacity increased type.

Capacity unchanged type. Under the capacity unchanged operation mode, the number of buses on the bus routes will not be adjusted. By adjusting the departure intervals of full-distance buses, some capacity will be reserved for the dispatch of rapid buses. Rapid buses and full-distance buses are consistent in terms of operating routes, vehicle configuration and road rights, and only differ in stops and departure frequencies. This mode is suitable for routes with relatively high current capacity levels, and under this dispatch mode, the vehicle purchase costs of bus companies will not be increased.

Capacity increase type. Under the capacity increase operation mode, the departure plan of the existing full-distance buses will not be adjusted, and the bus company will add additional vehicles for rapid bus dispatch. Rapid buses and full-distance buses are consistent in terms of operating routes and road rights, but will differ in terms of stops, departure frequency and vehicle configuration. This mode is suitable for routes with low current capacity levels. Under this dispatch mode, the vehicle acquisition cost of the bus company will increase to a certain extent.

When the passenger flow on a bus route has concentrated demand, presents a multi-peak distribution pattern, and the proportion of passengers traveling medium and long distances is large, it is necessary to adopt the rapid bus dispatching mode. Generally, parameters such as direction imbalance coefficient and station imbalance coefficient are used to determine the opening direction and service stations of rapid buses.

Determination of the opening direction. The opening direction of the BRT is mainly determined by the directional unevenness coefficient, which is the ratio of the larger value of the up and down passenger flow of the line to the average up and down passenger flow. By comparing the up and down direction unevenness coefficients of the line, when the passenger flow in one direction is significantly greater than the passenger flow in the other direction, the BRT is opened in the direction with less passenger flow to increase vehicle turnover.

Determine the stop sites. The number of BRT service sites has a direct impact on both passengers and bus companies. If the number of sites is too small, it will cause problems such as insufficient passenger flow, waste of resources and too little revenue; if the number of sites is too large, the operation time of BRT will increase, and the advantage over full-route buses will not be obvious. The selection of sites is mainly based on large passenger flow boarding and alighting sites and hub sites. In the actual selection process, some intermediate sites that are relatively close to the starting station or the terminal station have a small station imbalance coefficient due to a large number of passengers boarding but a small number of passengers disembarking, or a large number of passengers disembarking but a small number of passengers boarding. They cannot enter the BRT site set, and will lose some passenger flow to a certain extent. Therefore, when selecting BRT sites, these sites with large passenger flow need to be included in the BRT site set.

A single full-route vehicle dispatch cannot solve the problem of uneven distribution of passenger flow in time, stations and directions along the bus line. By studying the passenger flow distribution pattern at stations during peak periods on one-way lines and proposing a rapid bus dispatch strategy for one-way bus lines during peak periods, a basis can be provided for capacity allocation during off-peak periods and for bus lines with strong tidal characteristics.

The BRT dispatching strategy will also have a negative impact on line operations. For example, the BRT dispatching slows down the turnover of full-route vehicles, and the waiting time of passengers at non-BRT service stations increases. Therefore, the BRT dispatching problem is mainly to study the optimal service frequency and the optimal service station, so as to take into account both passenger benefits and bus company operating benefits.

In a possible embodiment, the constraints of the dispatching strategy mainly include: full load rate, passenger waiting time at the station and departure frequency.

Full load rate: The full load rate is the ratio of the actual passenger capacity in the vehicle to the rated passenger capacity of the vehicle, and is an indicator to measure the vehicle utilization rate. If the full load rate is too high, the passenger comfort will be greatly reduced; if the full load rate is too low, the economic benefits of the bus company cannot be guaranteed. Therefore, under the premise of ensuring social benefits, it is necessary to ensure the economic benefits of the bus company through full load rate constraints to achieve the best benefits for passengers and bus companies. According to the "Urban Public Transport Management Regulations", considering the passenger comfort, the full load rate should not exceed 120%, and considering the economic benefits of the bus company, the full load rate should not be less than 50%.

Passenger waiting time at the station: According to the changes in the psychological state of passengers while waiting for the bus, passengers generally feel very anxious and irritable after waiting for 15 minutes and want to change other modes of transportation, while they will despair after 50 minutes and decide not to take the bus anymore. Therefore, the upper and lower limits of the waiting time for this application are 25 minutes and 2 minutes respectively.

Departure frequency: The departure frequency is closely related to the interests of passengers and bus companies. If the departure frequency is too low, the waiting time of passengers will increase, and some passengers may be lost; if the departure frequency is too high, the economic benefits of the bus company cannot be guaranteed. According to relevant research, when passenger satisfaction reaches 80%, the bus departure interval during peak hours is 3 minutes. Therefore, the upper limit of the departure frequency is 20 buses/hour.

In a possible embodiment, Python is used as the language for simulation construction, and real-time passenger flow information of each bus line is obtained by fusion analysis of multi-source data such as bus GPS trajectory data, bus arrival table, bus card swiping data, and bus line network information. When the passenger flow of the line is too much or the road section is congested, the bus number is dynamically changed to meet the passenger's demand for riding and reduce road congestion. This application uses a capacity-increasing bus scheduling method, and reinforcement learning learns the optimal control strategy to obtain the maximum overall reward by observing the changes in each state in the environment and obtaining action feedback from the environment.

Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. A bus dispatching method based on Q-learning is characterized by comprising the following steps:

step one, acquiring historical operation data of a public transportation system;

step two, expected passenger flow data in preset time are obtained according to the historical operation data;

thirdly, constructing a scheduling model according to the expected passenger flow data by utilizing a Q-learning algorithm;

and step four, applying the scheduling model to actual bus system operation.

2. The bus dispatching method based on Q-learning according to claim 1, wherein in the first step, the historical operation data includes: departure time data, line data, arrival time data, GPS track data, and swipe card data.

3. The bus dispatching method based on Q-learning according to claim 1, wherein the second step comprises:

obtaining historical passenger flow data according to the historical operation data;

and obtaining the expected passenger flow data according to the historical passenger flow data by adopting a space-time diagram convolution network.

4. The bus dispatching method based on Q-learning according to claim 1, wherein in the third step, the step of constructing a dispatching model according to the expected passenger flow data by using Q-learning algorithm includes:

creating a Q matrix, wherein rows represent states and columns represent actions;

and training the Q matrix by adopting the expected passenger flow data to obtain a trained Q matrix, namely the scheduling model.

5. The bus dispatching method based on Q-learning according to claim 4, wherein in step three, the training the Q matrix using the expected passenger flow data comprises:

a, initializing a current state as an initial state;

b1, selecting a decision action by using an epsilon-greedy strategy according to the current state and the Q matrix;

b2, executing the decision action to obtain a new state;

b3, observing a new state and instant rewards;

b4, updating the new Q value into the Q matrix;

b5, setting the new state as the current state;

if the preset training step number is reached or the end point state is reached, entering the next step, otherwise returning to the step B1;

and C, if the preset training times are reached, finishing training, otherwise, returning to the step A.

6. The bus dispatching method based on Q-learning according to claim 5, wherein in step B3, the instant rewards are obtained by a pre-designed rewarding function;

the reward function includes: an operating revenue rewards function, an operating cost rewards function, and a passenger time cost rewards function.

7. The bus dispatching method based on Q-learning according to claim 5, wherein in step B4, Q value update is performed by using Q-learning update strategy.

8. The bus dispatching method based on Q-learning according to claim 4, wherein in step three, after the dispatching model is obtained, the expected passenger flow data is further adopted to test the dispatching model.

9. The bus dispatching method based on Q-learning as set forth in claim 1, wherein the fourth step comprises:

acquiring real-time operation data of a public transport system;

obtaining expected passenger flow data corresponding to the real-time operation data, inputting the expected passenger flow data into the scheduling model, and outputting a scheduling decision;

and carrying out actual scheduling according to the scheduling decision.

10. The bus dispatching method based on Q-learning according to claim 1, further comprising, after the fourth step: performing performance evaluation on the scheduling model;

and when the performance evaluation result is lower than the preset expectation, returning to the step one for circulation until the performance evaluation result is higher than or equal to the preset expectation.