CN110716550B - Gear shifting strategy dynamic optimization method based on deep reinforcement learning - Google Patents

Abstract

The invention belongs to the field of construction machinery and vehicle engineering, and in particular relates to a dynamic optimization method for shifting strategies based on deep reinforcement learning. It includes the following steps: (1): determine the state input variable and action output variable of the shifting strategy; (2): determine the Markov decision process of the shifting strategy according to the state input variable and the action output variable; (3): according to the shifting strategy The strategy objective establishes the reward function of the reinforcement learning shifting strategy; (4): According to the Markov decision process and the reward function, solve the deep reinforcement learning shifting strategy; (5): Put the prediction Q network calculated in step (4) into the Shift strategy controller, during the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller; (6): The prediction Q network is regularly updated during the driving process. The invention updates the shifting strategy through the deep reinforcement learning method, and realizes the dynamic optimization of the shifting strategy.

Description

A dynamic optimization method for shifting strategy based on deep reinforcement learning

technical field

The invention belongs to the field of construction machinery and vehicle engineering, and in particular relates to a dynamic optimization method of a gear shifting strategy based on deep reinforcement learning.

Background technique

Shift strategy is one of the core technologies of current construction machinery and vehicle control technology. The solution method is the key consideration in establishing the shifting strategy. The methods of solving the shifting strategy include graphical method, analytical method, genetic algorithm, dynamic programming method and so on. The solution and optimization of the shifting strategy is the core direction of the research on the shifting strategy, especially the dynamic optimization of the shifting strategy is one of the difficulties in the research of the shifting strategy.

"Optimal Dynamic AMT Shifting Law Correction Based on Variable Load", Li Hao, Control Engineering, Vol. 22, No. 1, pp. 50-54, January 2015. On the basis of the two-parameter shifting strategy, the acceleration is introduced as the shifting parameter, and the dynamic three-parameter shifting that considers the acceleration is realized. The solution method is an analytical method. In the solution process, the acceleration-speed curve needs to be fitted for each accelerator opening. The solution is complicated and the calculation amount is large. At the same time, it can only be performed for a single performance index, and it cannot be dynamically performed for the actual driving conditions. optimization.

"Performance Evaluation Approach Improvement for Individualized Gearshift Schedule Optimization", Yin X, May 2016. The genetic algorithm is used to optimize the shifting strategy, which improves the comprehensive performance of the shifting strategy, and solves the problem that the analytical method can only solve a single performance index, but it is also unable to dynamically optimize the actual driving conditions.

“Optimal gear shift strategies for fuel economy and driveability,” VietDacNgo, Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, Vol. 227, No. 10, pp. 1398-1413, October 2013. The shifting strategy is solved for a specific driving cycle by a dynamic programming method. The disadvantage is that when dynamic programming solves the shifting law, it needs to construct a complex state diagram, and the state diagram is represented in the form of a table. The complexity of the state diagram depends on the degree of discreteness in the dynamic programming algorithm. Overly complex state diagrams will experience slow or no convergence due to the Bellman latitude catastrophe. At the same time, due to the optimization for a specific driving cycle, dynamic optimization cannot be carried out during driving.

In the existing patent, Patent Application No. 201710887558.X discloses a method for optimizing the gear shift rule of a vehicle based on a dynamic programming algorithm. According to the embodiment, the shifting laws based on economy and power are respectively formulated. When dynamic programming solves the shifting law, it needs to construct a complex state diagram, and the complexity of the state diagram depends on the degree of discreteness in the dynamic programming algorithm. Overly complex state diagrams will experience slow or no convergence due to the Bellman latitude catastrophe. At the same time, dynamic optimization for the actual driving situation is not possible.

In the existing patent, Patent Application No. 201811306659.4 discloses a method and system for modifying a gear shift strategy based on driving intention. According to the driver's driving process, the current shift correction coefficient and compensation offset value are updated, and the original shift strategy is corrected to realize the dynamic update of the shift strategy. However, the dynamic update rules of its shifting strategy need to be formulated manually, and the optimization effect is greatly affected by the artificial formulation. At the same time, the optimization method is not universal and can only be used for a single model. The degree of intelligence is low.

In general, most of the existing shifting strategy solving or optimization methods cannot be dynamically optimized for actual driving conditions, and the adaptive ability is poor. Some shifting strategies that can achieve dynamic optimization require artificially formulating dynamic update rules for shifting strategies, which are less intelligent and versatile.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a dynamic optimization method of shifting strategy based on deep reinforcement learning. The invention constructs the shift strategy Markov decision process and reward function, then uses the deep reinforcement learning method to solve the shift strategy, and then puts the prediction Q network solved by the deep reinforcement learning method into the shift strategy controller to realize the shift position At the same time, the driving data is collected in the daily driving process, and the shifting strategy is updated through the deep reinforcement learning method to realize the dynamic optimization of the shifting strategy.

The technical solution for realizing the purpose of the present invention is: a dynamic optimization method for shifting strategy based on deep reinforcement learning, comprising the following steps:

Step (1): determine the state input variable and action output variable of the shifting strategy;

Step (2): According to the state input variables and action output variables of step (1), determine the Markov decision process of the shifting strategy;

Step (3): establish a reinforcement learning shift strategy reward function according to the shift strategy target;

Step (4): According to the Markov decision process in step (2) and the reward function in step (3), solve the deep reinforcement learning shifting strategy; specifically, first calculate the Markov decision process and reward function through the Markov decision process. Fu chain, save the Markov chain into the experience pool, and then update the prediction Q network in the deep reinforcement learning shifting strategy according to the data in the experience pool;

Step (5): put the predicted Q network calculated in step (4) into the shift strategy controller, and during the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller;

Step (6): During the driving process, collect construction machinery and vehicle driving data and save them into the experience pool, and update the predicted Q network regularly. After the update is completed, put the predicted Q network into the shift strategy controller to implement the shift strategy. Dynamic optimization.

和油门开度α_t，行驶坡度和地面摩擦阻力系数；动作输出变量包括档位操作和换挡操作，其中档位操作包括升档、降档或保持档位，换挡操作即选择的挡位n_g。Further, the state input variables in the step (1) include vehicle speed v, acceleration

and accelerator opening α _t , driving gradient and ground friction resistance coefficient; action output variables include gear operation and shifting operation, where gear operation includes upshifting, downshifting or maintaining gear, and shifting operation is the selected gear n _g .

Further, the shift strategy Markov decision process in the step (2) is expressed as the form of the transfer function where the state at the next moment is the current state and the selected action, and the form of the transfer function is as follows:

s _{t +1} =T(s _t ,at )

In the formula, s _t+1 is the state variable at the next moment, s _t is the current state variable, and at _is the selected action variable, where s ∈ S, a ∈ A, S is the set of state variables, and A is the action A collection of variables.

Further, the shift strategy reward function in the step (3) is positively correlated with the shift strategy target, and the shift strategy target includes power, economy and comfort.

Further, the target of the shifting strategy is a dynamic shifting strategy, which is described as the construction machinery and the vehicle reaching the maximum speed in the shortest time t under the condition of comfort constraints. The reward and punishment mechanism is:

In the formula, r is the reward calculated by the reward and punishment mechanism; r _t is the temporary reward, r _t =-0.001||V _Tamx -v||; v _Tmax is the maximum vehicle speed under the current accelerator opening α _t ; J is the construction machinery and the impact degree of the vehicle; J _max is the designed maximum allowable impact degree.

Further, the form of the Markov chain in the step (4) is:

<s _t ,a _t ,r _t ,s _t+1 >

In the formula, r _t is the temporary reward calculated according to the reward target.

Further, the deep reinforcement learning method in the step (4) includes two neural networks with the same structure but different parameters, called the prediction Q network and the target Q network, wherein the function of the prediction Q network is to calculate the different parameters in the current state. The Q value of the action, which the target Q network uses to update the prediction Q network.

Further, in the fourth step, in the established Markov chain, the selection of the action variable at is through a _greedy algorithm, and the greedy algorithm is expressed as:

In the formula, Q _p is the predicted Q network, θ _p is the predicted Q network parameter, and e is the greedy algorithm parameter;

In the step (4), the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool, and the prediction Q network is used to calculate the driving state s _t For the Q value under the lower gear set A, the output of the predicted Q network is Q _p (s, A, θ _p ).

Further, in the step (5), during the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shifting strategy controller, and the shifting controller selects the appropriate gear a according to the predicted Q network. *:

a*(s)=argmax _a [Q _p (s,a,θ _p )|a∈A]

In the formula, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter.

Further, the collected driving data in the step (6) includes: vehicle speed, accelerator opening, acceleration, driving gradient and ground frictional resistance coefficient;

There are two methods for updating the predicted Q network in the step (6): the first method is to reconstruct the transfer function in the step (2) through the construction machinery and vehicle driving data, and then according to the step (3) and the step (4) ) update the prediction Q network; Method 2 is to directly update the prediction Q network according to the prediction Q network update method in step (4);

Among them, the first method is to reconstruct the transfer function in step (2) by collecting construction machinery and vehicle driving data, and the reconstruction method is to recalculate the parameters in the transfer function to form a transfer function with the same structure but different parameters, or Fit transfer function by applying neural network, linear fitting and Fourier transform;

Among them, the second method is to collect the construction machinery and vehicle driving data and then update according to the predicted Q network update method in step (4). The update method of the predicted Q network is:

Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γmax _a Q _t (s,a,θ _t )-Q _p (s,a,θ _p )) ²

In the formula, γ is the decreasing reward value; α is the learning rate of the neural network; Q _t is the target Q network, and θ _t is the target Q network parameter

Compared with the prior art, the present invention has the following significant advantages:

(1) This application adopts the deep reinforcement learning method, because the Markov chain including the Markov decision process and the reward function can be constructed according to the construction machinery and the driving process of the vehicle to realize the principle of updating the prediction Q network, which can solve the problem of swapping. The problem of solving and dynamic optimization of blocking strategy; it has the characteristics of strong self-adaptive ability;

(2) This application adopts the deep reinforcement learning method, because the algorithm itself is unified in the steps of solving and dynamic optimization, and is not affected by the controlled object ontology, and can be applied to passenger cars, construction machinery and vehicles, and special vehicles. Different models such as electric vehicles and electric vehicles; the reason is that the transfer function can be fitted by using neural network, linear fitting and Fourier transform method, which is not affected by the applied object ontology, and has the characteristics of strong versatility;

(3) The present application solves and dynamically optimizes the shifting strategy by using the deep reinforcement learning method, because the algorithm itself is not affected by the controlled object ontology and can realize the dynamic optimization of the shifting strategy, and has the characteristics of strong intelligence;

(4) In this application, the selection of gears is realized by using the prediction Q network in deep reinforcement learning instead of the tabular form in the traditional method. Because the neural network has strong fitting ability, it can be applied to the gear shifting strategy under high-dimensional state variables. , which can solve the problem of the Bellman latitude catastrophe.

Description of drawings

FIG. 1 is a schematic diagram of the dynamic optimization method of shifting strategy based on deep reinforcement learning of the present invention.

FIG. 2 is a flow chart of the present invention for solving the deep reinforcement learning shifting strategy.

FIG. 3 is a diagram of a neural network structure model adopted in the present invention.

FIG. 4 is a schematic diagram of the dynamic optimization process of the shifting strategy of the present invention.

Detailed ways

The invention provides a dynamic optimization method of shifting strategy based on deep reinforcement learning. The invention constructs the shift strategy Markov decision process and reward function, and then uses the deep reinforcement learning method to solve the shift strategy. Then, the prediction Q network solved by the deep reinforcement learning method is put into the shift strategy controller to realize the gear selection. At the same time, the driving data is collected in the daily driving process, and the shifting strategy is updated through the deep reinforcement learning method. Realize dynamic optimization of shifting strategies.

A method for dynamic optimization of shifting strategy based on deep reinforcement learning, comprising the following steps:

Step 1: Determine the state variable and action variable of the shifting strategy.

Step 2: Determine the Markov decision process of shifting strategy according to state input variables and action output variables.

Step 3: Establish a reward function of the reinforcement learning shifting strategy according to the shifting strategy optimization objective.

Step 4, according to the Markov decision process of step 2 and the reward function of step 3, solve the deep reinforcement learning shifting strategy. First, the Markov chain is calculated through the established Markov decision process and reward function, and the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool.

Step 5, put the predicted Q network calculated in step 4 into the shift strategy controller. During the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller.

Step 6: During the driving process, collect the driving data of construction machinery and vehicles and save it into the experience pool, and update the predicted Q network regularly. After the update is completed, put the predicted Q network into the shift strategy controller to realize dynamic optimization of the shift strategy. .

Further, in the step 1, the state variable of the shifting strategy is the state variable of the construction machinery and the driving of the vehicle or the external environment variable. The action variable is gear operation or shift operation. Gear operations include upshifting, downshifting, or holding a gear; a shift operation is the selected gear.

In the second step, the Markov decision process of the shifting strategy is expressed in the form of a transition function T where the state at the next moment is the current state and the selected action. The transfer function has the form:

s _{t +1} =T(s _t ,at )

In the formula, s _t+1 is the state variable at the next moment, s _t is the current state variable, and at _is the selected action variable. Among them, s∈S, a∈A. S is a set of state variables, and A is a set of action variables. In the shifting strategy, the state variables are construction machinery and vehicle driving state variables or external environment variables, including vehicle speed, accelerator opening, acceleration, driving gradient and ground friction resistance coefficient. Action variables include gear operation or shift operation.

In the third step, the established shift strategy reward function is positively correlated with the shift target.

In the step 3, the shifting targets include power, economy, and comfort.

In the fourth step, the Markov chain is calculated through the established Markov decision process and reward function. A Markov chain has the form:

<s _t ,a _t ,r _t ,s _t+1 >

In the formula, r _t is the temporary reward calculated according to the reward target.

In the said step 4, in the established _Markov chain, the selection of the action at is through the greedy algorithm, and the greedy algorithm is expressed as:

In the formula, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter. e is the greedy algorithm parameter.

In the fourth step, the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool. The predicted Q network is used to calculate the Q value in the gear set A in the driving state _st . The output of the predicted Q network is Q _p (s, A, θ _p ), and the update method of the predicted Q network is:

Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γmax _a Q _t (s,a,θ _t )-Q _p (s,a,θ _p )) ²

In the formula, γ is the decreasing reward value; α is the learning rate of the neural network; Q _t is the target Q network. θ _t is the target Q network parameter.

In the fifth step, during the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller. The shift controller selects the appropriate gear a* according to the predicted Q network.

a*(s)=argmax _a [Q _p (s,a,θ _p )|a∈A]

In the formula, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter.

In the step 6, the collected driving data includes: vehicle speed, accelerator opening, acceleration, driving gradient and ground friction resistance coefficient.

In the above-mentioned step 6, there are two methods for updating the prediction Q network. The first method is to reconstruct the transfer function in the second step through the construction machinery and vehicle driving data, and then update the prediction Q network according to the third step and the fourth step. The second method is to directly update the predicted Q network according to the method for updating the predicted Q network in the fourth step.

In the above-mentioned step 6, the first method of updating the prediction Q network is to reconstruct the transfer function in the second step by collecting construction machinery and vehicle driving data. The reconstruction method is to recalculate the parameters in the transfer function to form a structure. The same transfer function but with different parameters, or by applying neural networks, linear fitting and Fourier transform methods to fit transfer functions, etc.

In the described step 6, the second method of updating the predicted Q network is to update the predicted Q network by collecting the construction machinery and vehicle driving data and then update it according to the predicted Q network update method in step 4. The update method of the predicted Q network is:

Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γmax _a Q _t (s,a,θ _t )-Q _p (s,a,θ _p )) ²

In the step 6, the dynamic optimization of the shifting strategy is realized through the update of the predicted Q network in the deep reinforcement learning.

Example

The invention provides a dynamic optimization method for shifting strategy based on deep reinforcement learning. The invention constructs the Markov decision process of the shifting strategy, and then uses the deep reinforcement learning method to solve the shifting strategy. After the solution is completed, the prediction Q network trained by deep reinforcement learning is put into the shift strategy controller to realize the gear selection. Then, during the driving process, the predicted Q network is updated by collecting the data of construction machinery and vehicle driving to realize the dynamic optimization of the shifting strategy. The update method of the prediction Q network includes: updating the prediction Q network by reconstructing the shift strategy transfer function of the construction machinery and vehicle driving data and updating the prediction Q network directly according to the deep reinforcement learning method. The principle of the dynamic optimization method of shifting strategy based on deep reinforcement learning is shown in Figure 1, including the following steps:

Step 1: Determine the state variable and action variable of the shifting strategy.

Step 2: Determine the Markov decision process of shifting strategy according to state input variables and action output variables.

Step 3: Establish a reward function of the reinforcement learning shifting strategy according to the shifting strategy optimization objective.

Step 4: According to the Markov decision process of step 2 and the reward function of step 3, the deep reinforcement learning shifting strategy is solved. First, the Markov chain is calculated through the established Markov decision process and reward function, and the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool.

Step 5, put the predicted Q network calculated in step 4 into the shift strategy controller. During the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller.

Step 6: During the driving process, collect the driving data of construction machinery and vehicles and save it into the experience pool, and update the predicted Q network regularly. After the update is completed, put the predicted Q network into the shift strategy controller to realize dynamic optimization of the shift strategy. .

The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

和油门开度α_t。动作变量为挡位n_g。Step 1: Determine the state variable and action variable of the shifting strategy. In an embodiment, the shift strategy state variables are vehicle speed v, acceleration

and the accelerator opening α _t . The action variable is the gear position n _g .

In an embodiment, the shift strategy Markov decision process is determined according to state variables (vehicle speed, acceleration, accelerator opening) and action variables (gear position). The state transition function T of the Markov decision process is:

In the formula, T _e is the output torque of the engine; i _g is the transmission ratio corresponding to the gear position n _g ; i ₀ is the transmission ratio of the main reducer; η _t is the efficiency of the transmission system; m is the total weight of the vehicle; β is the equivalent slope OK. C _d is the air resistance coefficient; A is the windward area of the car; F _b is the braking force; R is the effective turning radius of the tire; ρ is the air density.

Step 3: Establish a reinforcement learning shift strategy reward function according to the shift target. In this embodiment, the learning target is a dynamic shifting strategy, which is described as the construction machinery and the vehicle reaching the maximum speed in the shortest time t under the condition of comfort constraints. The reward and punishment mechanism is:

In the formula, r is the reward calculated by the reward and punishment mechanism; r _t is the temporary reward, r _t =-0.001||V _Tamx -v||; v _Tmax is the maximum vehicle speed under the current accelerator opening α _t ; J is the construction machinery and the impact degree of the vehicle; J _max is the designed maximum allowable impact degree.

Step 4: According to the Markov decision process of step 2 and the reward function of step 3, the deep reinforcement learning shifting strategy is solved. First, the Markov chain is calculated through the established Markov decision process and reward function, and the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool. The process of step 4 is shown in FIG. 2 . The specific steps are as follows.

Step 1: Initialize state variables and action variables, and calculate the state at the next moment according to the established Markov decision process transfer function.

Step 2: Calculate the reward through the designed reward and punishment mechanism.

Step 3: Represent the above state-action-next moment state and reward as a Markov chain and save it into the experience pool.

Step 4: Taking the state at the next moment as the current state, the prediction Q network calculates the Q value under each action according to the current state, and then calculates the actual selected gear in the current state according to the Q value under each action through the greedy algorithm. Then go back to the first step and repeat the cycle.

In the above steps, when the number of Markov chains in the experience pool reaches a predetermined number, the prediction Q network starts to be updated.

The update process of the predicted Q network is completed by the predicted Q network and the target Q network. The update method of the predicted Q network is:

Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γma _a xQ _t (s,a,θ _t )-Q _p (s,a,θ _p )) ²

In the formula, γ is the decreasing value of reward; α is the learning rate of the neural network; Q _t is the target Q network. θ _t is the target Q network parameter.

In the update process of the prediction Q network, the parameters of the prediction Q network need to be imported and copied to the target Q network regularly to realize the update of the target Q network.

The prediction Q-network and the target Q-network have the same neural network structure. In this embodiment, the neural network structure model adopted by the prediction Q network and the target Q network is shown in FIG. 3 . The neural network structure model used has five fully connected layers as intermediate layers, and adopts the linear rectification function ReLU as the neural network activation function. The linear rectification function ReLU is expressed as:

ReLU(x)=max(0,Wx+b)

In the formula: W is the weight of the neural network; b is the partial rank of the neural network; x is the input of the neural network.

In this embodiment, the neural network data input is state variables (vehicle speed, accelerator opening, acceleration). The output layer outputs the Q values corresponding to all gears n _g . The larger the Q value, the greater the maximum discount cumulative reward value can be obtained by selecting the gear corresponding to the Q value in the current state.

Step 5, put the predicted Q network calculated in step 4 into the shift strategy controller. During the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller.

In step 5, the construction machinery and the vehicle select the gear according to the shift strategy controller. Specifically:

a*(s)=argmax _a [Q _p (s,a,θ _p )|a∈A]

In the formula, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter.

Step 6: During the driving process, collect the driving data of construction machinery and vehicles and save it into the experience pool, and update the predicted Q network regularly. After the update is completed, put the predicted Q network into the shift strategy controller to realize dynamic optimization of the shift strategy. .

In step 6, there are two methods for updating the prediction Q network. The first method is to reconstruct the transfer function in the second step through the construction machinery and vehicle driving data, and then update the prediction Q network according to the third step and the fourth step. The second method is to directly update the predicted Q network according to the method for updating the predicted Q network in the fourth step.

In step 6, the first method of updating the prediction Q network is to reconstruct the transfer function in step 2 by collecting construction machinery and vehicle driving data. The reconstruction method is to recalculate the parameters in the transfer function to form the same structure but the parameters Different transfer functions, or by applying neural networks, linear fitting and Fourier transform methods to fit transfer functions, etc. In this embodiment, the reconstructed transfer function is:

According to the reconstruction method, in this embodiment, the parameters in the transfer function can be recalculated. Or fit the transfer function by applying neural networks, linear fitting and Fourier transform methods. No matter which form of reconstruction is performed, the reconstructed transfer function can be uniformly expressed as:

s _{t +1} =T _new (s _t ,at ,Θ)

where Ω is the transfer function parameter.

After the reconstruction is completed, steps 4 and 5 need to be performed again to obtain a new prediction Q network.

In step 6, the second method for updating the predicted Q network is to collect construction machinery and vehicle driving data and then update according to the method for updating the predicted Q network in step 4.

In step 6, the second method of updating the predicted Q network is to collect construction machinery and vehicle driving data and then update it according to the predicted Q network update method in step 4, so as to realize the dynamic optimization process of the shifting strategy, and the dynamic optimization of the shifting strategy. The optimization process is shown in Figure 4. The specific process is as follows:

Step 1: Collect construction machinery and vehicle driving data

Step 2: Process the collected construction machinery and vehicle driving data, and express the processed data in the form of a Markov chain, which can be expressed as:

<s _t , at _t , r _t , s _t+1 >

Step 3: Predict the update of the Q network. The update process of the predicted Q network is completed by the prediction of the Q network and the target Q network. The method is as follows:

Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γmax _a Q _t (s,a,θ _t )-Q _p (s,a,θ _p )) ²

In addition to the above-described embodiments, the present invention may also have other embodiments. All technical solutions formed by equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims (9)

1. a gear shifting strategy dynamic optimization method based on deep reinforcement learning, is characterized in that, comprises the steps: Step (1): determine the state input variable and action output variable of the shifting strategy; Step (2): According to the state input variables and action output variables of step (1), determine the Markov decision process of the shifting strategy; Step (3): establish a reinforcement learning shift strategy reward function according to the shift strategy target; Step (4): According to the Markov decision process in step (2) and the reward function in step (3), solve the deep reinforcement learning shifting strategy; specifically, first calculate the Markov decision process and reward function through the Markov decision process. Fu chain, save the Markov chain into the experience pool, and then update the prediction Q network in the deep reinforcement learning shifting strategy according to the data in the experience pool; Step (5): put the predicted Q network calculated in step (4) into the shift strategy controller, and during the driving process, the construction machinery and the vehicle select the gear according to the shift strategy controller; Step (6): During the driving process, collect construction machinery and vehicle driving data and save them into the experience pool, and update the predicted Q network regularly. After the update is completed, put the predicted Q network into the shift strategy controller to implement the shift strategy. dynamic optimization; Step (1), determine the shifting strategy state variables and action output variables, the shifting strategy state variables are vehicle speed v, acceleration

and the accelerator opening α _t , the action output variable is the gear n _g ; According to the state variables, namely vehicle speed, acceleration, accelerator opening and action output variables, namely gear position, the Markov decision-making process of the shifting strategy is determined. The state transition function T of the Markov decision-making process is:

In the formula, T _e is the output torque of the engine; i _g is the transmission ratio corresponding to the gear position n _g ; i ₀ is the transmission ratio of the main reducer; η _t is the efficiency of the transmission system; m is the total weight of the vehicle; β is the equivalent slope Drag coefficient; C _d is the air resistance coefficient; A is the windward area of the car; F _b is the braking force; R is the effective turning radius of the tire; ρ is the air density; In step (3), the target of the shifting strategy is a dynamic shifting strategy, which is described as the construction machinery and the vehicle reaching the maximum speed in the shortest time t under the condition of comfort constraints. The reward and punishment mechanism is:

In the formula, r is the reward calculated by the reward and punishment mechanism; r _t is the temporary reward, r _t =-0.001||V _Tamx -v||; v _Tmax is the maximum vehicle speed under the current accelerator opening α _t ; J is the construction machinery and the impact degree of the vehicle; J _max is the designed maximum allowable impact degree; The specific steps of step (4) are as follows; Step 1: First initialize state variables and action variables, and calculate the state at the next moment according to the established Markov decision process transfer function; Step 2: Calculate the reward through the designed reward and punishment mechanism; Step 3: Represent the above state-action-next moment state and reward in the form of a Markov chain and save it into the experience pool; Step 4: Taking the state at the next moment as the current state, the prediction Q network calculates the Q value under each action according to the current state, and then calculates the actual selected gear in the current state according to the Q value under each action through the greedy algorithm; then Go back to the first step and repeat the cycle; In the above steps, when the number of Markov chains in the experience pool reaches a predetermined number, the prediction Q network is updated; The update process of the predicted Q network is completed by the predicted Q network and the target Q network. The update method of the predicted Q network is: Q _p (s,a,θ _p )=Q _p (s,a,θ _p )+α(r+γmax _a Q _t (s,a,θ _t )-Q _p (s,a,θ _p )) ² In the formula, γ is the decreasing reward value; α is the learning rate of the neural network; Q _t is the target Q network, θ _t is the target Q network parameter, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter. 2. The method according to claim 1, wherein the state input variables in the step (1) include vehicle speed v, acceleration

and accelerator opening α _t , driving gradient and ground friction resistance coefficient; action output variables include gear operation and shifting operation, where gear operation includes upshifting, downshifting or maintaining gear, and shifting operation is the selected gear n _g . 3. The method according to claim 2, wherein the shift strategy Markov decision process in the step (2) is expressed as the next moment state is the form of the transfer function of the current state and the selected action, The transfer function has the following form: s _{t +1} =T(s _t ,at ) In the formula, s _t+1 is the state variable at the next moment, s _t is the current state variable, and at _is the selected action variable, where s ∈ S, a ∈ A, S is the set of state variables, and A is the action A collection of variables. 4 . The method according to claim 3 , wherein the shift strategy reward function in the step (3) is positively correlated with shift strategy objectives, and the shift strategy objectives include power, economy and comfort. 5 . sex. 5. method according to claim 4 is characterized in that, the form of the Markov chain in described step (4) is: <s _t ,a _t ,r _t ,s _t+1 > In the formula, r _t is the temporary reward calculated according to the reward target. 6. The method according to claim 5, wherein the deep reinforcement learning method in the step (4) comprises two neural networks with the same structure but different parameters, called prediction Q network and target Q network, wherein The role of the prediction Q network is to calculate the Q value of each action in the current state, and the target Q network is used to update the prediction Q network. 7. method according to claim 6 is characterized in that, in step (4), in the established _Markov chain, the selection of action variable at is by greedy algorithm, and greedy algorithm is expressed as:

In the formula, Q _p is the predicted Q network, θ _p is the predicted Q network parameter, and e is the greedy algorithm parameter; In the step (4), the Markov chain is saved in the experience pool, and then the prediction Q network in the deep reinforcement learning shifting strategy is updated according to the data in the experience pool, and the prediction Q network is used to calculate the driving state s _t For the Q value under the lower gear set A, the output of the predicted Q network is Q _p (s, A, θ _p ). 8. The method according to claim 7, characterized in that, in the step (5), during the driving process of the construction machinery and the vehicle, the construction machinery and the vehicle select the gear according to the shift strategy controller, and shift the gears. The controller selects the appropriate gear a* according to the predicted Q network: a*(s)=argmax _a [Q _p (s,a,θ _p )|a∈A] In the formula, Q _p is the prediction Q network, and θ _p is the prediction Q network parameter. 9 . The method according to claim 8 , wherein the collected driving data in the step (6) comprises: vehicle speed, accelerator opening, acceleration, driving gradient and ground frictional resistance coefficient; 10 . There are two methods for updating the predicted Q network in the step (6): the first method is to reconstruct the transfer function in the step (2) through the construction machinery and vehicle driving data, and then according to the step (3) and the step (4) ) update the prediction Q network; Method 2 is to directly update the prediction Q network according to the prediction Q network update method in step (4); Among them, the first method is to reconstruct the transfer function in step (2) by collecting construction machinery and vehicle driving data, and the reconstruction method is to recalculate the parameters in the transfer function to form a transfer function with the same structure but different parameters, or Fit transfer function by applying neural network, linear fitting and Fourier transform; Wherein, the second method is to collect construction machinery and vehicle driving data and then update according to the prediction Q network update method in step (4).

Images