WO2022252559A1 - 基于规则和双深度q网络的混合动力汽车能量管理方法 - Google Patents

基于规则和双深度q网络的混合动力汽车能量管理方法 Download PDF

Info

Publication number
WO2022252559A1
WO2022252559A1 PCT/CN2021/137803 CN2021137803W WO2022252559A1 WO 2022252559 A1 WO2022252559 A1 WO 2022252559A1 CN 2021137803 W CN2021137803 W CN 2021137803W WO 2022252559 A1 WO2022252559 A1 WO 2022252559A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
lithium battery
value
soc
energy
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/137803
Other languages
English (en)
French (fr)
Inventor
郑春花
许德州
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2022252559A1 publication Critical patent/WO2022252559A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
    • B60L50/00—Electric propulsion with power supplied within the vehicle
    • B60L50/40—Electric propulsion with power supplied within the vehicle using propulsion power supplied by capacitors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
    • B60L15/00—Methods, circuits, or devices for controlling the traction-motor speed of electrically-propelled vehicles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
    • B60L50/00—Electric propulsion with power supplied within the vehicle
    • B60L50/50—Electric propulsion with power supplied within the vehicle using propulsion power supplied by batteries or fuel cells
    • B60L50/60—Electric propulsion with power supplied within the vehicle using propulsion power supplied by batteries or fuel cells using power supplied by batteries
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
    • B60L58/00—Methods or circuit arrangements for monitoring or controlling batteries or fuel cells, specially adapted for electric vehicles
    • B60L58/10—Methods or circuit arrangements for monitoring or controlling batteries or fuel cells, specially adapted for electric vehicles for monitoring or controlling batteries
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/60—Other road transportation technologies with climate change mitigation effect
    • Y02T10/70—Energy storage systems for electromobility, e.g. batteries

Definitions

  • the present invention relates to the technical field of vehicle energy management, and more specifically, relates to a hybrid electric vehicle energy management method based on rules and double-depth Q networks.
  • New energy vehicles include hybrid vehicles, electric vehicles and fuel cell vehicles. Energy management is one of the control optimization problems that need to be solved urgently for new energy vehicles.
  • the rule-based energy management methods include deterministic rule control, fuzzy logic control, etc., which have strong real-time performance, but it is difficult to achieve the optimal control effect; although the optimization-based energy management method can obtain better control effect, it requires The working conditions are predictable and the amount of calculation is large, so it is difficult to apply in real time.
  • the overestimation of the Q value makes the control effect not good, and also has defects such as slow convergence speed and poor real-time performance.
  • the real-time and working condition adaptability of the control method are high. , so it is necessary to design an energy management method that can not only meet the actual application requirements but also have excellent control effects.
  • the current energy management method based on deep reinforcement learning is often within the preset constraints, such as lithium battery SOC (state of charge, current remaining power) is not higher than 0.9 and not lower than 0.3 to achieve control, however
  • the relevant state quantities cannot always be kept within the constraint range, and if the lithium battery is frequently overcharged or overdischarged, its service life will rapidly decay, which will significantly reduce the mileage of new energy vehicles and increase the cost of use. Therefore, it is necessary to formulate an additional control method outside the constraint range to obtain a more comprehensive and stable control method, so as to save energy as much as possible while prolonging the service life of lithium batteries, thereby reducing the cost of using new energy vehicles, which is beneficial to Large-scale promotion of new energy vehicles.
  • the purpose of the present invention is to overcome above-mentioned defective of prior art, provide a kind of hybrid electric vehicle energy management method based on rule and double depth Q network, is to realize optimality, real-time property and working condition adaptability by improving depth Q network
  • the technical solution of the invention is to provide a hybrid electric vehicle energy management method based on rules and double deep Q network.
  • the method includes the following steps:
  • Detect vehicle energy sources equipped with a composite energy storage system which includes a lithium battery energy source and a supercapacitor energy source;
  • the trained deep reinforcement learning model is used to determine the output power of the lithium battery; Set rules to protect lithium batteries;
  • the deep reinforcement learning model its agent includes an evaluation Q network and a target Q network
  • the state observations of the environment are the remaining power of the lithium battery, the remaining power of the supercapacitor, and the required power of the whole vehicle
  • the output power of the lithium battery is taken as Output actions, and aim at minimizing the energy loss of the composite energy storage system, and set a corresponding reward function.
  • the present invention has the advantages of prolonging the service life of lithium batteries as much as possible while saving energy, thereby effectively reducing the cost of using new energy vehicles, and taking into account the real-time and optimality of the control method.
  • Fig. 1 is a power architecture of a composite energy storage system for an electric vehicle according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of an energy management method based on a double-depth Q network according to an embodiment of the present invention
  • Fig. 3 is a control logic schematic diagram of an energy management method based on rules and double-depth Q networks according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of the process of a hybrid energy management method based on rules and double-depth Q networks according to an embodiment of the present invention
  • Fig. 5 is a flow chart of a method for energy management of a hybrid electric vehicle based on rules and double deep Q-networks according to an embodiment of the present invention.
  • vehicle models which include vehicle dynamics models, motor models, energy source models, and transmission system models.
  • relevant vehicle models include vehicle dynamics models, motor models, energy source models, and transmission system models.
  • related modeling software such as Matlab/Simulink is used to establish related models. It should be understood that, in addition to Matlab, other computing programs or tools can also be used to implement the present invention.
  • Deep reinforcement learning consists of 3 elements: environment, agent and reward function.
  • the environment is the corresponding vehicle model or no model
  • the agent is the method to be trained to perform related control
  • the reward function needs to be set according to the specific problem.
  • the quality of the setting will affect the learning ability of the agent. The effect is the training and convergence process.
  • we need to pay special attention to the problem of sparse rewards that is, a large number of execution actions are not rewarded.
  • the attributes and dimensions of the agent's state observations and output actions need to be properly set, which will also greatly affect the learning and training effect.
  • the design of the energy management method of DDQN involves the design of neural network, state observation, action amount and reward function, and the setting of some hyperparameters.
  • the design of neural networks involves the selection and design of the number of network layers, number of neurons, connection methods, and activation functions.
  • the DDQN algorithm separates the current action selection from the strategy evaluation, which in turn can reduce overly optimistic method estimates to improve the control effect. It can be said that it is an improvement on the DQN algorithm.
  • careful settings are also required to obtain ideal convergence speed and control effect.
  • the present invention will set actions by incorporating the optimal results of dynamic programming.
  • the experience playback technique In deep reinforcement learning, in order to improve learning efficiency, the experience playback technique will be introduced, which stores an array of past experience samples. During learning and training, the agent uses random sampling (reducing the correlation between samples) from the past experience. to study. However, in actual training, many samples get low rewards or even no rewards, so the learning reference value of these samples is not great.
  • a priority is proposed based on the experience playback technique.
  • the skill of experience playback is to learn by preferentially extracting samples with greater value (that is, to obtain greater rewards), so as to achieve faster convergence.
  • the embodiment of the present invention introduces the skill of priority experience playback to accelerate the convergence of the DDQN algorithm and enhance The real-time nature of the method.
  • Figure 1 is the power transmission system architecture of the composite energy storage system of electric vehicles.
  • Energy sources both of which can be used as energy sources for charging and discharging, that is, to drive vehicles or absorb braking energy
  • supercapacitors are used as auxiliary energy sources to reduce the charging and discharging frequency and discharge current of lithium batteries.
  • the cooperative work of lithium batteries and supercapacitors needs to formulate appropriate energy management methods to reasonably distribute the input and output power of the two.
  • the DCDC converter is responsible for stepping up the output voltage of the supercapacitor or stepping down the voltage from the bus, thereby reducing the requirement for supercapacitor configuration.
  • the vehicle model is established in related modeling software such as Matlab/Simulink according to the quasi-static principle and vehicle dynamics.
  • These models include: demand power calculation model, lithium battery equivalent circuit Model, supercapacitor equivalent circuit model, DCDC converter efficiency model, motor inverter assembly efficiency model, transmission model, etc.
  • the vehicle model can be built after the simulation is correct.
  • the hybrid electric vehicle energy management method based on rules and double-depth Q network uses the energy management method based on DDQN within the normal lithium battery SOC constraint range, and uses the rule-based energy management method to control.
  • the following will specifically introduce the energy management method based on DDQN, the energy management method based on rules and double deep Q network.
  • the interaction process between the environment and the agent is: each time step, in the environment state s t , the agent randomly selects an action from the set action set to output to the environment, the state of the environment Then it changes from st to st+1 .
  • the agent randomly selects an action from the set action set to output to the environment, the state of the environment Then it changes from st to st+1 .
  • the corresponding reward of the action is fed back to the agent.
  • the agent constantly adjusts the selected output action to achieve The maximum cumulative reward is obtained, and the process will be repeated until the reward function converges.
  • the DDQN algorithm separates the action selection from the strategy evaluation to avoid falling into an overestimation of the Q value and affecting the convergence speed and control effect.
  • the present invention also adds the technique of priority experience playback to accelerate the training and learning process, that is, to increase the frequency of important and valuable samples being extracted, which can effectively promote algorithm training and further shorten the convergence time, which is beneficial to the practicality of the algorithm. promotion and application.
  • the designed energy management method based on DDQN includes 2 parts, one part is a vehicle model built in Simulink and a closed-loop model built in conjunction with the Simulink reinforcement learning toolbox (reinforcement learning toolbox) RL agent module
  • the agent-environment (vehicle) model the other part is aimed at designing neural networks and training instructions based on the DDQN energy management method, for example, establishing the m file of Matlab to realize.
  • the design process of the DDQN-based energy management method is as follows.
  • S represents the set of state observations
  • s(t) represents the state observations at time t.
  • the vehicle demand power P dem is normalized, that is, the demand power P dem is reduced to [-1,1], and the arithmetic mean mean(x ) and standard deviation std(x), calculated according to the standard normalized general formula, expressed as:
  • mean(x) and std(x) represent the arithmetic mean and standard deviation of the input state data, respectively.
  • the arithmetic mean mean(x) and standard deviation std(x) are calculated as follows:
  • the feasible action output interval of the agent output action is set based on the global optimal control result based on dynamic programming, and the lithium battery output Power P batt is defined as output action. Since the DDQN algorithm needs to discretize the output action, that is to divide a feasible action interval into n parts.
  • the control result of dynamic programming has the advantage of global optimality, and the result is also a discrete optimal control action sequence, so refer to the energy management method based on dynamic programming with the same optimization objective and control object
  • the output action interval is appropriately set to n parts, that is, the action interval is P batt_max and P batt_min represent the maximum and minimum output power of the lithium battery, respectively.
  • A represents the set of output actions
  • a(t) represents the output action at time t.
  • the energy loss of the composite energy storage system is minimized as the optimization goal, starting from this, the corresponding reward function r(t) is set, and the reward function is established through the corresponding mathematics in Simulink
  • a model implementation for example represented as:
  • E loss represents the overall energy loss of the composite energy storage system
  • SOC b_tgt represents the target value of lithium battery SOC b
  • SOC sc_tgt represents the target value of super capacitor SOC sc
  • L sc , L batt , L dcdc represent super capacitor, lithium
  • m represents the coefficient of the energy loss item
  • n and p represent the coefficients of the SOC change of the balanced lithium battery and the super capacitor, respectively, and these three coefficients need to be adjusted during the training process.
  • constraints are expressed as:
  • I b is the current of the lithium battery
  • I sc is the current of the super capacitor
  • I b_min and I b_max represent the minimum and maximum value of the lithium battery current respectively
  • I sc_min and I sc_max represent the minimum and maximum value of the super capacitor current respectively
  • P dem represents the required power of the whole vehicle
  • P min and P max represent the minimum and maximum value of the required power of the vehicle, respectively.
  • the neural network of DDQN involves the design and selection of structure, number of layers, number of neurons and activation function, all of which need to be properly set and adjusted according to the actual training situation.
  • the neural network of DDQN is designed by writing codes in the m-file of Matlab to call related functions of the neural network, and the connection between the layers of the neural network is carried out by using the full connection method.
  • the activation function of the middle layer is set to the linear rectification function ReLU, for example, the activation function of the network output layer It is set to tanh, so that the output value is constrained to [-1,1].
  • the parameters of the DDQN evaluation Q network as ⁇
  • the parameters of the target Q network as ⁇ ′.
  • the network parameters ⁇ of the evaluation Q network will be set at regular intervals
  • the step size is copied to the target Q-network ⁇ ′.
  • reinforcement learning and deep reinforcement learning are algorithms based on the Bellman principle.
  • the calculation method of the Q value of the DDQN algorithm is expressed as:
  • r t+1 represents the reward at time t+1
  • ⁇ represents the discount factor, which is considered from the perspective of algorithm convergence
  • s t , st t+1 represent the state at time t and t+1 respectively
  • a t+1 represents the actions output at time t and t+1 respectively
  • the update of the Q value is:
  • ⁇ represents the learning rate, which will have a greater impact on the training speed and learning effect, and the symbolic meanings of other items are the same as formula (11).
  • the loss function L( ⁇ ) is defined as the square of the difference between the evaluation network Q value and the target network Q′ value, and the training process of the DDQN algorithm is to minimize the loss L( ⁇ ) to a certain
  • the process of a certain value is expressed as:
  • the Q network parameters ⁇ are updated with gradient descent on the loss function L( ⁇ ):
  • ⁇ -greedy The exploration and application of deep reinforcement learning is a problem that needs to be weighed. It is necessary to avoid too much exploration and too much application.
  • a greedy algorithm ( ⁇ -greedy) is used to balance the exploration and application of the agent, that is, to randomly select the execution action with the probability of ⁇ , and to select the action corresponding to the maximum Q value in the current state with the probability of (1- ⁇ ) , it is necessary to define an appropriate initial value of ⁇ at the beginning of training, and the value of ⁇ to terminate the exploration.
  • the initial value and termination value of ⁇ will greatly affect the training effect and convergence speed, so it needs to be carefully set and adjusted in relevant code files such as Matlab m files during training.
  • the DDQN algorithm belongs to the off-policy deep reinforcement learning algorithm, that is, the generation of experience samples has nothing to do with the current strategy, so it can be considered to learn from past experience samples to improve learning efficiency. Generally, experience playback skills are introduced. At the same time, in order to reduce the correlation between samples, empirical samples are drawn by random sampling.
  • the size of the experience playback pool is set to D in the m-file of Matlab, and the minimum number of batch samples is N*(st t , a t , r t , st t+1 ).
  • the process of experience playback is as follows: first define the size of the experience pool as D, and store a sample array (s t , a t , r t , s t+1 ) in the experience pool at each time step, when a certain After a large number of samples, the agent randomly selects a small batch of sample arrays N*(s t , a t , r t , s t+1 ) from the experience pool to learn from past samples.
  • the specific process is: input the state st and action at into the actual evaluation Q network to estimate Q value, while s t+1 is input to the target Q network to obtain the target Q value Q′, and is added to r t to calculate the root mean square error with the Q value estimated by the evaluation Q network. If the error is large, then It shows that more parameter updates are needed to reduce the error.
  • experience replay is uniformly distributed sampling, that is, all experience samples have the same probability of being sampled.
  • a technique of priority experience playback which is an improvement of the original experience playback technique.
  • Priority experience replay is mainly to increase the probability of sampling valuable (that is, to obtain greater rewards) samples, and to preferentially extract the most valuable samples for learning, so as to learn more efficiently and quickly. Therefore, in order to carry out the operation of priority experience playback, it is necessary to first measure the value of experience samples, which can be judged by TD error, and can be obtained from the update formula of Q value:
  • ⁇ t represents the TD error value
  • symbols and meanings of other items are the same as formula (11).
  • One of the optimization goals of the DDQN algorithm is to make the TD error as small as possible. If the TD error is large, it means that the current Q function is far from the target Q function, and more parameter updates are needed to reduce the TD error. Use TD error to measure the value of experience. In addition, in order to avoid over-fitting of the neural network, experience samples are also drawn by random probability to ensure that even samples with a reward of 0 have a probability of being drawn, so that the priority probability value of each sample experience is:
  • p i
  • ⁇ is a small positive number
  • the purpose is to prevent the sample with reward 0 from being drawn with a probability of 0.
  • the probability calculation of priority experience playback is also set in related code files such as Matlabm files.
  • Matlabm files Before training starts, some key parameters need to be initialized in related code files such as Matlabm files, such as learning rate (generally less than 1), number of training rounds (generally within 1000 rounds), maximum time or period of each round T, is generally equal to the time of the training data set, the time interval of data collection Ts (0.1s-1s) and the time step of training, generally equal to (per cycle T)/(time interval of data collection Ts).
  • learning rate generally less than 1
  • number of training rounds generally within 1000 rounds
  • maximum time or period of each round T is generally equal to the time of the training data set, the time interval of data collection Ts (0.1s-1s) and the time step of training, generally equal to (per cycle T)/(time interval of data collection Ts).
  • SOC state of lithium battery and super capacitor
  • the Drive cycle source module in the Simulink model by using its own open source standard driving cycle conditions such as the European standard driving cycle conditions NEDC (new European driving cycle), WLTC (worldwide harmonized light vehicle test procedure cycle ) or reorganize different standard driving cycles into a mixed driving cycle, etc., as a data set based on DDQN energy management method training.
  • NEDC new European driving cycle
  • WLTC worldwide harmonized light vehicle test procedure cycle
  • reorganize different standard driving cycles into a mixed driving cycle etc.
  • the instruction to call the Simulink agent-environment (vehicle) model is set and the DDQN method after the above-mentioned design is completed can start training.
  • the parallel computing that Matlab carries The parallel computing function of the toolbox (parallel computing toolbox) accelerates the training and convergence process of the energy management method based on DDQN, thereby significantly reducing the training time.
  • the specific training process of the energy management method based on DDQN is as follows.
  • the required power is calculated according to the standard driving cycle conditions used for training and the selected relevant vehicle states are input to the intelligent In the body, the input state observations will be input to the evaluation network that stores in the experience pool and estimates the Q value respectively.
  • the network selects the output action a t to the environment (vehicle) according to the existing method and the ⁇ -greedy principle.
  • the state of the environment (vehicle) changes from st to st +1 .
  • the environment (vehicle) immediately feeds back the reward of the corresponding action a t to the agent’s experience pool according to the reward function.
  • the network parameter ⁇ of the evaluation Q network will be copied to the target Q network at a specific frequency and become the network parameter ⁇ ′ of the target Q network. Extract a small batch of samples from the evaluation Q network and input them into the evaluation Q network and the target Q network respectively, and the target Q network outputs the target Q value corresponding to the next moment st +1 , which is added to the reward r t in the experience sample, and then Calculate the root mean square value together with the estimated Q value, which is the loss L of the network (formula (13)), and calculate the partial differential of the loss L to the network parameter ⁇ It is the loss gradient, and this value is fed back to the estimated Q network. According to the principle of minimizing the loss, the estimated Q network will continuously update the network parameters ⁇ , so that the output action can obtain the maximum cumulative reward. This process will always be in each The training epochs are repeated until eventually convergence.
  • the trained energy management method based on DDQN can be passed through the code
  • the instruction generates a trained and ready-to-simulate control method in the RL agent module in the environment (vehicle) model in Simulink.
  • the rule-based method is designed using Simulink/State flow and integrated with the DDQN method to form a hybrid energy management method.
  • the provided hybrid vehicle energy management method based on rules and double-depth Q-networks includes: Step S510, detecting the energy source of the vehicle equipped with a composite energy storage system , the composite energy storage system includes a lithium battery energy source and a supercapacitor energy source; step S520, when the lithium battery is in a preset normal constrained operating range, use the trained deep reinforcement learning model to determine the output power of the lithium battery, A rule-based approach is used to protect lithium batteries when they are not within the normal constrained operating range.
  • the deep reinforcement learning model is the double-depth Q network designed and trained according to the above.
  • the control logic of the proposed hybrid vehicle energy management method is: when the lithium battery is in the preset normal constraint operating range (SOC _min ⁇ SOC batt ⁇ SOC _max ), use the dual-depth Q network energy management method, and when the SOC of the lithium battery exceeds the normal constraint range (SOC ⁇ SOC _min &SOC>SOC _max ), the rule-based method is used to protect the lithium battery to avoid overcharging and overcharging of the lithium battery Put it in order to prolong the service life of the lithium battery.
  • the lithium battery SOC batt When the lithium battery SOC batt is lower than the set lower limit SOC_min , the lithium battery stops discharging and only accepts charging; then judge whether the supercapacitor SOC sc is higher than the lower limit SOC c_min , if If it is lower than the power requirement of the vehicle, the supercapacitor will be discharged for a short time; if it is lower than that, the driver can only be reminded to stop driving and find a charging pile to charge as soon as possible; 2) If the SOC batt of the lithium battery is higher than the upper limit SOC_max , then Make the lithium battery no longer charge, only discharge; judge whether the supercapacitor SOC sc is higher than the upper limit SOC c_max at this time, if it is higher, then give up the braking energy recovery; if it is lower, the supercapacitor absorbs all the braking Energy recovery power.
  • the hybrid energy management method based on rules and double-depth Q-networks utilizes dynamic programming optimal results and priority experience playback techniques to achieve the following advantages: it does not depend on known conditions (vehicle speed, road conditions, etc.) ) and the existing model; can achieve faster convergence speed, better optimization effect; can achieve a more comprehensive control range, for example, lithium battery SOC has a corresponding control method between [0,1] for energy Management, effectively solve the problem of sudden reduction in service life of lithium batteries due to frequent overcharging and overdischarging.
  • the present invention can be a system, method and/or computer program product.
  • a computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present invention.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • flash memory static random access memory
  • SRAM static random access memory
  • CD-ROM compact disc read only memory
  • DVD digital versatile disc
  • memory stick floppy disk
  • mechanically encoded device such as a printer with instructions stored thereon
  • a hole card or a raised structure in a groove and any suitable combination of the above.
  • computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
  • Computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, local area network, wide area network, and/or wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, Python, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect).
  • LAN local area network
  • WAN wide area network
  • an electronic circuit such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA)
  • FPGA field programmable gate array
  • PDA programmable logic array
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
  • each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by means of hardware, implementation by means of software, and implementation by a combination of software and hardware are all equivalent.

Landscapes

  • Engineering & Computer Science (AREA)
  • Power Engineering (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Sustainable Development (AREA)
  • Sustainable Energy (AREA)
  • Electric Propulsion And Braking For Vehicles (AREA)

Abstract

本发明公开了一种基于规则和双深度Q网络的混合动力汽车能量管理方法。该方法包括：检测设有复合储能系统的车辆能量源，该复合储能系统包括锂电池和超级电容；当锂电池在预设的正常约束工作范围时，利用经训练的深度强化学习模型确定锂电池的输出功率，当锂电池没有处于正常约束工作范围时，则使用设定规则对锂电池进行保护，其中深度强化学习模型的智能体包括评估Q网络和目标Q网络，环境的状态观测量是锂电池的剩余电量、超级电容的剩余电量以及整车需求功率，锂电池输出功率作为输出动作，并以最小化复合储能系统的能量损失作为目标，设置奖励函数。利用本发明能够实现能量的全范围管理、最优性和实时性的有机统一。

Description

基于规则和双深度Q网络的混合动力汽车能量管理方法 技术领域
本发明涉及车辆能量管理技术领域，更具体地，涉及一种基于规则和双深度Q网络的混合动力汽车能量管理方法。
背景技术
随着人工智能和计算机技术的快速发展，大量的机器学习算法已被应用到实际生活中，其中以深度强化学习算法为代表的机器学习更是受到了关注，并被广泛用于如机器视觉、自动驾驶、机器人控制、智能交通等领域。近年来，深度强化学习被逐渐应用于混合动力汽车能量管理研究中，并取得了较好的效果。双深度Q网络算法(double deep Q-network，双深度Q网络，简称DDQN)也属于深度强化学习算法，无需先验知识或模型已知，经过恰当的奖励函数、超参数设计和学习训练过程即可获得较理想的优化控制效果。
在碳达峰、碳中和的目标驱动下，在节能减排和可持续发展的时代要求下，可有效实现节能减排的新能源汽车，在各国得以大力发展，是未来技术和行业发展的主要方向。然而拥有不同工作性质能量源的新能源汽车亟需解决能量分配的问题，以达到节能、提高系统效率和延长主能量源使用寿命等目标。新能源汽车包括混合动力汽车、电动汽车和燃料电池汽车，能量管理问题是当前新能源汽车亟待解决的控制优化问题之一。
对于混合动力汽车，其能量管理问题是一个时变、复杂、非线性的决策控制问题，而现有的基于深度强化学习的能量管理方法，存在优化效果欠佳、实时性不足和控制范围不够全面等缺陷。
在现有技术中，针对新能源汽车的能量管理问题，研究者们提出了许多能量管理方法，例如包括基于规则的能量管理方法、基于优化的能量管理方法和基于学习的能量管理方法。其中基于规则的能量管理方法包括确定性规则控制、模糊逻辑控制等，其实时性较强，但难以达到最优的控制效果；基于优化的能量管理方法虽可得到较优的控制效果，但要求工况预知且计算量大，难以实时应用。
因此，基于规则和基于优化的能量管理方法难以同时实现实时性和最优性的统一，近年来，可实现最优性和实时性权衡的基于(机器)学习的能量管理方法受到大量研究者的关注，主要包括基于深度学习(deep learning)、强化学习(reinforcement learning)和深度强化学习(deep reinforcement learning)的能量管理方法，特别地，以基于深度强化学习为代表的能量管理方法，在计算机技术快速发展背景下成为新的研究热点。
现有基于深度强化学习的方法，尤其是基于传统深度Q网络(deep Q-network，深度Q网络，简称DQN)的能量管理方法，由于动作选择和策略评估均在同一个神经网络进行，容易导致Q值的过高估计而使得控制效果欠佳，同时还具有收敛速度慢、实时性差等缺陷，而新能源汽车在实际的运行过程中，对于控制方法的实时性和工况适应性要求较高，故需设计既能满足实际应用需求又具有优秀控制效果的能量管理方法。
此外，目前的基于深度强化学习的能量管理方法，往往是在所预设的约束范围内，如锂电池SOC(state of charge，当前剩余电量)不高于0.9且不低于0.3实现控制，然而在实际行车时，相关状态量并不能一直保持在约束范围内，并且，若锂电池经常过充、过放电会导致其使用寿命迅速衰减，进而显著减少新能源汽车的续航里程、增加使用成本，因此，需要额外制定在约束范围外的控制方法，以获得更全面、更稳定的控制方法，以在尽可能节能的同时尽量延长锂电池的使用寿命，从而降低新能源汽车的使用成本，有利于新能源汽车的大规模推广。
发明内容
本发明的目的是克服上述现有技术的缺陷，提供一种基于规则和双深度Q网络的混合动力汽车能量管理方法，是通过改进深度Q网络，实现最优性、实时性和工况适应性综合的新技术方案，适用于具有不同工作特性的多能量源混合动力汽车的能量管理。
本发明的技术方案是，提供一种基于规则和双深度Q网络的混合动力汽车能量管理方法。该方法包括以下步骤：
检测设有复合储能系统的车辆能量源，该复合储能系统包括锂电池能 量源和超级电容能量源；
在检测到锂电池处于预设的正常约束工作范围的情况下，利用经训练的深度强化学习模型确定锂电池的输出功率，在检测到锂电池没有处于正常约束工作范围的情况下，则使用设定规则对锂电池进行保护；
其中，对于所述深度强化学习模型，其智能体包括评估Q网络和目标Q网络，环境的状态观测量是锂电池的剩余电量、超级电容的剩余电量以及整车需求功率，锂电池输出功率作为输出动作，并以最小化所述复合储能系统的能量损失作为目标，设置相应的奖励函数。
与现有技术相比，本发明的优点在于，为在节能的同时尽量延长锂电池等的使用寿命从而有效降低新能源汽车的使用成本，并同时兼顾控制方法的实时性和最优性，本发明应用前沿的双深度Q网络算法，与基于规则的能量管理方法结合，形成适用于混合动力汽车的能量管理方法，能够实现全范围控制、最优性和实时性的有机统一
通过以下参照附图对本发明的示例性实施例的详细描述，本发明的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本发明的实施例，并且连同其说明一起用于解释本发明的原理。
图1是根据本发明一个实施例的电动汽车复合储能系统动力架构；
图2是根据本发明一个实施例的基于双深度Q网络的能量管理方法的原理图；
图3是根据本发明一个实施例的基于规则和双深度Q网络的能量管理方法的控制逻辑示意图；
图4是根据本发明一个实施例的基于规则和双深度Q网络的混合能量管理方法的过程示意图；
图5是根据本发明一个实施例的基于规则和双深度Q网络的混合动力汽车能量管理方法的流程图。
具体实施方式
现在将参照附图来详细描述本发明的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本发明的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本发明及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
在进行能量管理方法的设计时，通常需要先搭建相关车辆的模型，这些模型包括车辆动力学模型，电机模型、能量源模型和传动系统模型等。在本发明实施例中，基于准静态准则及车辆动力学利用相关建模软件如Matlab/Simulink建立相关模型。应理解的是，除Matlab外，也可采用其他的计算程序或工具实现本发明。
深度强化学习由3要素组成：环境、智能体和奖励函数。在混合动力汽车能量管理问题中，环境为对应的车辆模型或无模型，智能体则为执行相关控制的待训练方法，而奖励函数需要针对具体问题设置，设置的好坏会影响智能体学习的效果，即训练、收敛过程。同时需特别注意稀疏奖励的问题，即大量的执行动作没有获得奖励，在具体设置时需要根据实际训练情况不断调整奖励函数的各项和相关系数。
此外，对于智能体的状态观测量和输出动作的属性及维数都需要进行恰当的设置，这也会较大的影响学习训练效果。由于DDQN的方法需要对输出的动作进行离散化处理，故存在着动作间隔、输出动作的区间设置等问题。并且，DDQN的能量管理方法设计涉及神经网络、状态观测量、动作量和奖励函数的设计、以及一些超参数的设定等。特别地，对于神经网络的设 计，涉及到网络层数、神经元个数、连接方式和激活函数等的选择和设计。DDQN算法使得当前动作选择和策略评估分开，进而可降低过于乐观的方法估计以提高控制效果，可以说其是对DQN算法的改进。对于智能体最优控制动作区间和各动作间隔的选择，也需要仔细的设置以获得理想的收敛速度和控制效果，本发明将通过融入动态规划的最优结果进行动作的设置。
在深度强化学习中，为提高学习效率，会引入经验回放的技巧，其通过存储过去的经验样本数组，在学习训练时智能体通过随机抽样(减少样本之间的关联性)从过去的经验中进行学习。然而在实际训练中，很多样本获得的是较低奖励甚至没有奖励，那么这些样本的学习参考价值就不大，为进一步提高学习效率，加速学习收敛过程，在经验回放技巧的基础上提出了优先经验回放的技巧，通过优先抽取有较大价值(即能获得较大奖励)的样本进行学习，从而实现更快速的收敛，本发明实施例引入优先经验回放的技巧以加速DDQN算法的收敛，增强方法的实时性。
在下文中，以电动汽车复合储能系统为例，详细说明本发明的方案，图1是电动汽车复合储能系统的动力传动系统架构，该复合储能系统包括锂电池和超级电容两个不同的能量源，二者均可作为能量源进行充放电，即驱动车辆行驶或吸收制动能量，超级电容作为辅助能量源，目的是降低锂电池的充放电频率以及放电电流大小。锂电池和超级电容的协同工作需制定恰当的能量管理方法以合理地分配二者的输入输出功率。特别地，DCDC转换器负责升高超级电容的输出电压或降低来自母线的电压，从而降低对超级电容配置的要求。
为模拟实际电动汽车复合储能系统的状态变化，在相关建模软件如Matlab/Simulink中按照准静态原则和车辆动力学建立车辆的模型，这些模型包括：需求功率计算模型，锂电池等效电路模型、超级电容等效电路模型、DCDC转换器效率模型、电机逆变器总成效率模型、传动模型等，模型建立完成之后，经过仿真无误后即完成搭建该车辆模型。
简言之，本发明提供的基于规则和双深度Q网络的混合动力汽车能量管理方法，在正常锂电池SOC约束范围，使用基于DDQN的能量管理方法，而在约束范围之外，则使用基于规则的能量管理方法进行控制。以下将具体 介绍基于DDQN的能量管理方法、基于规则和双深度Q网络的能量管理方法。
一、设计基于DDQN的能量管理方法。
如图2所示，环境和智能体的交互过程为：每个时间步长，在环境状态s t下，智能体从已设定好的动作集中随机选取一个动作输出到环境中，环境的状态随即从s t转变为s t+1，与此同时，根据所设计的奖励函数将该动作相应的奖励反馈给智能体，智能体根据追求最大累计奖励的原则，不断地调整选取的输出动作以获得最大的累计奖励，该过程将一直重复至奖励函数收敛。DDQN算法通过将动作选择和策略评估分开，避免陷入过高估计Q值而影响收敛速度和控制效果。优选地，本发明还加入优先经验回放的技巧加速训练学习过程，即增大重要的、有价值的样本被抽取的频率，这可有效促使算法训练、收敛时间进一步缩短，有利于该算法的实际推广和应用。
总的来说，在一个实施例中，设计的基于DDQN的能量管理方法包括2部分，一部分是在Simulink中搭建的车辆模型以及联合Simulink强化学习工具箱(reinforcement learning toolbox)RL agent模块搭建的闭环智能体-环境(车辆)模型；另一部分是针对基于DDQN能量管理方法，设计神经网络和训练指令等，例如，建立Matlab的m文件来实现。
具体地，基于DDQN的能量管理方法设计过程如下。
1、确定关键元素
1)状态向量：
将锂电池的SOC b、超级电容SOC sc和整车需求功率P dem作为环境的状态观测量，表示为：
Figure PCTCN2021137803-appb-000001
其中，S表示状态观测量集合，s(t)表示t时刻的状态观测量。
为缩小计算量，加快训练速度，优选地，对整车需求功率P dem进行归一化处理，即把需求功率P dem缩小到[-1,1]之间，计算其算术平均值mean(x)和标准差std(x)，按照标准归一化通用公式计算，表示为：
Figure PCTCN2021137803-appb-000002
其中，mean(x)和std(x)分别表示输入状态数据的算术平均值和标准差。 算术平均值mean(x)和标准差std(x)的计算如下：
Figure PCTCN2021137803-appb-000003
Figure PCTCN2021137803-appb-000004
2)动作向量：
为加快基于DDQN的能量管理方法的收敛速度以及增强方法的控制效果，优选地，以基于动态规划的全局最优控制结果来对智能体输出动作的可行动作输出区间进行设定，将锂电池输出功率P batt定义为输出动作。由于DDQN算法需要对输出动作进行离散化处理，即将一个可行的动作区间分成n份。而基于优化的能量管理方法中，动态规划的控制结果具有全局全优的优点，且其结果也是一个离散的最优控制动作序列，故参考相同优化目标和控制对象的基于动态规划能量管理方法的最优控制动作序列结果，恰当地将输出动作区间设置为n份，即动作间隔为
Figure PCTCN2021137803-appb-000005
P batt_max和P batt_min分别表示锂电池输出功率的最大值和最小值。
Figure PCTCN2021137803-appb-000006
其中，A表示输出动作集合，a(t)表示t时刻输出的动作。
3)奖励函数：
在一个实施例中，为了提高系统效率，将最小化复合储能系统的能量损失作为优化目标，由此出发，设置相应的奖励函数r(t)，该奖励函数通过在Simulink中建立相应的数学模型实现，例如表示为：
Figure PCTCN2021137803-appb-000007
E loss＝L sc+L batt+L dcdc(7)
其中，E loss表示复合储能系统整体的能量损耗；SOC b_tgt表示锂电池SOC b的目标值；SOC sc_tgt表示超级电容SOC sc的目标值；L sc，L batt，L dcdc分别表示超级电容、锂电池和DCDC转换器的能量损失；m表示能量损失项的系数，n和p分别表示平衡锂电池和超级电容的SOC变化的系数，这三个系数需在训练过程中进行调整。
进一步地，能量管理优化问题需在一定的约束下寻找最优解，例如约束条件表示为：
Figure PCTCN2021137803-appb-000008
其中，I b为锂电池的电流，I sc为超级电容的电流，I b_min和I b_max分别表示锂电池电流的最小和最大值；I sc_min和I sc_max分别表示超级电容电流的最小和最大值，P dem表示整车需求功率，P min和P max分别表示车辆需求功率的最小和最大值。
2、设计神经网络
DDQN的神经网络涉及到结构、层数、神经元个数和激活函数的设计和选择，均需根据实际训练情况进行恰当的设置和调整。例如，通过在Matlab的m文件编写代码以调用神经网络相关函数的方式设计DDQN的神经网络，并利用全连接方式进行神经网络各层之间的连接，对于各层网络神经元个数(几十个至几百个不等)以及层数(通常为几层)，则需要根据实际数据量和训练结果进行设计，而中间层的激活函数例如设置为线性整流函数ReLU，网络输出层的激活函数则设置为tanh，使得输出值约束到[-1,1]之间。这两种激活函数的原理如下：
线性整流函数(ReLU)
Figure PCTCN2021137803-appb-000009
tanh激活函数
Figure PCTCN2021137803-appb-000010
3、DDQN算法的更新原理
首先，定义DDQN评估Q网络的参数为θ，目标Q网络的参数则为θ′，出于收敛和增强训练效果的考虑，在训练过程中，评估Q网络的网络参数θ会每隔一定的时间步长复制给目标Q网络θ′。强化学习和深度强化学习均是基于贝尔曼原理的算法，DDQN算法Q值的计算方式表示为：
Figure PCTCN2021137803-appb-000011
其中，
Figure PCTCN2021137803-appb-000012
表示期望；r t+1表示t+1时刻的奖励；γ表示折扣因子，是出于 算法收敛的角度考虑；s t，s t+1分别表示t和t+1时刻的状态，a t，a t+1分别表示t和t+1时刻输出的动作；
Figure PCTCN2021137803-appb-000013
表示在状态动作对(s t,a t)时由网络参数θ′估计的在s t+1采取动作a t+1的Q值。
根据DDQN算法的机制，Q值的更新为：
Figure PCTCN2021137803-appb-000014
其中，η表示学习率，会对训练速度及学习效果有较大的影响，其他项的符号意义同公式(11)。
为使得算法收敛，在一个实施例中，定义损失函数L(θ)为评估网络Q值和目标网络Q′值的差值的平方，DDQN算法的训练过程即最小化损失L(θ)至某一定值的过程，表示为：
Figure PCTCN2021137803-appb-000015
其中，
Figure PCTCN2021137803-appb-000016
表示由之前的网络参数θ′估计得出的目标Q值；Q(s t,a t,θ)则表示由网络参数θ估计的Q值。其他项的符号意义同公式(11)，r t表示t时刻的奖励。
Q网络参数θ在损失函数L(θ)上以梯度下降进行更新：
Figure PCTCN2021137803-appb-000017
其中，
Figure PCTCN2021137803-appb-000018
表示对网络参数θ的梯度，其他项的符号意义同公式(11)和(13)。
4、探索和应用的算法
深度强化学习的探索和应用是一个需要权衡的问题，既要避免过多的探索也要避免过多的应用。在一个实施例中，利用贪心算法(ε-greedy)平衡智能体的探索和应用，即以ε的概率随机选择执行动作，以(1-ε)的概率选择当前状态下对应最大Q值的动作，在训练开始时需先定义一个恰当的ε初始值，以及终止探索的ε值。ε的初始值和终止值会较大地影响训练效果和收敛速度，故需要在训练时在相关代码文件如Matlab m文件中仔细地进行设置、调整。
5、设计优先经验回放
DDQN算法属于离策略(off-policy)的深度强化学习算法，即经验样 本的生成与当前策略无关，故可考虑从过去的经验样本中进行学习以提高学习效率，一般会引入经验回放的技巧，同时为减少样本之间的关联度以随机抽样的方式抽取经验样本。在一个实施例中，在Matlab的m文件中设定经验回放池的大小为D，最小批采样数为N*(s t,a t,r t,s t+1)。经验回放的流程为，先定义经验池的大小为D，在每个时间步长，在经验池中存储一个样本数组(s t,a t,r t,s t+1)，当存储了一定量的样本后，智能体从经验池中随机抽取一小批样本数组N*(s t,a t,r t,s t+1)，以便从过去的样本中进行学习。对于经验样本数组(s t,a t,r t,s t+1)的利用，如图2所示，具体过程是：将状态s t、动作a t输入到实际的评估Q网络以估算出Q值，而s t+1则输入到目标Q网络得到目标Q值Q′，并与r t相加后与评估Q网络估计的Q值进行均方根误差的计算，若误差较大，则说明还需要进行较多次的参数更新，以减少误差。
然而，经验回放是均匀分布采样，即所有经验样本被采样到的概率相同，为进一步提高有价值样本(s t,a t,r t,s t+1)的采样概率，进而提高学习效率，加速算法的收敛过程，优选地，引入优先经验回放的技巧，其是原有经验回放技巧的改进。优先经验回放主要是增大有价值(即能获得较大奖励)样本被采样的概率，优先抽取最有价值的样本进行学习，从而可更高效更快速地学习。因此，要进行优先经验回放的操作，需要首先衡量经验样本的价值，可通过TD error来判断，由Q值的更新公式可得到：
Figure PCTCN2021137803-appb-000019
其中，δ t表示TD error值，其他项的符号意义同公式(11)。
DDQN算法的优化目标之一是让TD error尽可能小，若TD error较大，则说明当前的Q函数距离目标的Q函数差距较大，需要更多的进行参数更新来减小TD error，因此用TD error来衡量经验的价值。此外，为了避免神经网络过拟合，还通过随机概率的方式抽取经验样本，以保证即使是奖励为0的样本也有概率被抽取到，令每个样本经验的优先概率值为：
Figure PCTCN2021137803-appb-000020
其中，p i＝|δ t|+∈>0，∈是一个很小的正数，目的是防止奖励为0的样本被抽取到的概率为0。优先级因子λ决定优先经验回放的程度，该值需恰当的设置以达到较好的训练效果，当λ＝0时，表示不采用优先经验回放， 使用正常的经验回放进行训练。同时还需要对p i做出限制，以保证无论极值是何值时采样的稳定性，优先经验回放的概率计算也在相关代码文件如Matlabm文件中进行设置。
6、设置其他关键参数
在训练开始前，还需要在相关代码文件如Matlabm文件中初始化一些关键的参数，例如包括学习率(一般小于1)、训练的回合数(一般在1000回合以内)、每回合的最大时间或周期T，一般等于训练数据集的时间、数据采集的时间间隔Ts(0.1s-1s)和训练的时间步长，一般等于(每回合周期T)/(数据采集的时间间隔Ts)。为避免出现锂电池和超级电容的状态(如SOC)超过约束的范围时仍处于训练的情况，还需要设置每回合训练的终止条件，以重新开始下一回合的训练并使得所训练得到方法符合预期。
7、选取训练数据
例如，通过在Simulink模型中使用Drive cycle source模块并从其自带的开源标准驾驶循环工况中选取如欧洲标准驾驶循环工况NEDC(new European driving cycle)、WLTC(worldwide harmonized light vehicle test procedure cycle)或将不同的标准驾驶循环重新有机组合在一起成为混合驾驶循环工况等，作为基于DDQN能量管理方法训练的数据集。
8、基于DDQN能量管理方法的训练过程
在相关代码文件如Matlabm文件中设置调用Simulink智能体-环境(车辆)模型的指令并经上述设计完成之后的DDQN方法即可开始训练，在一个实施例中，还通过使用Matlab自带的并行计算工具箱(parallel computing toolbox)的并行计算功能加速基于DDQN的能量管理方法的训练、收敛过程，从而显著缩短训练时间。
如图2所示，基于DDQN的能量管理方法的具体训练过程如下，在每个时间步长，根据用来训练的标准驾驶循环工况计算得出需求功率以及选定的相关车辆状态输入到智能体中，其中输入的状态观测量分别会输入到经验池中存储以及估计Q值的评估网络中，该网络按照已有方法并依据ε-greedy原则选择输出动作a t到环境(车辆)中，一方面，环境(车辆)的状态随之由s t变成s t+1，另一方面，环境(车辆)则根据奖励函数将相应动 作a t的奖励立即反馈到智能体的经验池中。评估Q网络的网络参数θ会以一个特定的频率复制到目标Q网络中，成为目标Q网络的网络参数θ′，当经验池中存储了一定的经验样本后，利用优先经验回放技巧从经验池中提取一小批量的样本分别输入到评估Q网络和目标Q网络中，目标Q网络输出下一时刻s t+1对应的目标Q值，其与经验样本中的奖励r t相加后，再与估计的Q值一同计算均方根值，即为网络的损失L(公式(13))，计算该损失L对网络参数θ的偏微分
Figure PCTCN2021137803-appb-000021
即为损失梯度，将该值反馈到估计Q网络中，依据最小化损失的原则评估Q网络会不断地更新网络参数θ，以使得输出的动作能获得最大累计奖励，该过程将一直在每个训练回合重复，直至最终收敛。
二、设计基于规则和双深度Q网络的能量管理方法
参见图3的基于规则和双深度Q网络的混合能量管理方法的控制逻辑以及图4的混合能量管理方法的设计流程，在本实施例中，训练完成的基于DDQN的能量管理方法，可通过代码指令在Simulink中的环境(车辆)模型中的RL agent模块中生成已训练完成并可随时仿真使用的控制方法。此外，例如利用Simulink/State flow设计基于规则的方法，并将其与DDQN的方法整合到一起，形成混合能量管理方法。
在完成上述设计后，在实际应用中，参见图5所示，所提供的基于规则和双深度Q网络的混合动力汽车能量管理方法包括：步骤S510，检测设有复合储能系统的车辆能量源，该复合储能系统包括锂电池能量源和超级电容能量源；步骤S520，当锂电池在预设的正常约束工作范围的情况下，利用经训练的深度强化学习模型确定锂电池的输出功率，当锂电池没有处于正常约束工作范围时，则使用基于规则的方法对锂电池进行保护。其中深度强化学习模型即根据上文设计和训练的双深度Q网络。
具体地，结合图3所示，所提出的混合动力汽车能量管理方法的控制逻辑为：当锂电池在预设的正常约束工作范围(SOC _min≤SOC batt≤SOC _max)时，使用基于双深度Q网络的能量管理方法，而当锂电池的SOC超出正常的约束范围时(SOC<SOC _min&SOC>SOC _max)，则使用基于规则的方法对锂电池进行保护，从而避免锂电池的过充过放，以延长锂电池的使用寿命。具体包括：1)当锂电池SOC batt低于设定的下限值SOC _min，令锂电池停止放 电，只接受充电；再判断超级电容SOC sc是否高于限制的下限值SOC c_min，若高于，则令超级电容根据车辆需求功率短时放电；若低于，则只能提醒驾驶员停止行驶，尽快找充电桩充电；2)若锂电池的SOC batt高于上限值SOC _max，则令锂电池不再充电，只进行放电；判断此时超级电容SOC sc是否高于限制的上限SOC c_max，若高于，则放弃制动能量回收；若低于，则超级电容吸收全部的制动能量回收功率。
综上所述，本发明提供的基于规则和双深度Q网络的混合能量管理方法，利用动态规划最优结果和优先经验回放技巧，实现了以下优势：不依赖于已知条件(车速、路况等)和已有的模型；可实现更快的收敛速度，更好的优化效果；可实现更全面的控制范围，例如锂电池SOC在[0,1]之间都有相应的控制方法进行能量的管理，有效解决锂电池由于频繁过充电、过放电而导致使用寿命骤减的问题。
本发明可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本发明的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无 线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本发明操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++、Python等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本发明的各个方面。
这里参照根据本发明实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本发明的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算 机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本发明的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本发明的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本发明的范围由所附权利要求来限定。

Claims (10)

  1. 一种基于规则和双深度Q网络的混合动力汽车能量管理方法，包括以下步骤：
    检测设有复合储能系统的车辆能量源，该复合储能系统包括锂电池能量源和超级电容能量源；
    在检测到锂电池处于预设的正常约束工作范围的情况下，利用经训练的深度强化学习模型确定锂电池的输出功率，在检测到锂电池没有处于正常约束工作范围的情况下，则使用设定规则对锂电池进行保护；
    其中，对于所述深度强化学习模型，其智能体包括评估Q网络和目标Q网络，环境的状态观测量是锂电池的剩余电量、超级电容的剩余电量以及整车需求功率，锂电池输出功率作为输出动作，并以最小化所述复合储能系统的能量损失作为目标，设置相应的奖励函数。
  2. 根据权利要求1所述的方法，其特征在于，将所述深度强化学习模型的输出动作区间设置为n份，表示为：
    Figure PCTCN2021137803-appb-100001
    其中，动作间隔为
    Figure PCTCN2021137803-appb-100002
    P batt_max和P batt_min分别表示锂电池输出功率的最大值和最小值，A表示输出动作集合，a(t)表示t时刻输出的动作。
  3. 根据权利要求1所述的方法，其特征在于，在训练所述深度强化学习模型过程中，包括通过随机概率的方式抽取经验样本，每个样本经验的优先概率值表示为：
    Figure PCTCN2021137803-appb-100003
    其中，p i＝|δ t|+ε>0，ε是设定的正数，优先级因子λ用于决定优先经验回放的程度，δ t是经验样本的价值，表示为：
    Figure PCTCN2021137803-appb-100004
    其中，r t+1表示t+1时刻的奖励，γ表示折扣因子，a t，a t+1分别表示t和t+1时刻输出的动作，s t，s t+1分别表示t和t+1时刻的状态观察值，
    Figure PCTCN2021137803-appb-100005
    表示在状态s t+1采取动作a t+1时估计的Q值，Q(s t,a t)表示t时刻的Q值。
  4. 根据权利要求1所述的方法，其特征在于，所述奖励函数设置为：
    Figure PCTCN2021137803-appb-100006
    E loss＝L sc+L batt+L dcdc
    其中，E loss表示复合储能系统整体的能量损耗，SOC b_tgt表示锂电池剩余电量SOC b的目标值，SOC sc_tgt表示超级电容剩余电量SOC sc的目标值，L sc，L batt，L dcdc分别表示超级电容、锂电池和复合储能系统中DCDC转换器的能量损失，m表示能量损失项的系数，n和p分别表示平衡锂电池和超级电容的剩余电量变化的系数；
    利用所述深度强化学习模型求解能量优化问题的约束设置为：
    Figure PCTCN2021137803-appb-100007
    其中，I b为锂电池的电流，I sc为超级电容的电流，I b_min和I b_max分别表示锂电池电流的最小和最大值；I sc_min和I sc_max分别表示超级电容电流的最小和最大值，P dem表示整车需求功率，P min和P max分别表示整车需求功率的最小和最大值，P batt表示锂电池输出功率，P batt_min和P batt_max分别表示锂电池输出功率的最小和最大值。
  5. 根据权利要求1所述的方法，其特征在于，所述使用设定规则对锂电池进行保护包括以下步骤：
    当锂电池剩余电量SOC batt低于设定的下限值SOC _min时，令锂电池停止放电，只接受充电；并判断超级电容剩余电量SOC sc是否高于限制的下限值SOC c_min，若高于，则令超级电容根据车辆需求功率短时放电；若低于，则提醒驾驶员停止行驶；
    若锂电池的剩余电量SOC batt高于上限值SOC _max，则令锂电池不再充电，只进行放电；并判断超级电容剩余电量SOC sc是否高于限制的上限SOC c_max，若高于，则放弃制动能量回收；若低于，则超级电容吸收全部的制动能量回收功率。
  6. 根据权利要求1所述的方法，其特征在于，所述状态观测量表示为：
    Figure PCTCN2021137803-appb-100008
    其中，S表示状态观测量集合，s(t)表示t时刻的状态观测量，SOC b表示锂电池的剩余电量，SOC sc表示超级电容的剩余电量，P dem表示整车需求功率且该整车需求功率是经归一化处理的值。
  7. 根据权利要求1所述的方法，其中，训练所述深度强化学习模型的损失函数L(θ)设置为评估网络Q值和目标网络Q′值的差值的平方，表示为：
    Figure PCTCN2021137803-appb-100009
    其中，
    Figure PCTCN2021137803-appb-100010
    表示由之前的网络参数θ′估计得出的目标Q值；Q(s t,a t,θ)则表示由网络参数θ估计的Q值，s t和s t+1分别表示t和t+1时刻的状态，a t和a t+1分别表示t和t+1时刻输出的动作，γ表示折扣因子，r t表示t时刻的奖励。
  8. 根据权利要求1所述的方法，其特征在于，所述深度强化学习模型的训练过程包括：
    在每个时间步长，根据用于训练的标准驾驶循环工况计算得出需求功率和选定的相关车辆状态输入到智能体中，其中输入的状态观测量分别输入到经验池中存储以及评估Q网络中，该评估Q网络依据ε-greedy原则选择输出动作a t到环境中，环境的状态由s t变成s t+1；
    环境根据奖励函数将相应动作a t的奖励反馈到智能体的经验池中，在该过程中，评估Q网络的网络参数θ以设定的频率复制到目标Q网络中，成为目标Q网络的网络参数θ′；
    当经验池中存储一定的经验样本后，从经验池中提取一小批量的样本分别输入到评估Q网络和目标Q网络中，目标Q网络输出下一时刻s t+1对应的目标Q值，基于目标Q值和估计的Q值计算损失值及其对评估Q网络参数的损失梯度，并该值反馈到估计Q网络中，估计Q网络依据最小化损失的原则更新网络参数θ，以使输出的动作获得最大累计奖励。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其中，该程序被处理器执行时实现根据权利要求1至8中任一项所述方法的步骤。
  10. 一种计算机设备，包括存储器和处理器，在所述存储器上存储有 能够在处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现权利要求1至8中任一项所述的方法的步骤。
PCT/CN2021/137803 2021-05-31 2021-12-14 基于规则和双深度q网络的混合动力汽车能量管理方法 Ceased WO2022252559A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110602198.0A CN113511082B (zh) 2021-05-31 2021-05-31 基于规则和双深度q网络的混合动力汽车能量管理方法
CN202110602198.0 2021-05-31

Publications (1)

Publication Number Publication Date
WO2022252559A1 true WO2022252559A1 (zh) 2022-12-08

Family

ID=78065129

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/137803 Ceased WO2022252559A1 (zh) 2021-05-31 2021-12-14 基于规则和双深度q网络的混合动力汽车能量管理方法

Country Status (2)

Country Link
CN (1) CN113511082B (zh)
WO (1) WO2022252559A1 (zh)

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116054285A (zh) * 2022-12-30 2023-05-02 国网湖北省电力有限公司经济技术研究院 一种基于联邦强化学习算法的输配调频资源协同控制方法
CN116050505A (zh) * 2023-02-22 2023-05-02 西南交通大学 一种基于伙伴网络的智能体深度强化学习方法
CN116123702A (zh) * 2023-02-23 2023-05-16 中交武汉智行国际工程咨询有限公司 基于支持向量机和深度强化学习的建筑空调温度控制方法
CN116208545A (zh) * 2023-02-07 2023-06-02 哈尔滨工业大学 网络路由优化方法及装置、智能路由方法及装置
CN116257937A (zh) * 2023-01-05 2023-06-13 东南大学 基于多目标深度强化学习的混合动力汽车生态驾驶方法
CN116384250A (zh) * 2023-04-12 2023-07-04 中国长江三峡集团有限公司 一种综合能源系统多个智能体的协作控制方法与能源系统
CN116373673A (zh) * 2023-03-23 2023-07-04 国网江苏省电力有限公司南通供电分公司 一种电动汽车充放电管理方法及系统
CN116561579A (zh) * 2023-05-17 2023-08-08 四川大学 自适应多工况的递归深度强化学习混动汽车能量管理策略
CN116582860A (zh) * 2023-05-08 2023-08-11 南京航空航天大学 一种基于信息年龄约束的链路资源分配方法
CN116605092A (zh) * 2023-04-26 2023-08-18 中国科学技术大学 一种能量管理方法及能量管理平台
CN116736700A (zh) * 2023-05-16 2023-09-12 武汉理工大学 多动力源两栖艇能量管理控制系统及方法
CN116853073A (zh) * 2023-09-04 2023-10-10 江西五十铃汽车有限公司 一种新能源电动汽车能量管理方法及系统
CN116985646A (zh) * 2023-09-28 2023-11-03 江西五十铃汽车有限公司 车辆超级电容器控制方法、设备和介质
CN116995641A (zh) * 2023-09-26 2023-11-03 北京交通大学 一种应用于轨道交通储能的能量管理架构及方法
CN117021066A (zh) * 2023-05-26 2023-11-10 浙江大学 一种基于深度强化学习的机器人视觉伺服运动控制方法
CN117034443A (zh) * 2023-07-13 2023-11-10 哈尔滨工业大学 基于ddqn算法的机动观测策略生成方法、改进ddqn算法的机动观测策略生成方法
CN117021971A (zh) * 2023-08-15 2023-11-10 北京工业大学 一种多动力源车辆能量管理策略优化方法、系统及设备
CN117058468A (zh) * 2023-10-11 2023-11-14 青岛金诺德科技有限公司 用于新能源汽车锂电池回收的图像识别与分类系统
CN117227700A (zh) * 2023-11-15 2023-12-15 北京理工大学 串联混合动力无人履带车辆的能量管理方法及系统
CN117507789A (zh) * 2023-12-06 2024-02-06 江苏开放大学（江苏城市职业学院） 一种用于agv小车的多模混合动力驱动系统
CN117578679A (zh) * 2024-01-15 2024-02-20 太原理工大学 基于强化学习的锂电池智能充电控制方法
CN117650522A (zh) * 2023-12-04 2024-03-05 北京北交本有科技有限公司 一种基于深度强化学习的多储能系统分层协同控制方法
CN117650553A (zh) * 2023-10-25 2024-03-05 四川大学 基于多智能体深度强化学习的5g基站储能电池充放电调度方法
CN117933666A (zh) * 2024-03-21 2024-04-26 壹号智能科技(南京)有限公司 一种密集仓储机器人调度方法、装置、介质、设备及系统
CN117944527A (zh) * 2024-02-02 2024-04-30 山东德维鲁普新材料有限公司 基于氢燃料电池发电车的能量管理方法
CN118544901A (zh) * 2024-06-05 2024-08-27 郑州大学 一种基于双q学习与实时速度预测的氢-电混动系统能量管理方法、系统、设备及存储介质
CN118596882A (zh) * 2024-06-11 2024-09-06 徐州徐工重型车辆有限公司 纯电动矿用自卸车分布式线控轮边驱动系统及驱动方法
CN118705133A (zh) * 2024-06-15 2024-09-27 曲阜师范大学 一种基于深度强化学习的风力机转速参考值确定方法
CN118859730A (zh) * 2024-09-25 2024-10-29 中南大学 自适应感知的重载列车群组运行能量管理方法及控制装置
CN118928165A (zh) * 2024-10-12 2024-11-12 国网浙江省电力有限公司嘉善县供电公司 氢燃料电池车的多电源协调控制方法、系统、设备及介质
CN119187463A (zh) * 2024-10-30 2024-12-27 扬州市江都区飞跃泵业有限公司 一种双叶片半开式污水泵叶轮及其制造装置、方法
CN119359063A (zh) * 2024-09-09 2025-01-24 华北电力大学 电动汽车充放电调度策略生成方法
CN119675213A (zh) * 2025-02-20 2025-03-21 南昌科晨电力试验研究有限公司 一种应急移动电源的控制方法及应急移动电源
CN119882416A (zh) * 2024-12-04 2025-04-25 重庆大学 基于深度强化学习的混合动力汽车队列节能驾驶分层控制方法
CN119872340A (zh) * 2025-03-27 2025-04-25 北京迅巢科技有限公司 一种基于自适应算法的新能源电池负载优化系统及方法
CN119911256A (zh) * 2023-10-31 2025-05-02 北京罗克维尔斯科技有限公司 一种驾驶策略确定方法、装置、系统及存储介质
CN120142941A (zh) * 2025-02-25 2025-06-13 中山大学·深圳 一种基于模型短分支推演的锂电池充电策略样本效率增强方法及系统
CN120387367A (zh) * 2025-04-15 2025-07-29 北京信息科技大学 融合多车运动交互感知的网联混合动力汽车能量管理方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113511082B (zh) * 2021-05-31 2023-06-16 深圳先进技术研究院 基于规则和双深度q网络的混合动力汽车能量管理方法
CN114475280B (zh) * 2022-03-01 2024-09-13 武汉理工大学 一种电动汽车混合动力系统能量管理方法及系统
CN114489144B (zh) * 2022-04-08 2022-07-12 中国科学院自动化研究所 无人机自主机动决策方法、装置及无人机
CN115001907A (zh) * 2022-05-06 2022-09-02 河北华万电子科技有限公司 一种irs辅助微型配电网智能计算方法
CN115190489B (zh) * 2022-07-07 2024-11-29 内蒙古大学 基于深度强化学习的认知无线网络动态频谱接入方法
CN115534764B (zh) * 2022-08-29 2024-11-15 华东理工大学 基于深度强化学习的车载燃料电池系统控制方法及系统
CN116231803B (zh) * 2023-03-02 2024-01-23 深圳市华南英才科技有限公司 一种电容笔快速充电方法、装置及存储介质
CN116777033A (zh) * 2023-03-15 2023-09-19 阿维塔科技(重庆)有限公司 车辆剩余里程的相关方法、装置及设备
CN116767024A (zh) * 2023-04-14 2023-09-19 联合汽车电子有限公司 一种基于强化学习的电池均衡方法及装置
CN119010616B (zh) * 2024-08-02 2025-03-28 广东电网有限责任公司 Lcl型并网逆变器控制方法、装置

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1323564A2 (en) * 2001-12-28 2003-07-02 Nissan Motor Co., Ltd. Control system for hybrid vehicle
CN109552079A (zh) * 2019-01-28 2019-04-02 浙江大学宁波理工学院 一种基于规则与Q-learning增强学习的电动汽车复合能量管理方法
CN109657194A (zh) * 2018-12-04 2019-04-19 浙江大学宁波理工学院 一种基于Q-learning和规则的混合动力车辆运行实时能源管理方法
CN110850877A (zh) * 2019-11-19 2020-02-28 北方工业大学 基于虚拟环境和深度双q网络的自动驾驶小车训练方法
CN111731303A (zh) * 2020-07-09 2020-10-02 重庆大学 一种基于深度强化学习a3c算法的hev能量管理方法
CN112287463A (zh) * 2020-11-03 2021-01-29 重庆大学 一种基于深度强化学习算法的燃料电池汽车能量管理方法
CN112670982A (zh) * 2020-12-14 2021-04-16 广西电网有限责任公司电力科学研究院 一种基于奖励机制的微电网有功调度控制方法及系统
CN113511082A (zh) * 2021-05-31 2021-10-19 深圳先进技术研究院 基于规则和双深度q网络的混合动力汽车能量管理方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210269152A1 (en) * 2018-06-27 2021-09-02 H3 Dynamics Holdings Pte. Ltd. Distributed electric energy pods network and associated electrically powered vehicle
JP7151689B2 (ja) * 2019-11-05 2022-10-12 トヨタ自動車株式会社 電池管理システム、電池管理方法、及び組電池の製造方法
CN110941202A (zh) * 2019-12-12 2020-03-31 中国科学院深圳先进技术研究院 一种汽车能量管理策略的验证方法和设备

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1323564A2 (en) * 2001-12-28 2003-07-02 Nissan Motor Co., Ltd. Control system for hybrid vehicle
CN109657194A (zh) * 2018-12-04 2019-04-19 浙江大学宁波理工学院 一种基于Q-learning和规则的混合动力车辆运行实时能源管理方法
CN109552079A (zh) * 2019-01-28 2019-04-02 浙江大学宁波理工学院 一种基于规则与Q-learning增强学习的电动汽车复合能量管理方法
CN110850877A (zh) * 2019-11-19 2020-02-28 北方工业大学 基于虚拟环境和深度双q网络的自动驾驶小车训练方法
CN111731303A (zh) * 2020-07-09 2020-10-02 重庆大学 一种基于深度强化学习a3c算法的hev能量管理方法
CN112287463A (zh) * 2020-11-03 2021-01-29 重庆大学 一种基于深度强化学习算法的燃料电池汽车能量管理方法
CN112670982A (zh) * 2020-12-14 2021-04-16 广西电网有限责任公司电力科学研究院 一种基于奖励机制的微电网有功调度控制方法及系统
CN113511082A (zh) * 2021-05-31 2021-10-19 深圳先进技术研究院 基于规则和双深度q网络的混合动力汽车能量管理方法

Cited By (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116054285A (zh) * 2022-12-30 2023-05-02 国网湖北省电力有限公司经济技术研究院 一种基于联邦强化学习算法的输配调频资源协同控制方法
CN116257937A (zh) * 2023-01-05 2023-06-13 东南大学 基于多目标深度强化学习的混合动力汽车生态驾驶方法
CN116208545A (zh) * 2023-02-07 2023-06-02 哈尔滨工业大学 网络路由优化方法及装置、智能路由方法及装置
CN116050505A (zh) * 2023-02-22 2023-05-02 西南交通大学 一种基于伙伴网络的智能体深度强化学习方法
CN116123702A (zh) * 2023-02-23 2023-05-16 中交武汉智行国际工程咨询有限公司 基于支持向量机和深度强化学习的建筑空调温度控制方法
CN116373673A (zh) * 2023-03-23 2023-07-04 国网江苏省电力有限公司南通供电分公司 一种电动汽车充放电管理方法及系统
CN116384250A (zh) * 2023-04-12 2023-07-04 中国长江三峡集团有限公司 一种综合能源系统多个智能体的协作控制方法与能源系统
CN116605092A (zh) * 2023-04-26 2023-08-18 中国科学技术大学 一种能量管理方法及能量管理平台
CN116582860A (zh) * 2023-05-08 2023-08-11 南京航空航天大学 一种基于信息年龄约束的链路资源分配方法
CN116736700A (zh) * 2023-05-16 2023-09-12 武汉理工大学 多动力源两栖艇能量管理控制系统及方法
CN116561579A (zh) * 2023-05-17 2023-08-08 四川大学 自适应多工况的递归深度强化学习混动汽车能量管理策略
CN116561579B (zh) * 2023-05-17 2026-04-21 四川大学 自适应多工况的递归深度强化学习混动汽车能量管理策略
CN117021066A (zh) * 2023-05-26 2023-11-10 浙江大学 一种基于深度强化学习的机器人视觉伺服运动控制方法
CN117034443A (zh) * 2023-07-13 2023-11-10 哈尔滨工业大学 基于ddqn算法的机动观测策略生成方法、改进ddqn算法的机动观测策略生成方法
CN117021971A (zh) * 2023-08-15 2023-11-10 北京工业大学 一种多动力源车辆能量管理策略优化方法、系统及设备
CN116853073B (zh) * 2023-09-04 2024-01-26 江西五十铃汽车有限公司 一种新能源电动汽车能量管理方法及系统
CN116853073A (zh) * 2023-09-04 2023-10-10 江西五十铃汽车有限公司 一种新能源电动汽车能量管理方法及系统
CN116995641B (zh) * 2023-09-26 2023-12-22 北京交通大学 一种应用于轨道交通储能的能量管理架构及方法
CN116995641A (zh) * 2023-09-26 2023-11-03 北京交通大学 一种应用于轨道交通储能的能量管理架构及方法
CN116985646B (zh) * 2023-09-28 2024-01-12 江西五十铃汽车有限公司 车辆超级电容器控制方法、设备和介质
CN116985646A (zh) * 2023-09-28 2023-11-03 江西五十铃汽车有限公司 车辆超级电容器控制方法、设备和介质
CN117058468A (zh) * 2023-10-11 2023-11-14 青岛金诺德科技有限公司 用于新能源汽车锂电池回收的图像识别与分类系统
CN117058468B (zh) * 2023-10-11 2023-12-19 青岛金诺德科技有限公司 用于新能源汽车锂电池回收的图像识别与分类系统
CN117650553A (zh) * 2023-10-25 2024-03-05 四川大学 基于多智能体深度强化学习的5g基站储能电池充放电调度方法
CN119911256A (zh) * 2023-10-31 2025-05-02 北京罗克维尔斯科技有限公司 一种驾驶策略确定方法、装置、系统及存储介质
CN117227700A (zh) * 2023-11-15 2023-12-15 北京理工大学 串联混合动力无人履带车辆的能量管理方法及系统
CN117227700B (zh) * 2023-11-15 2024-02-06 北京理工大学 串联混合动力无人履带车辆的能量管理方法及系统
CN117650522A (zh) * 2023-12-04 2024-03-05 北京北交本有科技有限公司 一种基于深度强化学习的多储能系统分层协同控制方法
CN117507789A (zh) * 2023-12-06 2024-02-06 江苏开放大学（江苏城市职业学院） 一种用于agv小车的多模混合动力驱动系统
CN117507789B (zh) * 2023-12-06 2024-06-07 江苏开放大学（江苏城市职业学院） 一种用于agv小车的多模混合动力驱动系统
CN117578679B (zh) * 2024-01-15 2024-03-22 太原理工大学 基于强化学习的锂电池智能充电控制方法
CN117578679A (zh) * 2024-01-15 2024-02-20 太原理工大学 基于强化学习的锂电池智能充电控制方法
CN117944527A (zh) * 2024-02-02 2024-04-30 山东德维鲁普新材料有限公司 基于氢燃料电池发电车的能量管理方法
CN117933666A (zh) * 2024-03-21 2024-04-26 壹号智能科技(南京)有限公司 一种密集仓储机器人调度方法、装置、介质、设备及系统
CN118544901A (zh) * 2024-06-05 2024-08-27 郑州大学 一种基于双q学习与实时速度预测的氢-电混动系统能量管理方法、系统、设备及存储介质
CN118596882A (zh) * 2024-06-11 2024-09-06 徐州徐工重型车辆有限公司 纯电动矿用自卸车分布式线控轮边驱动系统及驱动方法
CN118705133A (zh) * 2024-06-15 2024-09-27 曲阜师范大学 一种基于深度强化学习的风力机转速参考值确定方法
CN119359063A (zh) * 2024-09-09 2025-01-24 华北电力大学 电动汽车充放电调度策略生成方法
CN118859730A (zh) * 2024-09-25 2024-10-29 中南大学 自适应感知的重载列车群组运行能量管理方法及控制装置
CN118928165A (zh) * 2024-10-12 2024-11-12 国网浙江省电力有限公司嘉善县供电公司 氢燃料电池车的多电源协调控制方法、系统、设备及介质
CN119187463A (zh) * 2024-10-30 2024-12-27 扬州市江都区飞跃泵业有限公司 一种双叶片半开式污水泵叶轮及其制造装置、方法
CN119882416A (zh) * 2024-12-04 2025-04-25 重庆大学 基于深度强化学习的混合动力汽车队列节能驾驶分层控制方法
CN119882416B (zh) * 2024-12-04 2025-09-30 重庆大学 基于深度强化学习的混合动力汽车队列节能驾驶分层控制方法
CN119675213A (zh) * 2025-02-20 2025-03-21 南昌科晨电力试验研究有限公司 一种应急移动电源的控制方法及应急移动电源
CN120142941A (zh) * 2025-02-25 2025-06-13 中山大学·深圳 一种基于模型短分支推演的锂电池充电策略样本效率增强方法及系统
CN119872340A (zh) * 2025-03-27 2025-04-25 北京迅巢科技有限公司 一种基于自适应算法的新能源电池负载优化系统及方法
CN120387367A (zh) * 2025-04-15 2025-07-29 北京信息科技大学 融合多车运动交互感知的网联混合动力汽车能量管理方法

Also Published As

Publication number Publication date
CN113511082A (zh) 2021-10-19
CN113511082B (zh) 2023-06-16

Similar Documents

Publication Publication Date Title
WO2022252559A1 (zh) 基于规则和双深度q网络的混合动力汽车能量管理方法
CN110341690B (zh) 一种基于确定性策略梯度学习的phev能量管理方法
Du et al. Heuristic energy management strategy of hybrid electric vehicle based on deep reinforcement learning with accelerated gradient optimization
Wu et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus
Zhao et al. A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles
Bo et al. A Q-learning fuzzy inference system based online energy management strategy for off-road hybrid electric vehicles
Yu et al. Real time energy management strategy for a fast charging electric urban bus powered by hybrid energy storage system
Wang et al. Deep reinforcement learning with deep-Q-network based energy management for fuel cell hybrid electric truck
Lin et al. Intelligent energy management strategy based on an improved reinforcement learning algorithm with exploration factor for a plug-in PHEV
CN112215434A (zh) 一种lstm模型的生成方法、充电时长预测方法及介质
Li et al. An improved energy management strategy of fuel cell hybrid vehicles based on proximal policy optimization algorithm
CN119167791B (zh) 一种氢燃料电池混合动力无人机能量自适应优化系统及方法
CN110297452B (zh) 一种蓄电池组相邻型均衡系统及其预测控制方法
El Ouazzani et al. MSCC-DRL: Multi-Stage constant current based on deep reinforcement learning for fast charging of lithium ion battery
Zhang et al. A review of energy management optimization based on the equivalent consumption minimization strategy for fuel cell hybrid power systems
Sun et al. A new design of fuzzy logic control for SMES and battery hybrid storage system
US20250045594A1 (en) Energy storage system, and thermal management method for energy storage system
Wang et al. An efficient power control scheme for heavy-duty hybrid electric vehicle with online optimized variable universe fuzzy system
US20250214564A1 (en) Multi-objective optimization method and system assisted by gradient boosted neural network and device
Gao et al. Real-time three-level energy management strategy for series hybrid wheel loaders based on WG-MPC
Yang et al. A DOD-SOH balancing control method for dynamic reconfigurable battery systems based on DQN algorithm
Gao et al. An energy management strategy for fuel cell hybrid electric vehicle based on a real-time model predictive control and pontryagin’s maximum principle
Hu et al. Intelligent energy management for the fuel cell electric vehicle: A real-time, deep reinforcement learning application to multi-objective optimization
Li et al. Adaptive hierarchical energy management strategy for fuel cell hybrid engineering vehicles based on deep reinforcement learning
CN117214755A (zh) 一种基于混合神经网络的锂离子电池soc、soe及soh联合估计方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21943911

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21943911

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21943911

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 31/12/2024)

122 Ep: pct application non-entry in european phase

Ref document number: 21943911

Country of ref document: EP

Kind code of ref document: A1