CN117613983A - Energy storage charge and discharge control decision method and device based on fusion rule reinforcement learning - Google Patents

Abstract

The specification relates to the technical field of electric power, in particular to an energy storage charging and discharging control decision method and device based on fusion rule reinforcement learning, which are applied to a power grid and a user side optical energy storage device and comprise the following steps: determining a state space by using user electric power, the charge state of an energy storage battery, outdoor temperature, solar irradiance, power grid electricity unit price and power grid selling unit price in any time period; inputting the state space into a charge and discharge control decision model based on fusion rule reinforcement learning to obtain an optimal charge and discharge decision variable, wherein the optimal charge and discharge decision variable comprises: the optimal charge and discharge power and the optimal coefficient of the energy storage battery are obtained through training a charge and discharge control decision model based on fusion rule reinforcement learning through a sample state space and a photovoltaic power generation uncertain model. The method and the device integrate the predefined rules, improve the speed of convergence of reinforcement learning training to the optimal charge-discharge control strategy, improve the electricity economy and reduce the power fluctuation and burden of the power grid.

Description

Energy storage charge and discharge control decision-making method and device based on fusion rule reinforcement learning

Technical field

This specification relates to the field of electric power technology, especially the energy storage charge and discharge control decision-making method and device based on fusion rule reinforcement learning.

Background technique

As the total electricity consumption and proportion of electricity consumption in the tertiary industry and urban and rural residential users continue to increase, the peak load demand of the power grid during peak electricity consumption continues to increase, and the peak-valley electricity price gap further widens and remains high. In order to reduce grid load fluctuations, reduce grid expansion costs and improve the economic efficiency of the power system, user-side energy storage technology has received widespread attention and has great development potential.

In the field of user-side energy storage technology, user-side photovoltaic energy storage also includes photovoltaic power generation, energy storage batteries and grid power supply. Efficient coordination between different power equipment needs to be solved urgently. Existing research mainly focuses on the optimal configuration of user-side energy storage facilities and energy storage planning methods. By calculating the first power and first capacity that the energy storage system needs to configure during the charging period, and the second power and second capacity that need to be configured during the discharge period, the maximum value of the first power and the second power is taken as the required configuration of the energy storage system. The minimum value of the first capacity and the second capacity is taken as the capacity to be configured for the energy storage system, which improves the benefits of peak shaving and valley filling.

There are also user energy storage planning methods in the existing technology. A user-side energy storage planning model is constructed. By minimizing the comprehensive objective function of initial consumption, operation and maintenance consumption, power consumption and basic consumption, the optimal planning model parameters are obtained. The model is simple and It cannot be adjusted adaptively, and photovoltaic power generation factors are not considered. In addition, photovoltaic power generation is affected by uncertain factors such as dust, temperature and shading, and its power generation is uncertain. Therefore, user-side energy storage operation control considering the uncertainty of photovoltaic power generation needs to be further optimized.

Contents of the invention

In order to solve the user-side energy storage problem in the above-mentioned prior art that does not consider the uncertainty of photovoltaic power generation, embodiments of this specification provide a charging and discharging control decision model training method based on fusion rule reinforcement learning.

The example in this manual provides an energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning. The method is applied to the power grid and user-side photovoltaic energy storage equipment. The user-side photovoltaic energy storage equipment includes: photovoltaic power generation device, user Electric devices and energy storage batteries, the method includes: determining the user's power consumption in any period of time, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, and grid electricity sales unit price. State space; input the state space into the charge and discharge control decision model based on fusion rule reinforcement learning to obtain the optimal charge and discharge decision variables for any time period, where the optimal charge and discharge decision variables include: the optimal charge and discharge decision variables of the energy storage battery Optimal charge and discharge power and optimal coefficients are obtained by training the sample state space and photovoltaic power generation uncertainty model based on the fusion rule reinforcement learning charge and discharge control decision-making model.

According to one aspect of the embodiments of this specification, the charging and discharging control decision-making model based on fusion rule reinforcement learning is trained through the following steps: according to the state variables and environmental variables related to the user's power device, energy storage battery, and power grid during the sample time period, determine Sample state space; construct a basic model for charge and discharge control decision-making based on the photovoltaic power generation uncertainty model, where the photovoltaic power generation uncertainty model is a Gaussian distribution model that considers environmental variables; input the state variables in the sample state space into the charge and discharge control decision-making model. The discharge control decision-making basic model outputs charging and discharging decision variables; the charging and discharging decision variables are constrained according to the preset charging and discharging rules, and the charging and discharging control decision-making basic model is iteratively trained using the reward function to obtain the trained charging and discharging control decision-making model.

According to one aspect of the embodiment of this specification, the state variable of the energy storage battery in the sample time period is determined in the following manner: obtaining the charging power of the energy storage battery in the historical sample period before the sample time period; according to the charging power of the energy storage battery in the historical sample period , determine the state of charge of the energy storage battery during the sample time period through the following formula:

; Among them, k represents the current sample time period, /> It is the state of charge of the energy storage battery in the first sample time period (i.e. the initial moment);/> is the self-discharge rate of the energy storage battery;/> is the open circuit voltage of the energy storage battery;/> is the rated capacity of the energy storage battery;/> is the charging and discharging efficiency of the energy storage battery;/> ,/> ,/> is the charging and discharging power of the energy storage battery in the i -th sample time period. When the energy storage battery is discharged,/> , when the energy storage battery is charging,/> ;/> ,/> , is the efficiency index of the i -th time period, which satisfies:/> .

According to one aspect of the embodiments of this specification, constructing a basic model for charge and discharge control decision-making of the energy storage device based on the photovoltaic power generation uncertainty model includes: determining the photovoltaic power generation power in the sample time period based on the photovoltaic power generation uncertainty model that obeys Gaussian distribution. ; Based on the difference between the photovoltaic power generation power and the user's electricity consumption during the sample time period, determine the power output by the photovoltaic power generation device to the power grid during the sample time period.

According to one aspect of the embodiment of this specification, the method includes: determining the photovoltaic power generation uncertainty model through the following formula: ;wherein,/> is the photovoltaic power generation during the sample time period;/> is the photoelectric conversion efficiency;/> is the photovoltaic material area;/> Indicates the solar irradiance during the sample time period;/> is the variance; photovoltaic power generation obeys the mean expectation as/> , the variance is/> Gaussian distribution; according to the photovoltaic power generation obtained by the photovoltaic power generation uncertainty model, the power output from the photovoltaic power generation device to the grid is determined by the following formula:/> ;wherein,/> is the power output by the photovoltaic power generation device to the grid during the sample time period; Pload is the user's power consumption;/> Indicates photovoltaic power generation;/> is the proportional weight,/> .

According to one aspect of the embodiment of this specification, determining the preset charging and discharging rules in the following manner includes: if the current period belongs to the peak period of electricity consumption, setting the charging and discharging power of the energy storage battery to be negative; if the current period belongs to the low period of electricity consumption section, set the charging and discharging power of the energy storage battery to be positive;

If the photovoltaic power generation is less than the load power, the proportional weight is set to zero; if the photovoltaic power generation is greater than the load power, the proportional weight is set to the original value of the proportional weight; the charge and discharge control decisions are constrained according to the above preset charge and discharge rules. Decision variables in the model.

According to one aspect of the embodiments of this specification, using the reward function to iteratively train the charge and discharge control decision-making basic model includes: determining the economic reward function based on the unit price of electricity sales, the unit price of electricity in each period, and the input power from the power grid in each period; Based on the charging and discharging power of the energy storage battery at the current moment and the charging and discharging power of the energy storage battery at the previous moment, a ride comfort reward function is constructed; according to the input power of the power grid in each period, the maximum demand power reward function is determined; according to the minimum power of the energy storage battery The state of charge, the maximum state of charge of the energy storage battery and the current state of charge of the energy storage battery are used to determine the constraint penalty reward function; according to the economic reward function, ride comfort reward function, maximum demand power reward function and constraint penalty Reward function, iteratively train the charge and discharge control decision model. When the above reward reaches the maximum value and remains stable within the preset number of iterations, it is determined that the charge and discharge control decision model training is completed.

The embodiment of this specification also provides a photovoltaic energy storage system based on fusion rule reinforcement learning. The system includes: a power grid, communicatively connected to the user-side photovoltaic energy storage equipment, and used to supply power to the user-side photovoltaic energy storage equipment; Photovoltaic energy storage equipment includes: photovoltaic power generation device, user power device and energy storage battery, wherein the photovoltaic power generation device is used to provide photovoltaic power generation to the user power device, and is used to use the remaining power when the power is remaining. The energy storage battery is used to store the remaining power of the photovoltaic power generation device and supply power to the user's electrical device.

The embodiment of this specification also provides a charging and discharging control decision model training device based on fusion rule reinforcement learning. The device includes: a sample state space construction unit, which is used to communicate with the user's power device, energy storage battery, The state variables and environmental variables related to the power grid determine the sample state space; the basic model construction unit is used to construct a basic model for charge and discharge control decision-making of the energy storage device based on the photovoltaic power generation uncertainty model, where the photovoltaic power generation uncertainty model is A Gaussian distribution model that considers environmental variables; an output unit for inputting state variables in the sample state space into the charge and discharge control decision basic model and outputting charge and discharge decision variables; a model training unit for charging and discharging according to preset rules , iteratively train the charge and discharge control decision-making basic model until the basic model meets the preset conditions, and obtain the fully trained charge and discharge control decision-making model.

Embodiments of this specification provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the fusion rule-based reinforcement learning. The charging and discharging control decision model training method.

Embodiments of this specification also provide a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, the charge-discharge control decision-making model based on fusion rule reinforcement learning is implemented. Training methods.

This manual integrates predefined rules to improve the speed of reinforcement learning training to converge to the optimal charge and discharge control strategy, and reduce power fluctuations and burdens on the power grid.

Description of drawings

In order to more clearly explain the embodiments of this specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 shows a flow chart of a charging and discharging control decision-making method based on fusion rule reinforcement learning according to an embodiment of this specification;

Figure 2 shows a flow chart of a method for training a charging and discharging control decision-making model based on fusion rule reinforcement learning according to an embodiment of this specification;

Figure 3 shows a flow chart of a method for constructing a basic model for charge and discharge control decision-making according to an embodiment of this specification;

Figure 4 shows a flow chart of a method for determining state variables of an energy storage battery in a sample time period according to an embodiment of this specification;

Figure 5 shows a flow chart of a method for determining preset charging and discharging rules according to an embodiment of this specification;

Figure 6 shows a flow chart of a method for iteratively training the charge and discharge control decision-making basic model using a reward function according to an embodiment of this specification;

Figure 7 shows a schematic diagram of a photovoltaic energy storage system based on fusion rule reinforcement learning according to an embodiment of this specification;

Figure 8 shows a schematic structural diagram of a charging and discharging control decision model training device based on fusion rule reinforcement learning according to an embodiment of this specification;

FIG. 9 is a schematic structural diagram of a computer device according to an embodiment of this specification.

Explanation of drawing symbols:

701. Power grid;

702. User-side photovoltaic energy storage equipment;

703. Photovoltaic power generation device;

704. User electrical equipment;

705. Energy storage battery;

801. State space building unit;

802. Optimal charging and discharging decision variable determination unit;

902. Computer equipment;

904. Processor;

906. Memory;

908. Driving mechanism;

910. Input/output module;

912. Input device;

914. Output device;

916. Presentation equipment;

918. Graphical user interface;

920. Network interface;

922. Communication link;

924. Communication bus.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described The embodiments are only some of the embodiments of this specification, but not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this specification.

It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the specification described herein can be practiced in sequences other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, apparatus, product or equipment that includes a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

This specification provides method operation steps as described in the examples or flow charts, but more or less operation steps may be included based on routine or non-inventive efforts. The sequence of steps listed in the embodiment is only one way of executing the sequence of many steps, and does not represent the only execution sequence. When the actual system or device product is executed, the methods shown in the embodiments or drawings may be executed sequentially or in parallel.

It should be noted that the energy storage charge and discharge control decision-making method and device based on fusion rule reinforcement learning in this specification can be used in the field of electric power technology. The application fields of this specification are to the energy storage charge and discharge control decision-making method and device based on fusion rule reinforcement learning. No restrictions.

Figure 1 shows a flow chart of a charging and discharging control decision-making method based on fusion rule reinforcement learning according to an embodiment of this specification, which specifically includes the following steps:

Step 101: Determine the state space based on the user's power consumption in any time period, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, and grid electricity sales unit price.

This step is applied in the model application stage. It is similar to the state sample space constructed in the model training stage. It takes the user's power consumption, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, The unit price of electricity sold by the power grid determines the state space.

Step 102: Input the state space into the charge and discharge control decision-making basic model based on fusion rule reinforcement learning to obtain the optimal charge and discharge decision variables for any time period. Among them, the optimal charge and discharge decision variables include: the optimal charge and discharge power and optimal coefficient of the energy storage battery. The charge and discharge control decision model based on fusion rule reinforcement learning is obtained through sample state space and photovoltaic power generation uncertainty model training. .

In this step, the state space is used as the input of the optimal policy network, and the optimal charging and discharging actions of the user-side energy storage battery in this time period are output. . In this step, the training process of the charge and discharge control decision-making model based on fusion rule reinforcement learning is detailed in Figure 2.

Figure 2 shows a flow chart of a method for training a charging and discharging control decision-making model based on fusion rule reinforcement learning according to an embodiment of this specification, which specifically includes the following steps:

Step 201: Determine the sample state space based on the state variables and environmental variables related to the user's power device, energy storage battery, and power grid during the sample time period. In this step, the state variable related to the user's electrical device is the user load power, where the user load power in the sample time period can be calculated by means; the state variable related to the energy storage battery is the state of charge of the energy storage battery, which can be expressed by/> represents; the state variables related to the power grid are the unit price of electricity consumption and the unit price of electricity sales. Among them, the unit price of electricity during the sample time period can be expressed by/> represents; the electricity sales unit price during the sample period can be expressed by/> express. Environmental variables are outdoor air temperature and solar irradiance. Among them, the outdoor air temperature in the sample time period can be calculated by/> represents; the solar irradiance in the sample time period can be expressed by/> express.

The above-mentioned state variables and environmental variables in the same sample time period are determined as the sample state space, which is subsequently used as input to the basic model of charge and discharge control decision-making.

Step 202: Construct a basic model for charge and discharge control decision-making based on the photovoltaic power generation uncertainty model, where the photovoltaic power generation uncertainty model is a Gaussian distribution model that takes into account environmental variables. In this step, by constructing the sample state space and action space, combined with the photovoltaic power generation uncertainty model, a basic model for charge and discharge control decision-making of user-side photovoltaic energy storage equipment is constructed. The basic model of charge and discharge control decision-making is a policy network, and the basic model is constructed by a fully connected neural network. Among them, the basic model of charge and discharge control decision-making is obtained by randomly initializing the parameters of the policy network.

The action space of the embodiment of this specification can be expressed as follows: ;wherein,/> Indicates the maximum charging power of the energy storage battery,/> Indicates the maximum discharge power of the energy storage battery.

Step 203: Input the state variables in the sample state space into the charge and discharge control decision basic model, and output the charge and discharge decision variables. The decision variables are the charge and discharge power and proportional weight of the energy storage battery. Among them, the proportional weight determines the power of photovoltaic output to the grid, with the purpose of selling excess photovoltaic power into the grid to earn economic profits.

Step 204: Constrain the charging and discharging decision variables according to the preset charging and discharging rules, use the reward function to iteratively train the charging and discharging control decision basic model, and obtain the trained charging and discharging control decision model. In this step, the reinforcement learning method of fusion rules is used to iteratively train the charge and discharge control decision-making basic model until the model that maximizes the reward is obtained to obtain the charge and discharge control decision-making model.

In the embodiment of this specification, the state variables of the charge and discharge control decision model are As the input of the policy network, the action network outputs the action variables of the charge and discharge control decision model/> ; Use predefined rules to correct action variables.

Furthermore, the reward function is used to train the charge and discharge control decision-making basic model. When the value of the reward function of the charge and discharge control decision model reaches the maximum and remains stable within a certain iteration time or number of iterations, the policy network corresponding to the maximum reward value is determined. Basic model of charge and discharge control decision-making, model training completed.

This manual integrates predefined rules to improve the speed of reinforcement learning training to converge to the optimal charging and discharging control strategy, improve the economy of electricity consumption, and reduce power fluctuations and burdens on the power grid.

Figure 3 shows a flow chart of a method for constructing a basic model for charge and discharge control decision-making according to an embodiment of this specification, which specifically includes the following steps:

Step 301: Determine the photovoltaic power generation power in the sample time period based on the photovoltaic power generation uncertainty model that obeys Gaussian distribution. In the embodiment of this specification, the photovoltaic power generation uncertainty model can determine the photovoltaic power generation power in the kth sample time period. The photovoltaic power generation uncertainty model is used to calculate the photovoltaic power generation power, taking into account environmental factors, including ambient temperature and solar irradiation. Degree, the photovoltaic power generation in the kth sample time period obeys the mean expectation as , the variance is/> Gaussian distribution, where, /> is the photovoltaic power generation during the sample time period;/> is the photoelectric conversion efficiency;/> is the photovoltaic material area;/> Indicates the solar irradiance during the sample time period;/> is the variance. This manual takes into account the uncertainty of photovoltaic power generation caused by environmental factors, achieves coordination between photovoltaic power generation devices, energy storage batteries and the power grid, and reduces user economic costs and power grid power supply load.

Step 302: Determine the power output by the photovoltaic power generation device to the grid during the sample time period based on the decision variables output by the charge and discharge control decision-making basic model and the difference between the photovoltaic power generation power in the sample time period and the user's power consumption.

In this step, the decision variables output by the charge and discharge control decision-making basic model include proportional weights and the charging and discharging power of the energy storage battery. According to the previous steps, it can be seen that the photovoltaic power generation power in the sample time period is given by/> express.

In this step, the power output from the photovoltaic power generation device to the grid during the sample time period is expressed by the following formula: ;wherein,/> is the power output by the photovoltaic power generation device to the grid in the k-th sample time period;/> It is the user’s power consumption in the k-th sample time period;/> Represents the photovoltaic power generation power in the kth sample time period;/> is the proportional weight,/> . Among them, the user's electrical power can be collected directly. The power formula of the above-mentioned photovoltaic power generation device output to the power grid indicates that the power generated by the photovoltaic power generation device is given priority to the user, and if there is remaining power, it is allocated to the energy storage battery or the power grid. In the embodiment of this specification, the power and proportional weight output by the photovoltaic power generation device to the power grid are decision variables for training the basic model of charge and discharge control decision-making.

In other embodiments of this specification, the power balance equation is used to describe the balance of power input and output in the operating state of a system composed of the power grid and user-side photovoltaic energy storage equipment.

The power balance equation is expressed by the following formula: ,/> The power input from the power grid in the kth sample time period;/> is the power output to the grid in the k -th sample time period, which is provided by the photovoltaic power generation device;/> It is the user’s power consumption in the k-th sample time period;/> Represents the photovoltaic power generation power in the kth sample time period.

Figure 4 shows a flow chart of a method for determining the state variables of an energy storage battery during a sample time period according to an embodiment of this specification, which specifically includes the following steps:

Step 401: Obtain the charging power of the energy storage battery in the historical sample period before the sample time period. In this step, the historical sample periods before the sample period are: all historical sample periods before the current sample period. For example, the current sample time period is the k-th sample time period, then the historical sample time periods before the sample time period are the k-1th sample time period, the k-2nd sample time period... the 2nd sample time period, The first sample time period.

In the embodiment of this specification, the charging power of the energy storage battery during the historical sample period can be collected and obtained.

Step 402: Determine the state of charge of the energy storage battery in the sample time period based on the charging power of the energy storage battery in the historical sample time period. In this step, the state of charge of the energy storage battery during the sample time period is determined through the following formula:

; Among them, k represents the current sample time period, /> It is the state of charge of the energy storage battery in the first sample time period (i.e. the initial moment);/> is the self-discharge rate of the energy storage battery;/> is the open circuit voltage of the energy storage battery;/> is the rated capacity of the energy storage battery;/> is the charging and discharging efficiency of the energy storage battery;/> ,/> ,/> is the charging and discharging power of the energy storage battery in the i -th sample time period. When the energy storage battery is discharged,/> , when the energy storage battery is charging,/> ;/> ,/> , is the efficiency index of the i-th time period, which satisfies:/> .

In the embodiment of this specification, the photovoltaic power generation uncertainty model is determined through the following formula: ;wherein,/> is the photovoltaic power generation during the sample time period;/> is the photoelectric conversion efficiency;/> is the photovoltaic material area;/> Indicates the solar irradiance during the sample time period;/> is the variance; photovoltaic power generation obeys the mean expectation as/> , the variance is/> Gaussian distribution;

According to the photovoltaic power generation obtained by the photovoltaic power generation uncertainty model, the power output from the photovoltaic power generation device to the grid is determined by the following formula: ;wherein,/> is the power output by the photovoltaic power generation device to the grid in the k-th sample time period;/> It is the user’s power consumption in the k-th sample time period;/> Indicates the photovoltaic power generation of light in the kth sample time period;/> is the proportional weight,/> .

Figure 5 shows a flow chart of a method for determining preset charging and discharging rules according to an embodiment of this specification, which specifically includes the following steps:

Step 501: If the current period belongs to the peak power consumption period, set the charging and discharging power of the energy storage battery to be negative. The preset charging and discharging rules in this step indicate that charging of the energy storage battery is prohibited during the peak period of electricity consumption, thereby ensuring that the energy storage battery can fully supply the user's electrical devices during the peak period of electricity consumption, and improving the efficiency of the energy storage battery. Utilization efficiency during peak electricity consumption periods.

Step 502: If the current period belongs to a low power consumption period, set the charging and discharging power of the energy storage battery to be positive. The preset charging and discharging rules in this step indicate that the energy storage battery is allowed to be charged during the period when power consumption is underestimated. Therefore, ; Among them, F represents the set of peak power consumption time periods, and G represents the set of low power consumption time periods;/> represents the charging and discharging power of the energy storage battery in the kth time period,

Step 503: If the photovoltaic power generation is less than the load power, set the proportional weight to zero. The preset charging and discharging rules in this step indicate that if the power generation of the photovoltaic power generation device is less than the load power of the user's electrical device and the photovoltaic power generation device can no longer supply power to the energy storage battery or the grid, then the proportional weight is set to 0.

Step 504: If the photovoltaic power generation is greater than the load power, set the proportional weight to the original value of the proportional weight. In this step, the preset rule indicates that when the power generated by the photovoltaic power generation device is greater than the power consumption of the user's power device, the photovoltaic power generation device can supply power to the energy storage battery or the power grid, and the proportional weight remains at its original value.

In the embodiment of this specification, according to the above-mentioned preset charging and discharging rules, the decision variables in the charging and discharging control decision model can be constrained, and the rules can be used to accelerate the efficiency of reinforcement learning and the efficiency of model iteration.

Figure 6 shows a flow chart of a method for iteratively training the charge and discharge control decision-making basic model using a reward function according to an embodiment of this specification, which specifically includes the following steps:

Step 601: Determine the economic reward function based on the unit price of electricity sold in each period, the unit price of electricity used in each period, and the input power from the power grid in each period. In the embodiment of this specification, the economic reward function can be determined by the following formula:

in, is the economic reward in the kth time period;

Represents the power input from the power grid in the i-th sample time period;

represents the power output to the grid in the i-th sample time period, and N represents the total number of time periods;

Represents the unit price of electricity in the i-th sample time period,/> Indicates the electricity sales unit price in the i-th sample time period.

Step 602: Construct a ride comfort reward function based on the charging and discharging power of the energy storage battery at the current moment and the charging and discharging power of the energy storage battery at the previous moment.

Specifically, the ride comfort reward function is expressed by the following formula:

;

In the formula, is the smoothness reward for the k -th sample time period;/> and/> are the charging and discharging power of the energy storage battery in the k- th time period and the k- 1th time period respectively.

Step 603: Determine the maximum demand power reward function based on the input power of the power grid in each period. In this step, the maximum demand power reward function is expressed by the following formula: ;

In the formula, is the maximum demand power reward function in the kth time period. The maximum demand reward function is used to minimize the input power of the power grid and reduce the burden on the power grid during peak power consumption periods.

Step 604: Determine the constraint penalty reward function based on the minimum state of charge of the energy storage battery, the maximum state of charge of the energy storage battery, and the current state of charge of the energy storage battery.

The constrained penalty reward function can be expressed by the following formula:

;

In the formula, is the constrained penalty reward function of the k -th sample time period,/> is the exponential parameter;/> and are respectively the minimum state of charge and the maximum state of charge of the energy storage battery. The purpose of the constraint penalty reward function is to constrain the state of charge of the energy storage battery and avoid damaging the battery.

Step 605: Iteratively train the charge and discharge control decision model according to the economic reward function, ride comfort reward function, maximum demand power reward function and constraint penalty reward function. When the above reward reaches the maximum value and is within the preset number of iterations, Maintain stability and confirm that the charge and discharge control decision model training is completed.

In the embodiment of this specification, the reward function of the charge and discharge control decision-making model is determined based on the economic reward function, ride comfort reward function, maximum demand power reward function, and constraint penalty reward function constructed in the previous steps, as expressed by the following formula: ;

In the formula, is the reward for the kth time period;/> ,/> ,/> and/> They are the economic reward, ride comfort reward, maximum demand power reward and constraint penalty reward in the k -th sample time period;/> ,/> ,/> and/> is the weight coefficient.

The charge and discharge control decision model is iteratively trained according to the reward function. When the reward reaches the maximum value and remains stable within the preset number of iterations or remains stable within a period of model training, it is determined that the charge and discharge control decision model training is completed.

Figure 7 shows a schematic diagram of a photovoltaic energy storage system based on fusion rule reinforcement learning according to an embodiment of this specification. The system includes:

Power grid 701, user-side photovoltaic energy storage equipment 702. Among them, the power grid 701 is communicatively connected to the user-side photovoltaic energy storage device 702 and is used to supply power to the user-side photovoltaic energy storage device 702 . The user-side photovoltaic energy storage equipment 702 includes: a photovoltaic power generation device 703, a user power device 704, and an energy storage battery 705. The photovoltaic power generation device 703 is used to provide photovoltaic power generation to the user power device 704, and is used for current use. When the power is remaining, the remaining power is transferred to the energy storage battery 705 or the power grid 701. The energy storage battery 705 is used to store the remaining power of the photovoltaic power generation device 703 and to provide power to the user's electrical device 704 .

Figure 8 is a schematic structural diagram of a charge and discharge control decision-making device based on fusion rule reinforcement learning according to an embodiment of this specification. This figure describes the basic structure of a charge and discharge control decision-making device based on fusion rule reinforcement learning, in which Functional units and modules can be implemented in software, or general chips or specific chips can be used to implement charge and discharge control decisions based on fusion rule reinforcement learning. The device specifically includes:

The state space construction unit 801 is used to determine the state space based on the user's power consumption in any period of time, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, and grid electricity sales unit price;

The optimal charging and discharging decision variable determination unit 802 is used to input the state space into the charging and discharging control decision model based on fusion rule reinforcement learning to obtain the optimal charging and discharging decision variables for any period of time, wherein the optimal charging and discharging decision variable is The discharge decision variables include: the optimal charge and discharge power and optimal coefficient of the energy storage battery. The charge and discharge control decision model based on fusion rule reinforcement learning is obtained through sample state space and photovoltaic power generation uncertainty model training.

This manual aims at the coordination problem between photovoltaic power generation, energy storage batteries and power grid of user-side photovoltaic energy storage facilities. It considers the uncertainty of photovoltaic power generation of user-side photovoltaic energy storage facilities, establishes a photovoltaic power generation uncertainty model, and integrates pre-defined rules, improve the speed of reinforcement learning training to converge to the optimal charge and discharge control strategy, and design economic rewards, ride comfort rewards, maximum demand power rewards and constraint penalty rewards, improve the user-side electricity economy and reduce the power grid Power fluctuations and burdens.

Figure 9 shows a computer device provided by an embodiment of this specification. The computer device is used to execute the energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning. The computer device 902 may include one or more A processor 904, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. Computer device 902 may also include any memory 906 for storing any kind of information such as code, settings, data, and the like. For example, without limitation, the memory 906 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of computer device 902. In one instance, when processor 904 executes associated instructions stored in any memory or combination of memories, computer device 902 may perform any operation of the associated instructions. Computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive, an optical disk drive, and the like.

Computer device 902 may also include an input/output module 910 (I/O) for receiving various inputs (via input device 912 ) and for providing various outputs (via output device 914 ). One particular output mechanism may include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input/output module 910 (I/O), the input device 912 and the output device 914 may not be included, and may only be used as a computer device in the network. Computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922 . One or more communication buses 924 couple together the components described above.

Communication link 922 may be implemented in any manner, such as through a local area network, a wide area network (eg, the Internet), a point-to-point connection, etc., or any combination thereof. Communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

Corresponding to the methods in Figures 1 to 6, embodiments of this specification also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program executes the above method when run by a processor. step.

Embodiments of this specification also provide computer-readable instructions, wherein when the processor executes the instructions, the program therein causes the processor to perform the methods shown in FIGS. 1 to 6 .

It should be understood that in the various embodiments of this specification, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of this specification. The implementation process constitutes any limitation.

It should also be understood that in the embodiment of this specification, the term "and/or" is only an association relationship describing associated objects, indicating that three relationships can exist. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in this manual generally indicates that the related objects are an "or" relationship.

Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented with electronic hardware, computer software, or a combination of both. In order to clearly illustrate the hardware and software Interchangeability, in the above description, the composition and steps of each example have been generally described according to functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this specification.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling or direct coupling or communication connection between each other shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution in this specification is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this specification. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code.

Specific examples are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification; at the same time, for those of ordinary skill in the art, based on this specification The ideas of the specification may be changed in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as a limitation of this specification.

Claims (11)

1. An energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning, characterized in that the method is applied to the power grid and user-side photovoltaic energy storage equipment, wherein the user-side photovoltaic energy storage equipment includes: photovoltaic power generation device, User electrical devices and energy storage batteries, the method includes: Determine the state space by determining the user's electricity power, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, and grid electricity sales unit price in any period of time; Input the state space into the charge and discharge control decision model based on fusion rule reinforcement learning to obtain the optimal charge and discharge decision variables for any time period, where the optimal charge and discharge decision variables include: optimal charge and discharge of the energy storage battery Power and optimal coefficients, the charge and discharge control decision-making model based on fusion rule reinforcement learning is obtained through sample state space and photovoltaic power generation uncertainty model training. 2. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 1, characterized in that the charge and discharge control decision-making model based on fusion rule reinforcement learning is obtained through the following steps: Determine the sample state space based on the state variables and environmental variables related to user power devices, energy storage batteries, and power grids during the sample time period; Construct a basic model for charge and discharge control decision-making based on the photovoltaic power generation uncertainty model, where the photovoltaic power generation uncertainty model is a Gaussian distribution model that considers environmental variables; Input the state variables in the sample state space into the basic charge and discharge control decision model, and output the charge and discharge decision variables; The charging and discharging decision-making variables are constrained according to the preset charging and discharging rules, and the charging and discharging control decision-making basic model is iteratively trained using a reward function to obtain a fully trained charging and discharging control decision-making model. 3. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 2, characterized in that the state variables of the energy storage battery in the sample time period are determined in the following manner: Obtain the charging power of the energy storage battery in the historical sample period before the sample time period; According to the charging power of the energy storage battery during the historical sample period, the state of charge of the energy storage battery during the sample period is determined by the following formula: ; Among them, k represents the current sample time period, /> It is the state of charge of the energy storage battery in the first sample time period (i.e. the initial moment);/> is the self-discharge rate of the energy storage battery;/> is the open circuit voltage of the energy storage battery;/> is the rated capacity of the energy storage battery;/> is the charging and discharging efficiency of the energy storage battery;/> ,/> ,/> is the charging and discharging power of the energy storage battery in the i -th sample time period. When the energy storage battery is discharged,/> , when the energy storage battery is charging,/> ;/> ,/> , is the efficiency index of the i-th time period, which satisfies:/> . 4. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 2, characterized in that, after outputting the charge and discharge decision variables, the method further includes: Based on the photovoltaic power generation uncertainty model that obeys Gaussian distribution, determine the photovoltaic power generation power in the sample time period; According to the decision variables output by the charge and discharge control decision-making basic model and the difference between the photovoltaic power generation power in the sample time period and the user's power consumption, the power output by the photovoltaic power generation device to the grid during the sample time period is determined. 5. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 4, characterized in that the method includes: The photovoltaic power generation uncertainty model is determined through the following formula: ;wherein,/> is the photovoltaic power generation during the sample time period;/> is the photoelectric conversion efficiency;/> is the photovoltaic material area; Indicates the solar irradiance during the sample time period;/> is the variance; photovoltaic power generation obeys the mean expectation as , the variance is/> Gaussian distribution; according to the photovoltaic power generation power obtained by the photovoltaic power generation uncertainty model, the power output from the photovoltaic power generation device to the grid is determined by the following formula: ;wherein,/> is the power output by the photovoltaic power generation device to the grid in the k-th sample time period;/> It is the user’s power consumption in the k-th sample time period;/> Indicates the photovoltaic power generation of light in the kth sample time period;/> is the proportional weight,/> . 6. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 5, characterized in that determining the preset charge and discharge rules in the following manner includes: If the current period belongs to the peak period of electricity consumption, set the charging and discharging power of the energy storage battery to be negative; If the current period belongs to the low power consumption period, set the charging and discharging power of the energy storage battery to be positive; If the photovoltaic power generation is less than the load power, set the proportional weight to zero; If the photovoltaic power generation is greater than the load power, the proportional weight is set to the original value of the proportional weight; According to the above preset charging and discharging rules, the decision variables in the charging and discharging control decision model are constrained. 7. The energy storage charge and discharge control decision-making method based on fusion rule reinforcement learning according to claim 6, characterized in that the iterative training of the charge and discharge control decision-making basic model using a reward function includes: Determine the economic reward function based on the unit price of electricity sold in each period, the unit price of electricity consumed in each period, and the input power from the power grid in each period; Based on the charging and discharging power of the energy storage battery at the current moment and the charging and discharging power of the energy storage battery at the previous moment, a ride comfort reward function is constructed; According to the input power of the power grid in each period, determine the maximum demand power reward function; According to the minimum state of charge of the energy storage battery, the maximum state of charge of the energy storage battery and the current state of charge of the energy storage battery, determine the constraint penalty reward function; According to the economic reward function, ride comfort reward function, maximum demand power reward function and constraint penalty reward function, the charge and discharge control decision-making model is iteratively trained. When the above reward reaches the maximum value and remains stable within the preset number of iterations, It is determined that the charging and discharging control decision model training is completed. 8. A photovoltaic energy storage system based on fusion rule reinforcement learning, characterized in that the system includes: The power grid is connected to the user-side photovoltaic energy storage equipment for communication and is used to supply power to the user-side photovoltaic energy storage equipment; User-side photovoltaic energy storage equipment includes: photovoltaic power generation device, user power device and energy storage battery, wherein the photovoltaic power generation device is used to provide photovoltaic power generation to the user power device, and is used to store the photovoltaic power when the power is remaining. The remaining power is transferred to the energy storage battery or power grid; The energy storage battery is used to store the remaining power of the photovoltaic power generation device and to provide power to the user's electrical device. 9. An energy storage charge and discharge control decision-making device based on fusion rule reinforcement learning, characterized in that the device includes: The state space construction unit is used to determine the state space by combining the user's electricity power, the state of charge of the energy storage battery, outdoor temperature, solar irradiance, grid electricity unit price, and grid electricity sales unit price in any period of time; The optimal charging and discharging decision variable determination unit is used to input the state space into the charging and discharging control decision model based on fusion rule reinforcement learning to obtain the optimal charging and discharging decision variables for any time period, wherein the optimal charging and discharging The decision variables include: the optimal charge and discharge power and optimal coefficient of the energy storage battery. The charge and discharge control decision model based on fusion rule reinforcement learning is obtained through sample state space and photovoltaic power generation uncertainty model training. 10. A computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, any one of claims 1 to 7 is realized. method described in the item. 11. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.