CN113885328A - Nuclear power tracking control method based on integral reinforcement learning - Google Patents

Abstract

The invention discloses a nuclear power tracking control method based on integral reinforcement learning. The algorithm trains the evaluation network and corrects the network weights. The evaluation network is used to approximate the tracking error performance index function, and the evaluation network weights are used to evaluate the performance of the current tracking error control system, and the optimal control strategy is selected through the execution process to minimize once The total cost of the global iteration; judge whether the current local iteration is completed, if not, return to the local iteration, otherwise update the iterative performance index function and tracking control law to obtain the optimal tracking control strategy; the global strategy iteration is completed, and the optimal tracking control is obtained strategy, track to the desired power point, and calculate the total cost. Therefore, the present invention can continuously learn and adjust the current strategy to track the desired power point.

Description

A nuclear power tracking control method based on integral reinforcement learning

technical field

The embodiments of the present invention relate to the technical field of nuclear power unit power control, and in particular, to a nuclear power power tracking control method based on integral reinforcement learning.

Background technique

In recent years, due to the burning of coal for power generation, the greenhouse effect and air pollution have become increasingly serious, and its resource reserves are also decreasing year by year. As a kind of clean energy, nuclear energy has the advantages of no pollution and low transportation cost. The safety of nuclear power systems has also been concerned by all walks of life, so the regulation of its power has become the focus. A stable, safe and efficient power control method for nuclear power plants is particularly important to the entire nuclear power industry.

In view of this, the present invention is proposed.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention is proposed to provide a nuclear power tracking control method based on reinforcement integral learning that at least partially solves the above problems.

In order to achieve the above object, according to one aspect of the present invention, the following technical solutions are provided:

A nuclear power tracking control method based on reinforcement integral learning, the method includes:

S1: Initial strategy selection, related parameter initialization, initial power point and expected power point selection;

S2: Perform global iteration, and update the iterative tracking error performance index function according to the iterative control sequence to obtain the optimal tracking error performance index function;

S3: Perform local iteration, train an evaluation network by using an integral reinforcement learning algorithm, modify the weights of the evaluation network, and use the optimal tracking error performance index function to obtain an optimal error control strategy;

S4: Determine whether the current local iteration is completed, if not, return to the local iteration step, otherwise update the iterative tracking error performance index function and control law to obtain the optimal tracking error performance index function;

S5: The global strategy iteration is completed, the optimal tracking control strategy is obtained, the desired power point is tracked, and the total cost is calculated.

Compared with the prior art, the above technical solution at least has the following beneficial effects:

The self-learning power tracking controller based on the adaptive dynamic programming algorithm constructed by the neural network in the embodiment of the present invention can continuously learn, adjust and adapt to different nuclear power states through real-time operations, and can track the working conditions of different nuclear power units point.

Description of drawings

The accompanying drawings, as a part of the present invention, are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings in the following description are only some embodiments, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort. In the attached image:

FIG. 1 is a schematic diagram of a nuclear power system model according to an exemplary embodiment;

FIG. 2 is a schematic flowchart of a method for power tracking control of a nuclear power plant based on integral enhancement according to an exemplary embodiment.

Detailed ways

In order to illustrate the objectives, technical solutions and advantages of the present invention more clearly, the present invention will be further described in detail below with reference to specific examples and accompanying drawings.

Adaptive dynamic programming has developed rapidly since it was proposed by Paul J. Werbos in the 1980s. It is mainly used to solve the "curse of dimension" problem in dynamic programming, and its specific solution is to solve it through multiple iterative optimization. In recent years, adaptive dynamic programming algorithms have shown great advantages in solving optimal control. The adaptive dynamic programming method generally uses the controller-evaluator (actor-critic) structure and neural network to approximate the tracking error performance index function and control strategy, and uses an iterative method to gradually approximate the analytical solution of the equation, and finally converges to the optimal tracking Error performance index function and optimal tracking control strategy.

The adaptive dynamic programming method uses the function approximation structure (such as: neural network) to approximate the tracking error performance index function and control strategy in the dynamic programming equation to meet the optimization principle, so as to obtain the system optimal error control and optimal tracking error performance index function . The adaptive dynamic programming structure mainly includes dynamic system, control network and evaluation network. The evaluation network is used to approximate the optimal cost function, and the evaluation guidance is given to execute the network to generate optimal control. After the output of the execution network acts on the dynamic system, the evaluation network is affected by the rewards/penalties generated in different stages of the dynamic system, and the control strategy of the implementation of the network update is known to make the overall cost (that is, the sum of the rewards/punishments) optimal.

The integral reinforcement learning adaptive dynamic programming method does not rely on the system model, but adjusts the weights of the controller and evaluator neural network based on the system state generated in real time and the corresponding control actions. Finally, the integral reinforcement learning adaptive dynamic programming method can run online and make the controller and evaluator neural network finally iteratively converge to the optimal control policy and optimal tracking error performance index function. It is especially suitable for online solution of optimal control problems for linear or nonlinear continuous systems.

FIG. 1 is a schematic diagram of a nuclear power system to which an embodiment of the present invention is applied, which schematically shows a reaction heat transfer model diagram of the nuclear power system. The nuclear power system consists of one reactor and two cooling reactors. Among them, Q only represents heat transfer and has no practical significance for nuclear power system models. The nuclear power system includes five system states, Power percentage indicates the percentage of the power generation of the system (its full-load power generation is 2500MW); Delayed neural concentration indicates the relative concentration of delayed neutrons inside the reactor of the nuclear power system; Reactor core Temperature is the nuclear power The average temperature of the system reactor core (we can also use T _f to represent it); Coolant outlet Temperature represents the average temperature of the coolant inside the nuclear power system; Reactor coefficient represents the reactivity change of the nuclear power system caused by the up and down movement of the control rod. The system only uses the reaction speed of the control rod as a control signal. When the control rod moves up and down at a certain speed, the reaction inside the reactor core will change accordingly. The faster the control stick is moved up, the more responsive it is. The control stick moves down and vice versa.

As shown in FIG. 2 , an embodiment of the present invention provides a power tracking control method for a nuclear power system based on integral reinforcement learning. The method may include step S1 and step S5.

S1: The initialization parameters include: nuclear power system parameters, evaluation network parameters, global iteration duration, integration time constant, local iteration duration, convergence accuracy, and target parameters; wherein, the nuclear power system parameters are power model system parameters of nuclear power, and the The model includes five system input and output states.

The nuclear power system model mainly includes the neutron reaction equation inside the core, two temperature feedback models of the reactor, and the reactivity equation of the control rod. In the research of reactor characteristics, the control rod control method is mostly used. Because the control rod has a strong neutron absorption capacity, and the moving speed is easy to control, the operation is convenient, and the accuracy of the reactivity control is high. Variety.

In addition, it is necessary to select the initial power operating point and the expected power operating point, and to determine an initial stable control strategy. The following parameters are also initialized: global training time step, local iteration time step, neural network structure (such as number of input nodes, number of hidden layer nodes, and number of output layer nodes), neural network weights.

For example, the structure of the evaluation network is set to 5-15-1, where 5 is the number of input nodes of the evaluation network, 15 is the number of hidden layer nodes of the evaluation network, 1 is the number of output layer nodes of the evaluation network, and the number of hidden layer nodes can be based on experience. Adjustments are made for the best approximation, and the convergence accuracy is defined to be 1.0×10 ⁻² .

In the execution stage, the embodiments of the present invention use simplified finite-dimensional control variables, that is, set finite and definite nuclear power operating points for tracking.

In practical applications, the selection of the initial operating point and the expected operating point can be set according to actual needs, and the power model and parameter settings of the nuclear power unit also need to have practical significance.

S2: During global training, update the iterative tracking error performance index function according to the iterative control sequence to obtain the optimal tracking error performance index function;

Specifically, according to the requirements of the integral reinforcement learning method of the controller, it is necessary to perform weight initialization training on the evaluation network.

Use the integral reinforcement learning algorithm to train the evaluation network: the input values of the evaluation network include: five states x (t) of the nuclear power unit operating point, five states x _d (t) of the nuclear power unit expected operating point, and the nuclear power unit tracking error control strategy u _e (t), the output value is the tracking error performance indicator function J _e (t). Among them, the J _e (t) tracking error performance index function is abbreviated as the J function. The optimal tracking error control strategy _ue (t) is approximated by the tracking error performance index function obtained from the evaluation network.

The weight initialization of the evaluation network is performed within the global iteration. Preferably, the weights can be re-initialized at the beginning of each global iteration, so as to better ensure the convergence of the evaluation network on the basis of ensuring the stability and convergence speed of the evaluation network, so as to find the optimal tracking of the nuclear power system power as soon as possible Control Strategy.

In the execution stage, the input data of the evaluation network is the difference x _e (t) between the five state outputs x (t) of the nuclear power unit and the expected power point x _d (t) and the optimal tracking obtained according to the trained evaluation network Error control strategy _ue (t). The output data of the evaluation network is the tracking error performance index function J _e (t).

According to the Bellman equation, the output data _Je (t) at the current moment is calculated by using the output J _e (t+T) of the evaluation network at the next moment and the utility function U (t), and the calculation formula is as follows:

来更新全局迭代J_e函数。Utilize global iterative error control law

to update the global iterative _Je function.

The following example illustrates the process of obtaining the optimal tracking error performance index function in detail.

Set time t, x(t) is the five input and output states of the nuclear power unit at this time, x _d (t) is the expected power point, we have the system tracking error x _e (t), u _e (t) is the tracking error control strategy; its error control system can be defined as:

x _e (t+1)=f(x(t)-x _d (t),u _e (t),t)

where f can be derived from the nuclear power unit power model. Define the utility function as follows:

U(t)=α[x _e (t)] ² +β[u _e (t)] ²

Among them, α and β are constants; u _e (t) is the difference between the control law of the nuclear power unit at the current time and the expected work control law. The utility function U(t) represents the difference between the current operating point and the expected operating point of the nuclear power unit at time t and the sum of the utility of the control rod control law.

We give the utility function a new form:

where Q and R are positive definite matrices, respectively, and our global tracking error performance indicator function can be defined as:

Its Hamiltonian equation can be derived as follows:

使得满足如下等式：then we have a

so that the following equation is satisfied:

Its optimal tracking error control law can be expressed as:

对于

和

我们有Define the initial error control law

for

and

We have

Among them, i=0,1,2,..., then the error tracking control law can be obtained by the following formula:

会收敛于最优值。When T→∞,

will converge to the optimal value.

S3: Perform local iteration, train an evaluation network by using an integral reinforcement learning algorithm, modify the weights of the evaluation network, and use the optimal tracking error performance index function to obtain an optimal error control strategy;

The goal of local training iterations is to obtain the optimal

Given the initial stable control strategy, we let its control law be _ue ⁰ . Let the integration duration T=1, and select the local training iteration duration as 30 steps.

Its tracking error performance indicator function update rule is:

The update rule of the optimal error control law is as follows:

会收敛于最优值

When T→∞,

will converge to the optimal value

Then, the weights of the evaluation network are updated to approximate the optimal tracking error performance indicator function.

Among them, the update rules are as follows:

W _CL = -(X ^T X) ^-1 (X ^T Y)

为评价网络的权重向量偏差，X为评价网络的权重向量内积差值，Y为评价网络近似的效用函数值，W_CL为评价网络的权值。in,

In order to evaluate the weight vector deviation of the network, X is the weight vector inner product difference of the evaluation network, Y is the approximate utility function value of the evaluation network, and W _CL is the weight value of the evaluation network.

Since the error control strategy and the tracking error performance index function change with the weights of the controller and the evaluator neural network, adjusting the weights of the controller and the evaluator neural network means that the error control strategy and the tracking error performance index function are not equal to each other. renew. In the execution phase, the limited control variables are substituted into the optimal tracking error performance indicator function approximated by the evaluation network

The optimal error control strategy is approximated according to the tracking error performance index function obtained by the evaluation network, and the control variable that minimizes the optimal tracking error performance index function is selected as the optimal tracking error control strategy:

The evaluation network is used to approximate the optimal tracking error performance index function, and the evaluation network weight is used to evaluate the performance of the nuclear power control rod system, and the optimal tracking control strategy is selected through the execution process to minimize the total cost of the global training tracking error.

S4: Determine whether the current local iteration is completed, if not, return to the local iteration step, otherwise update the iterative tracking error performance index function and error control law to obtain the optimal tracking error performance index function;

Specifically, after completing the local iteration, it is determined whether the current number of iterations reaches the iteration threshold, and if so, the iterative tracking error performance index function and error control law are updated to obtain the optimal tracking error performance index function and optimal error control strategy.

If not completed, go to step S3; otherwise, go to step S5.

S5: The global strategy iteration is completed, the optimal tracking error control strategy is obtained, the desired power point is tracked, and the total cost (tracking error and control rod control cost) is calculated.

代入实际模型，这里由于效用函数U(x_e,u_e)的定义依赖于实际模型，所以总成本可近似为最终得到的最优性跟踪误差性能指标函数

The calculation of the total cost requires the optimal tracking error control strategy

Substitute into the actual model, where since the definition of the utility function U(x _e , u _e ) depends on the actual model, the total cost can be approximated as the final optimal tracking error performance index function

Although the steps are described in the above-mentioned order in this embodiment, those skilled in the art can understand that, in order to achieve the effect of this embodiment, different steps need not be performed in this order, and they can be performed simultaneously ( parallel) or in reverse order, simple variations of these are within the scope of the present invention. The technical solutions provided by the embodiments of the present invention have been described in detail above. Although specific examples are used to illustrate the principles and implementations of the present invention, the descriptions of the above embodiments are only suitable for helping to understand the principles of the embodiments of the present invention; meanwhile, for those skilled in the art, according to this Changes may be made in the embodiments of the invention within the specific implementation manner and application scope.

It should be noted that the flowcharts involved in this document are not limited to the forms shown in this document, and may also be divided and/or combined.

It should be noted that the symbols and characters in the accompanying drawings are only for the purpose of illustrating the present invention more clearly, and are not regarded as improper limitation of the protection scope of the present invention.

The present invention is not limited to the above-mentioned embodiments, and any modifications, improvements or substitutions that can be conceived by those of ordinary skill in the art without departing from the essence of the present invention fall into the protection scope of the present invention.

Claims (8)

1. a nuclear power system power tracking control method based on integral reinforcement learning, is characterized in that, described method comprises: S1: Initial strategy selection, related parameter initialization, initial power point and expected power point selection; S2: Perform global iteration, and update the iterative tracking error performance index function according to the iterative control sequence to obtain the optimal tracking error performance index function; S3: perform local iteration, train an evaluation network by using an integral reinforcement learning algorithm, modify the weights of the evaluation network, and obtain an optimal tracking control strategy by using the optimal tracking performance index function; S4: Determine whether the current local iteration is completed, if not, return to the local iteration step, otherwise update the iterative tracking error performance index function and tracking control law to obtain the optimal tracking error performance index function; S5: The global strategy iteration is completed, the optimal tracking control strategy is obtained, the desired power point is tracked, and the total cost is calculated. 2. The method according to claim 1, wherein in the step S1, the initialization parameters include: nuclear power system parameters, evaluation network parameters, global iteration duration, integration time constant, local iteration duration, convergence accuracy and target parameters; wherein, the nuclear power system parameters are power model system parameters of nuclear power, and the model includes five system input and output states. 3. The method according to claim 2, wherein the structure of the evaluation network is set as 5-15-1, and the convergence accuracy is defined as 1.0× ^10-2 , wherein 5 is the number of input nodes of the evaluation network , 15 is the number of nodes in the hidden layer of the evaluation network, and 1 is the number of nodes in the output layer of the evaluation network. 4 . The method according to claim 1 , wherein the step S1 further comprises the selection of an initial control strategy, and the error control strategy can be obtained by a traditional PID or MPC strategy, in order to obtain an initial stable control rate. 5 . 5 . The method according to claim 1 , wherein, in the step S3 , the input data of the evaluation network includes five working states x(t) of the nuclear power unit and the working state point x _d of the expected power. 6 . (t) of the tracking error value x _e (t), and the tracking control strategy ue (t) of the nuclear power control rod; the output data of the evaluation network includes: the tracking error performance index function J _{e (} t); According to the Bellman equation, the output J _e (t+T) and the utility function U(t) of the evaluation network at the next integration moment are used, and the output data J _e (t) at the current moment is calculated by the following formula:

Among them, x _e (t) is the tracking error value x _e (t) between the five working states x (t) of the nuclear power unit and the working state point x _d (t) of the expected power; the utility function U(t) represents the time t The utility sum of the tracking error value x _e (t) and the tracking control strategy _ue (t) of the nuclear power control rod. 6. The method according to claim 5, wherein the calculation formula of the utility function U(t) is: U(t)=α[x _e (t)] ² +β[u _e (t)] ² Among them, α and β are constants; u _e (t) is the difference between the control law of the nuclear power unit at the current time and the expected work control law. 7 . The method according to claim 1 , wherein, in the step S3 , the input data of the execution stage of the evaluation network includes the relative power coefficient of the controlled nuclear power unit, the relative concentration of delayed neutrons, the reactor Average core temperature, average temperature of coolant and reactivity of control rods; the output data of the execution phase of the evaluation network includes an optimal tracking control strategy; wherein, the optimal tracking control strategy is obtained according to the evaluation network The tracking error performance indicator function is approximated by . 8. The method according to claim 1, wherein, in the step S3, the update rule of the evaluation network is as follows:

W _CL = -(X ^T X) ^-1 (X ^T Y) in,

In order to evaluate the weight vector deviation of the network, X is the weight vector inner product difference of the evaluation network, Y is the approximate utility function value of the evaluation network, and W _CL is the weight value of the evaluation network.

Images