RU2458390C1 - Modified intelligent controller - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: modified intelligent controller includes a control object, a solver, where the output of the control object is connected to the first input of the solver, a support computing unit, an action unit, a Kalman filter, a Kalman filter memory and an action selection unit.

EFFECT: faster operation of the device.

1 dwg

Description

The invention relates to the class of intelligent controllers that use the principle of reinforcement learning, and can be used to create systems for managing objects operating in a non-deterministic environment.

Known US patent IPC G06F 15/18 6532454 "Stable adaptive control using critic designs", which implements reinforced learning using neural networks. The device according to this patent consists of a solver simulating a neural network, a block of criticism, and also a block for calculating forecast errors and connections between blocks.

The principle of operation of the device according to U.S. IPC G06F 15/18 6532454 is as follows - the solver (the neural network acts in its capacity) receives the reinforcement value, calculates the action at this iteration and passes it to the modeling neural network, which calculates the predicted value of the system operating parameter. After the action is completed, the system receives the real value of the working parameter, the critic calculates the new reinforcement value and corrects the operation of the modeling neural network.

Also known is an intelligent controller based on adaptive criticism networks - US patent IPC G06F 15/18 5448681. This device consists of a control object, a critic unit and a solver based on a neural network. The outputs of the control object are connected with the inputs of the critic block, as well as the inputs of the critical neural network, the output of the critical neural network is connected with the control object and the block of the critic, the output of the critic network is connected with the input of the critical neural network.

The principle of operation of the device according to the IPC patent G06F 15/18 5448681 is as follows - the control object gives a signal about its state, the critic unit predicts the quality of the selected action based on the current time iteration and the state of the object, the solver calculates the control action.

The disadvantages of the devices according to the patent of IPC G06F 15/18 6532454 are that they do not remember the history of the system and the critic works according to the initially configured parameters.

The disadvantages of the devices according to the patent of IPC G06F 15/18 5448681 are - there is no block for storing the history of the system and low adaptive properties due to the rigidly set principle of operation of the critic block.

A common disadvantage of the two devices described above is the lack of performance due to the use of neural networks.

The technical result of the proposed device is to increase the speed parameters.

The task is to develop a modified intelligent controller with high speed performance characteristics.

The technical result is achieved by the fact that in a modified intelligent controller containing a control object, a solver, the output of the control object is connected to the first input of the solver, a reinforcement calculation block, an action block, a Kalman filter, a Kalman filter memory, an action selection block are introduced, and the control object output also connected to the input of the reinforcement calculation block and the first input of the action block, the first output of the action block is connected to the second input of the solver, the second output of the action block is connected to the third input of the Kal filter mana and the first input of the action selection block, the output of the reinforcement calculation block is connected to the second input of the action block and the second input of the Kalman filter, the solver output is connected to the first input of the Kalman filter, the first output of the Kalman filter is connected to the second input of the action selection block, the second output of the Kalman filter is connected with the first input of the Kalman filter memory, the output of the Kalman filter memory is connected to the fourth input of the Kalman filter, the first output of the action selection block is connected to the input of the control object and the third input of the action block, the second output action selection unit coupled to the second input of the storage Kalman filter.

Increasing the speed parameters of the device is achieved due to the fact that instead of neural networks, a Kalman filter is introduced into the device to predict reinforcements. In this case, the Kalman filter memory is used, which saves the Kalman filter parameters for the selected action with the subsequent restoration of the Kalman filter parameters without additional calculations. Separation of the action block into a separate component also leads to an increase in speed characteristics, since it accumulates the history of the control object for subsequent use.

Thus, the totality of the existing features set forth in the claims, allows to achieve the desired technical result.

The drawing shows a diagram of a modified intelligent controller with a Kalman filter.

The system consists of several structural components: control object 1, solver 2, reinforcement calculation block 3, action block 4, Kalman filter 5, Kalman filter memory 6 and action selection block 7.

Also in the system there are the following connections: output 8 of the control object is connected with the first input 8.1 of the solver, the first input 8.2 of the action selection block, input 8.3 of the reinforcement calculation block; the output 9 of the reinforcement calculation unit is connected in accordance with 9.1 with the action block and in 9.2 with the Kalman filter, the output of the 10 action block is connected with the solver, the output of the solver is connected with the Kalman filter through communication 11, the output of the 12 action block is connected 12.1 with the action and communication block 12.2 with Kalman filter, Kalman filter output 13 is connected to Kalman filter memory, Kalman filter memory 14 is connected to Kalman filter, Kalman filter 15 is connected to action selection block, action selection block is connected 16 to Kalman filter memory, output 17 of action selection blockoedinen at 17.1 with action block and 17.2 with the control object.

Solver 2 is a device that implements a mathematical formula (or several formulas) that describes those variables of the control object that can be directly calculated.

The reinforcement calculation block 3 implements a mathematical formula that calculates the real value of the reinforcement after the action (control) signal 17.2 is worked out by the control object 1.

Action block 4 stores a table of possible actions in specific situations.

Kalman filter 5 is designed to calculate an unobservable value. The Kalman filter is performed as standard, for example, according to US patent IPC G06F 15/20 No. 5115391.

The memory of the Kalman filter 6 is intended for temporary storage of the parameters of the Kalman filter block 5. The block stores as many sets of Kalman filter parameters as the possible actions are selected in the action block 4.

The action selection block 7 is intended to select an action from possible in a given situation based on a "greedy rule".

, при этом 0<ε<<1 (Sutton R., Barto A. Reinforcement Learning: An Introducion. - Cambridge: MIT Press, 1998). После выбора действия сигнал 17 идет на объект управления 1 и записывается в блок действий 4, а также на память фильтра Калмана 6, которая восстанавливает параметры фильтра Калмана 5 для выбранного действия.The principle of operation of the intelligent controller is as follows. The control object 1 performs an action and forms a status signal 8 at the output. Next, the status signal is sent to action block 4, reinforcement calculation block 3 and solver 2. Reinforcement calculation block 3 calculates the real resulting reinforcement 9 and feeds it to action block 4 and Kalman filter 5 Solver 2, in turn, processes the signal from the control object and generates the observed signal 11 at the output. Having received the status signal, action block 4 selects possible actions in this situation, taking into account real reinforcement at the last teratsii and outputs them sequentially to the Kalman filter 5, thus it takes the synchronizing signal 10 on the resolver 2 so that it simultaneously acts to block gave the observed value the value in the Kalman filter 11 5. 5 Kalman filter sequentially receives a pair of values {11 observed signal; possible action 12.2} and calculates the possible reinforcement 15 for each possible action. After calculating the reinforcement, the Kalman filter gives the value of reinforcement 15 to the action selection block 7, which also takes action 12.1 from the action block 4, which was taken into account when calculating the unobserved signal (reinforcement) by the Kalman filter 5. Also, the current Kalman filter parameters are stored in the memory of Kalman filter 6 5. The action selection block 7 selects the action based on the “greedy” rule, which can be written as: with probability (1-ε), the action is selected that corresponds to the maximum reinforcement value

with 0 <ε << 1 (Sutton R., Barto A. Reinforcement Learning: An Introducion. - Cambridge: MIT Press, 1998). After selecting the action, the signal 17 goes to the control object 1 and is recorded in the action block 4, as well as to the memory of the Kalman filter 6, which restores the parameters of the Kalman filter 5 for the selected action.

Thus, the algorithm of the device is as follows (only signals are indicated by numbers):

1. The control object calculates its state signal 8 (both on the basis of information on the external environment and on its own indicators) and submits it 8.1 to the solver, 8.2 to the action block and 8.3 to the reinforcement calculation block.

2. The solver calculates the observed parameter 11 of the system.

3. The reinforcement calculation unit calculates the resulting (real) reinforcement 9 and submits its value to 9.1 - the action block and 9.2 - Kalman filter.

4. The action block, taking into account the latest reinforcement 9.1, selects the possible actions in this particular situation 12.

5. The action block sequentially supplies the selected actions to 12.2 — the Kalman filter and the synchronizing signal 10 to the solver, by which the solver synchronously supplies the observed parameter 11 to the Kalman filter.

6. At the first applied signal for calculation 12.2, before starting work, the Kalman filter saves its parameters of 13 in the memory of the Kalman filter.

7. The Kalman filter sequentially receives pairs of values {observed signal 11; possible action 12.2} and calculates the reinforcement forecast (unobserved signal) 15. The Kalman filter works in a standard way.

8. After calculating the reinforcement forecast for each possible action 12.2, the Kalman filter saves its parameters in the Kalman filter memory by 13 and provides the value of the predicted reinforcement 15 to the action selection block.

9. The action selection block accumulates value pairs {possible action 12.1; predicted reinforcement 15}.

10. After the reinforcements are calculated for all possible actions, a signal 12.1 is sent from the action block to the action selection block about the end of forecasting. After receiving this signal, the action selection block selects the action based on the “greedy rule” and feeds it to 17.1 to the control object, 17.2 to the action block, and 16 to the Kalman filter memory. According to 17.2, the predicted reinforcement for the selected action is also fed to the action block.

11. The action block saves the selected signal 17.1, possible reinforcement 17.1, the state of the control object 8.2, real reinforcement 9.1, thereby accumulating a history for further selection of actions in possible situations.

12. The Kalman filter memory restores 14 Kalman filter parameters for the selected action.

13. The control object fulfills the submitted action 17.2. Next, cycle to step 1.

Claims (1)

A modified intelligent controller containing a control object, a solver, the output of the control object is connected to the first input of the solver, characterized in that it includes a reinforcement calculation block, an action block, a Kalman filter, a Kalman filter memory, an action selection block, while the output of the control object is also connected to the input of the reinforcement calculation block and the first input of the action block, the first output of the action block is connected to the second input of the solver, the second output of the action block is connected to the third input of the Kalman filter and the first input the house of the action selection block, the output of the reinforcement calculation block is connected to the second input of the action block and the second input of the Kalman filter, the solver output is connected to the first input of the Kalman filter, the first output of the Kalman filter is connected to the second input of the action selection block, the second output of the Kalman filter is connected to the first input Kalman filter memory, the Kalman filter memory output is connected to the fourth input of the Kalman filter, the first output of the action selection block is connected to the input of the control object and the third input of the action block, the second output of the de Corollary connected to the second input of the storage Kalman filter.