KR20200082909A - Imitational reinforcement learning system for controlling devices remotely and the method thereof - Google Patents

Abstract

The present invention relates to an imitation reinforcement learning system for remote device control and to a method thereof. Provided are the imitation reinforcement learning system for remote device control, which utilizes a traditional machine control model defined in advance to control a specific device to generate and store transition information for the specific device, and imitates the stored transition information, so that the reinforcement learning for the reinforcement learning model for controlling the specific device can be performed through initial stable data. When trying to control the device exposed to various environments through the reinforcement learning model, the device can be accurately and effectively controlled by rapidly adapting to the environment, and the method thereof.

Description

IMITATIONAL REINFORCEMENT LEARNING SYSTEM FOR CONTROLLING DEVICES REMOTELY AND THE METHOD THEREOF}

The present invention relates to an imitation reinforcement learning system for remote control of a device and a method therefor, using a traditional machine control model previously defined to control a specific device, and generating transition information for the specific device and By storing and imitating the stored transition information, it is possible to perform reinforcement learning on the reinforcement learning model for controlling the specific device through initial stable data, thereby exposing the environment to various environments through the reinforcement learning model. When it is desired to control a device, the present invention relates to an imitation reinforcement learning system and a method for remote control of a device to quickly adapt to the corresponding environment and to precisely and effectively control the device.

Due to the recent rapid development of industrial technology and information and communication technology, ICT-based real physical and cyber environments in various fields such as big data analysis and processing field, robotics field, IoT field, unmanned vehicle transport field, etc. By fusion, all products of the real world were connected to the network and became intelligent, so the 4th industrial revolution occurred to provide users with more convenient and suitable services.

Machine learning such as deep learning is at the center of this fourth industrial revolution. In particular, reinforcement learning, a field of machine learning, involves selecting an action or sequence of actions that maximizes the reward among the selectable actions by recognizing the current state of a specific device located in a real physical environment by an agent defined in a specific environment. Through the development of the behavioral mode of the agent, control of a device located in the physical environment is optimized, so that the device can be efficiently controlled.

However, the reinforcement learning is overwhelmingly used in arcade game simulation rather than being used in a control system for controlling a real device.

The reason is that it is very difficult to construct a reinforcement learning network because it is necessary to grasp the high-dimensional environmental state of the real physical environment.

In addition, the conventional reinforcement learning method for controlling the device generates a control command for controlling the device by directly applying the reinforcement learning network with randomly set weights to the real environment, and results according to the generated control command. According to the reinforcement learning is performed. This is because of the non-linear complexity of the actual physical environment, it takes a very long time for the reinforcement learning network to have perfection to adapt to the physical environment and precisely control the device, and furthermore, Since it does not generate an accurate control command for controlling a device according to a preset episode, it implies that the device cannot be precisely controlled.

Therefore, in the present invention, by utilizing the traditional machine control model, by controlling the device through the traditional machine control model, the current state information, control information, reward information of the device and transition information of the device including the next state information To generate and store, and to rapidly adapt to the physical environment of the device by performing reinforcement learning by imitating a series of control processes for the device performed through the traditional machine control model based on the stored transition information. We would like to propose a technical solution that enables precise control from a remote location.

That is, the present invention allows the device to be rapidly adapted to the actual physical environment in which the device is located, by enabling rapid reinforcement learning based on the initial stable data according to the stored transition information by utilizing the traditional machine control model. It is intended to provide a reinforcement learning system and a method that can be precisely controlled remotely.

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters to be achieved differently from the prior art will be described.

First, Korean Patent Publication No. 2018-0034553 (2018.04.04.) relates to continuous control using in-depth reinforcement learning, receives observations that specify each state of the environment, and responds to the observations to perform the next action from the action neural network. Select and perform the action, and receive the status of the environment switching and the compensation for the environment as a result of performing the action, and generate an experience tuple including the current observation, the selected action, the reward, and the next observation. After storing it, it relates to continuous control using deep reinforcement learning to perform reinforcement learning for the action neural network using the stored experience tuple.

That is, the prior art observes the current state of a specific environment, generates an action for this, observes the state information after the environment has changed for the generated action, sets compensation for the action, and then After accumulating and storing experience tuples including the current state, action, reward, and next state, reinforcement learning is performed on an action neural network for generating the action using the stored test tuples.

In addition, Korean Patent Publication No. 2018-0137562 (2018.12.27.) relates to a method and apparatus for pruning experience memory for deep neural network-based cue-learning, and provides information about the environment through an agent that interacts with the environment. After collecting a first state, inputting the collected first state to a neural network, and generating an operation for the first state, providing it to the environment, and then compensating for the second state and compensation for the operation from the environment By receiving, after forming a new experience including the first state, operation, reward, and second state, the similarity between the new experience and the previously stored experience is calculated, and the new experience is stored in the existing A method and apparatus for pruning empirical memory for deep neural network based cue-learning to relieve memory size constraints by discarding the new wariness, in a very similar case to experience.

The prior art generates an operation for a specific state of the environment through a neural network, applies it to the environment, receives a compensation for it from the environment, and stores the experiences created based on the memory in the memory, By applying the experience of the stored environment to the neural network, the neural network is reinforced learning.

As described above, since most of the prior arts generate an action on the environment, it is entirely dependent on the learning of the initial neural network, so it takes a long time for the neural network to adapt to the environment, and the learning is insufficient. There is a problem in that it is impossible to accurately control the environment.

On the other hand, the present invention adapts to the actual physical environment in which a specific device is located by quickly collecting initial stable data for controlling the device through a traditional control model and performing fast reinforcement learning based on the collected data. At the same time minimizing the time to be performed, and to enable precise and effective control of the device, the prior arts do not describe or suggest this technical feature of the present invention.

The present invention has been devised to solve the above problems, and the device is rapidly adapted to the physical environment through an enhanced learning model for device control that can remotely and automatically control a plurality of devices located in the physical environment in a cyber environment. It is an object of the present invention to provide an imitation reinforcement learning system and a method for remote control of a device that enables accurate and efficient control of the device.

In addition, according to the present invention, the current state information for a specific device through a traditional device control model, control information for controlling the device in the current state, reward information for a state in which the device has changed based on the control information, and the changed state Transition information of the device including the following state information is collected and stored in an experience memory, and reinforcement learning is performed by imitating the stored transition information, thereby performing rapid reinforcement learning based on the initial stable data and promptly returning to the physical environment. Another object of the present invention is to provide an imitation reinforcement learning system and a method for remotely controlling a device to adaptively control a device located in the physical environment.

In addition, according to the present invention, when performing reinforcement learning to control the device according to a preset episode, when the transition information stored in the experience memory is less than a preset number, control of the device through the traditional device control model By performing it, the imitation reinforcement learning system for remote control of a device to collect the stable and excellent transition information for the device located in the physical environment effectively and quickly perform the reinforcement learning through the initial stable transition information, and Another object is to provide the method.

In addition, according to the present invention, after performing reinforcement learning on a reinforcement learning model for device control for remotely controlling the device based on transfer information collected through the traditional control model, the device is controlled through the reinforcement learning model for device control. Generates control information for the current status information, controls the device, collects and stores the transfer information about it in the experience memory, and controls the device based on the transfer information stored in the experience memory whenever the episode ends Providing an imitation reinforcement learning system and a method for remotely controlling a device to enable the reinforcement learning model for reinforcement learning model to advance the reinforcement learning model for controlling the device and to precisely control the device For another purpose.

In addition, according to the present invention, the experience memory is composed of a first in first out (FIFO) memory structure, and the transfer information is stored in the FIFO format as many as a preset number, so that specific transfer information is repeated in the reinforcement learning. Another object is to provide an imitation reinforcement learning system for remote control of a device and a method for minimizing the use thereof, thereby maximizing the efficiency of reinforcement learning.

In addition, according to the present invention, when transmitting/receiving the state information, control information, and transition information, the traffic on the network for the state information, control information, and transition information is adjusted to have the highest priority, so that the device is real-time without network delay. Another object of the present invention is to provide an imitation reinforcement learning system and a method for remote control of a device to enable control.

The imitation reinforcement learning system for remote control of a device according to an embodiment of the present invention is a first in first FIFO (transition information) generated based on a control result for at least one device located in a physical environment according to a preset episode. and an emulation reinforcement learning unit for performing reinforcement learning on the reinforcement learning model for device control for controlling the device step by step using the experience memory stored in an out) manner and the transition information stored in the experience memory, wherein the transition information is It is characterized by including the current status information of the device, control information for controlling the device, the next status information of the device controlled according to the control information, and reward information for the next status information.

In addition, the imitation reinforcement learning system, when the transition information is stored in the experience memory less than a preset number, by controlling the device according to the episode through a PID controller using a PID control model, the transition information By generating, storing in the experience memory, the imitation reinforcement learning unit, the transfer information generated and stored through the PID controller becomes the preset number, or the transfer information is transferred through the PID controller. By performing the reinforcement learning based on the transfer information generated and stored through the PID controller whenever it is stored in the experience memory or when the episode ends, the experience of the PID controller controlling the device is mimicked by the Characterized in that it further comprises performing reinforcement learning.

In addition, the imitation reinforcement learning system, when the transition information is stored in the experience memory in a preset number, inputs the state information of the device into the reinforcement learning model for controlling the device that performs the imitation reinforcement learning to control the device And by extracting the control information for, further comprising a reinforcement learning agent for controlling the device step by step according to the episode, the reinforcement learning agent receives device status information from the device controlled based on the control information , Generating transition information for the device and storing it in the FIFO method in the experience memory.

In addition, the reinforcement learning is performed to extract control information in which the sum of the reward information is maximum to the final state of the device, which is preset according to device state information for a specific state of the device, and the experience memory is the transition By storing the information in a FIFO manner, it is characterized in that the transition information is prevented from being applied repeatedly during the reinforcement learning.

In addition, the reward information is applied by applying a positive compensation value when the state information of the device is a control result within a preset threshold range, and by applying a negative compensation value when the control result exceeds a preset threshold range. It is characterized by being generated.

In addition, the imitation reinforcement learning method for remote control of a device according to an embodiment of the present invention is a first in FIFO (transition information) generated based on a control result for at least one device located in a physical environment according to a preset episode. and a imitation reinforcement learning step of performing reinforcement learning on a reinforcement learning model for device control for controlling the device using a storage step of storing in an experience memory in a first out method and transition information stored in the experience memory, wherein the The transition information includes current status information of the device, control information for controlling the device, next status information of the device controlled according to the control information, and reward information for the next status information.

In addition, in the imitation reinforcement learning method, when the transition information is stored in the experience memory in less than a preset number, the device is controlled according to the episode through a PID controller using a PID control model to control the transition information. By generating, storing in the experience memory, and in the imitation reinforcement learning step, the transition information generated and stored through the PID controller becomes the preset number, or the transition information is transmitted through the PID controller. Each time it is stored in the experience memory, or when the episode ends, the reinforcement learning is performed based on the transfer information generated and stored through the PID controller to mimic the experience of the PID controller that controls the device. It characterized in that it further comprises performing the reinforcement learning.

In addition, in the imitation reinforcement learning method, when the transition information is stored in the experience memory in a preset number, the reinforcement learning agent inputs state information of the device into the reinforcement learning model for controlling the device that has performed the reinforcement learning. Further comprising, by extracting control information for controlling the device, the step of controlling the device in accordance with the episode; further comprising, the reinforcement learning agent, device status information from the device controlled based on the control information And receiving, generating transition information for the device and storing it in the FIFO method in the experience memory.

As described above, the imitation reinforcement learning system and method for remote control of a device of the present invention collect stable transfer information for the device through a traditional control model, and remotely control the device based on the collected stable transfer information By performing reinforcement learning on the reinforcement learning model for device control, there is an effect of quickly adapting to the physical environment of the device and precisely controlling the device.

In addition, by storing the transition information in an FIFO format through an experience memory having a FIFO structure, it is possible to minimize the repetitive application of specific transition information to the reinforcement learning, so that the reinforcement learning can be efficiently performed.

In addition, by automatically adjusting the priority for the network traffic of the related data necessary for the reinforcement learning to have the maximum priority, there is an effect to control the device in real time.

FIG. 1 is a view schematically illustrating an imitation reinforcement learning system and method for remote control of a device according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a method for performing imitation reinforcement learning for remote control of a device through a device remote control apparatus and an imitation reinforcement learning unit according to an embodiment of the present invention.
3 is a diagram showing the structure of a reinforcement learning network according to an embodiment of the present invention.
4 is a block diagram showing the configuration of a device remote control apparatus according to an embodiment of the present invention.
5 is a view showing the configuration of the imitation reinforcement learning unit of the imitation reinforcement learning system for remote control of a device according to an embodiment of the present invention.
6 is a flowchart illustrating a procedure for performing reinforcement learning according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the imitation reinforcement learning system and method for remote control of a device of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals in each drawing denote the same members. In addition, specific structural or functional descriptions of the embodiments of the present invention are exemplified for the purpose of describing the embodiments according to the present invention, and are used herein, including technical or scientific terms, unless otherwise defined. The terms have the same meaning as generally understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined herein. It is desirable not to.

FIG. 1 is a view schematically illustrating an imitation reinforcement learning system and method for remote control of a device according to an embodiment of the present invention.

As shown in FIG. 1, the imitation reinforcement learning system 10 for remote control of a device according to an embodiment of the present invention includes at least one device 400 located in a physical environment, at least located in the physical environment Device remote control device 100 for remotely controlling one or more devices 400 in a cyber environment, device control for extracting control information for the device 400 so that the device 400 can be controlled remotely By creating a reinforcement learning model and performing reinforcement learning on the created reinforcement learning model for device control to advance the reinforcement learning model for controlling the device, an imitation reinforcement learning unit for precisely controlling the device 400 ( 200), including a network controller 300 and a user terminal 500 for managing and coordinating traffic of a network for transmitting and receiving data between the device remote control apparatus 100 and the device 400 through a network. .

Meanwhile, the plurality of devices 400 are located in various physical environments, such as a smart factory or outdoor, indoor, and various sensors for sensing and collecting specific information such as a motor, an actuator, and the like. With, it is intended to perform a unique function, means a variety of devices that are controlled remotely through the imitation reinforcement learning system 10 for remote control of the device of the present invention.

For example, the device 300 is a rotary inverted pendulum for erecting an inverted pendulum, or a robot arm for moving a specific object to a preset position, or a specific space automatically It may be various devices such as a cleaning robot for cleaning.

That is, the device 300 is connected to the device remote control device 100 for remotely controlling the device 400 through a network, and controls the device remote control device 100 according to a predetermined episode. Through this, it means various devices that perform their unique functions.

In addition, the device remote control apparatus 100 is for device control for extracting control information for remotely controlling the device 400 according to a preset episode from the imitation reinforcement learning unit 200 implemented in the form of a server. Receiving a reinforcement learning model, it performs a function of controlling the device 400 based on the provided reinforcement learning model for device control.

Further, the device remote control apparatus 100 performs interaction with the device 300 to collect actual physical state information from each device 300, and based on the collected state information, the device 400 ) Can be configured to include a reinforcement learning agent 120 and a PID controller 110 to allow remote control.

The reinforcement learning agent 120, according to the state information of the device 400 through the reinforcement learning model for device control provided by the imitation reinforcement learning unit 200 to remotely control the device 400 Control information for controlling the 400 is extracted and transmitted to the device 400 so that the device 400 can be remotely controlled.

That is, while the reinforcement learning agent 120 stores the reinforcement learning model for device control in its own memory (not shown), the device 400 located in a physical environment is previously used by using the reinforcement learning model for device control. When it is desired to control the device 400 according to a set episode, the reinforcement learning agent 110 generates a control command for initializing the device 400 and transmits it to the device 400, so that the device 400 ), and receives the status information from the device 400.

Also, the reinforcement learning agent 120 inputs state information according to the received initialization into the reinforcement learning model for controlling the device, extracts control information for controlling the device 400 according to the entered state information Provided to the device 400, and receives the status information after being controlled according to the control information from the device 400.

In this case, the reinforcement learning agent 120 calculates a reward for the status information when the currently transmitted status information is a result of controlling the device 300 according to the control information provided to the device 400. When the received status information is the initial status information according to the initialization, the reward is not calculated.

In addition, the reinforcement learning agent 120 generates experience information for the device 400 based on the received status information, and experiences memory provided in the imitation reinforcement learning unit 200 (not shown) By storing in, it is possible to perform reinforcement learning on the reinforcement learning model for controlling the device by the imitation reinforcement learning unit 200.

Meanwhile, the transition information includes current state information, control information, reward information, and next state information of the device 400, and the current state information is a state before the device 400 is controlled based on the control information. Information, and the next state information refers to state information of the device 400 that is controlled and transferred to a specific state based on the control information.

In addition, the reward information means a compensation value for the next state information for the device 400 controlled based on the control information.

Meanwhile, the current state information and the next state information of the device 400 included in the transition information may be variously set according to the type of the device 400, and a motor provided to operate the device 400 It may include an angle, angular velocity, or location information of the device 400.

In addition, the control information may also be variously set according to the type of the device 400, and may include an amount of power and a fuel required to drive the device 400 according to a preset episode.

In addition, the reward information means that it is determined whether the device 400 is properly controlled within a predetermined threshold range according to the control information, and is given as a preset compensation value according to the determination result. .

For example, when the device 300 is a rotary inverted pendulum that erects an inverted pendulum, the current state information and the next state information are angular velocity, angle, and angle of the motor provided in the rotary inverted pendulum device. The angular velocity and angle of an inverted pendulum may be included, and the control information may include an amount of power for operating the motor, and the reward information may include an angular speed, angle, and inverted pendulum of the motor according to the control information. When the angular velocity and angle are operated within a preset range, a positive compensation value (for example, +1) may be provided, and when operated in excess of the preset range, a negative compensation value (eg -100) Can be given as

On the other hand, when the device 400 is to be controlled according to a preset episode through the reinforcement learning model for controlling the device, the reinforcement learning model for device control generated initially does not have many times of reinforcement learning, and the reinforcement learning model for controlling the corresponding device Since the weight of is randomly or randomly set according to a user's selection, the control information extracted to control the device 400 according to the state information of the device 400 by adapting to the environment of the device 400 Accuracy may decrease.

Accordingly, the present invention accumulates and stores a preset number of transition information in the experience memory through the PID controller 110, and enables rapid reinforcement learning based on the initial stable transition information. The device 400 is quickly adapted to the physical environment to precisely control the device 400.

In this case, when the device remote control apparatus 100 wants to control the device 400 according to a preset episode for the device 400, it is preset in the experience memory of the imitation reinforcement learning unit 200 When a single number of transition information is not stored, control of the device 400 is preferentially performed through the PID controller 110.

On the other hand, the PID controller 110 is for controlling the device 400 through a PID control model (proportional integral derivative), which is a traditional control model, and reinforced learning of the reinforcement learning model for controlling the device initially. It is intended to quickly collect the initial data for.

On the other hand, the PID control model is defined in advance by a user, and is for stably controlling the device 400. The reference value is stored by presetting an output value (that is, status information) of a specific device 400 to be controlled. Alternatively, it means a mathematical model that calculates an error by comparing with a set value, and calculates a value necessary for control of the device 400 using the calculated error value.

The PID control model is for quickly collecting transition information, which is initial data for reinforcement learning of the reinforcement learning model for device control, and may be defined and built in advance for each device 400 by a user.

In addition, the PID controller 110 receives status information of the device 400 until a preset episode of the corresponding device 400 ends, and generates control information for the status information according to the PID control model. And controls the corresponding device 400 by providing the generated control information to the device, receives status information according to the control result from the device 400, and receives rewards for status information according to the control result. By calculating, the transfer information for the device 400 is generated and stored in the experience memory of the imitation reinforcement learning unit 200.

In addition, the device remote control apparatus 100, when a preset number of transition information is stored in the warp memory, the device remote control apparatus 100 uses the transition information stored by the PID controller 110 By doing so, the imitation reinforcement learning unit 200 can perform reinforcement learning on the reinforcement learning model for controlling the device.

That is, the device remote control apparatus 100, through the PID controller 110, quickly collects transition information for reinforcement learning for the reinforcement learning model for controlling the device at an early stage, and is collected by the PID controller 110. By enabling the initial reinforcement learning of the reinforcement learning model for controlling the device by using the transition information, the device 400 can be quickly adapted to the environment where the device 400 is located so that the device 400 can be precisely controlled. Is to do.

Thereafter, the device remote control apparatus 100 stops control of the device 400 by the PID controller 110 and allows the device 400 to be controlled through the reinforcement learning agent 120.

Controlling the device 400 through the reinforcement learning agent 120 is performed using the reinforcement learning model for device control, and the status information received from the device 400 is transmitted to the reinforcement learning model for device control. Input and extract control information for controlling the device 400 according to a preset episode.

In addition, the reinforcement learning agent 120 transmits the extracted control information to the device 400 so that the corresponding device 400 can be controlled based on the control information, and the device 400 controls the control information. Based on the control status of the device 400 is transmitted to the reinforcement learning agent 120.

Subsequently, the reinforcement learning agent 120 generates transition information for reinforcement learning performed by the imitation reinforcement learning unit 200 based on the received state information of the device 400, and the imitation reinforcement learning unit 200 Let's store it sequentially in the experience memory.

Then, the imitation reinforcement learning unit 200 performs random reinforcement learning on the reinforcement learning network, which is the basis of the reinforcement learning model for controlling the device, by randomly minimizing the transfer information stored in the experience memory. .

In addition, the reinforcement learning is performed by updating the reinforcement learning network of the reinforcement learning model for controlling the device based on the transfer information stored in the experience memory, and is performed whenever the episode ends.

Meanwhile, as illustrated in FIG. 1, the PID controller 110 using a PID control model is illustrated as a means for quickly collecting transition information for initial reinforcement learning of the reinforcement learning model for device control, but in addition, linear quadratic regulation (LQR) ) It is possible to use various controllers to control a specific device through a traditional mathematical model, such as an LQR controller using a control model.

Also, when performing the initial reinforcement learning, the imitation reinforcement learning unit 200 performs the reinforcement learning by using the transfer information stored in the experience memory by the PID controller 110 as described above.

That is, when performing reinforcement learning for the initial reinforcement learning network (ie, when the transition information stored in the experience memory is equal to or less than a preset number), the imitation reinforcement learning unit 220 is connected to the PID controller 110. The reinforcement learning is performed based on the stored transfer information, and then the transfer information stored by the PID controller 110 gradually performs the reinforcement learning based on the transfer information stored by the reinforcement learning agent 120. .

Meanwhile, data (that is, status information and control information) between the device remote control apparatus 100 and the device 400 is transmitted and received through a wired/wireless communication network, so that the device 400 is transmitted through the device remote control apparatus 100. ) To be controlled remotely.

The wired/wireless network means a well-known wired/wireless communication network such as 5G (fifth generation), Ethernet, or the like.

As described above, the imitation reinforcement learning system 10 for remote control of a device of the present invention includes a reinforcement learning model for device control for remotely controlling the device 400 for each device 400 for remote control. When it is desired to control according to a set episode, if the preset number of transition information is not stored in the experience memory, the device is repeatedly performed through the process of controlling the device 400 through a PID controller using a traditional control model. By generating and storing transition information for performing reinforcement learning for the reinforcement learning model for control through the PID controller 110, the reinforcement learning model for initial device control by mimicking the transition information by the stored PID controller 110 By rapidly performing reinforcement learning for the device, the reinforcement learning model for controlling the device can be quickly adapted to the environment in which the device 400 is located.

Subsequently, using the reinforcement learning model for device control, the reinforcement learning agent 120 generates and stores transfer information for performing reinforcement learning on the reinforcement learning model for device control, and uses the stored transfer information By continuously updating the reinforcement learning model for device control, the reinforcement learning model for device control is upgraded to precisely control the corresponding device 400.

In addition, the network controller 300 performs a function of adjusting the priority of the network traffic so that the device 300 can be controlled in real time through the device remote control device 100.

That is, the network controller 300 transmits status information transmitted from the device 400 to the device remote control device 100 through the network and from the device remote control device 100 to the device 400. By setting the priority for network traffic of the control information to be the highest priority, the status information and control information can be transmitted and received in real time, so that the device 400 can be remotely controlled in real time.

At this time, the network controller 300 communicates with a plurality of network nodes (not shown) constituting the network through an open flow protocol to set an operation for each network node. Perform.

At this time, the network node performs a function of transmitting data transmitted through a network to a destination, and is composed of a switch (eg, an open virtual switch (OVS)), a router, and the like.

The present invention, by separating the control plane (control plane) and data plane (data plane) of each network node through the network controller 300, regardless of the network details through the central network controller 300, the Allows centralized control and management of network knocks.

That is, the network controller 300 in the present invention receives the node information from the plurality of network nodes in real time, and performs a function of managing and controlling each network node.

Meanwhile, the node information includes a flow table of network traffic for all data transmitted and received through each network node, and information about congestion of a data transmission path in each network node.

Therefore, the network controller 300, based on the node information received in real time from each network node, the network traffic for all data transmitted and received through each network node and whether or not congestion in each data transmission/reception path is congested in real time. Can be monitored.

Also, a plurality of devices 400 located in a physical environment and a device remote control device 100 located in a cyber environment to remotely control each of the devices 300 may be controlled through a plurality of network nodes constituting the network. Data for control (ie, status information and control information) is transmitted and received.

Meanwhile, in order to control each of the plurality of devices 400 in real time through the device remote control device 100, device status information transmitted from the plurality of devices 400 and the device remote control device 100 are transmitted. It is very important that the control information to be transmitted and received in real time.

Therefore, the network controller 300 monitors the flow of network traffic based on the flow table for network traffic received from the plurality of network nodes, and the bandwidth for the network traffic for the status information and the control command is the maximum bandwidth. Each of the above network nodes is set so as to be assigned as a.

In addition, the network controller 300, by adjusting the priority for the network traffic for the status information and the control command, so that each network node can transmit and receive the status information and the control command with priority over other data. Set.

That is, the network controller 300 checks network traffic for each network node with reference to the flow table, and adjusts the priority for each network traffic, so that the traffic other than the status information and control information is low. By setting the priority and setting the network traffic for the status information and the control information to a high priority, the status information and the control information can be preferentially transmitted and received.

In addition, the network controller 300 calculates the shortest path for the status information and the control command by referring to information on whether the received node information is congested with respect to the data transmission path, and the calculated shortest path is the By transmitting to each network node, the status information and control information can be transmitted from each network node according to the shortest path.

That is, the network controller 300 of the present invention, the status information between the device 400 and the device remote control apparatus 100 through the maximum bandwidth allocation, priority adjustment and shortest path calculation for the status information and control information And transmitting and receiving control information in real time, so that the device 400 can be remotely controlled in real time through the device remote control apparatus 100.

Meanwhile, the network controller 300 provides the node information and the priority information for the network traffic to the user terminal 500 using a RESTal API (representational state transfer API), thereby allowing the user to receive the node information and priority. It enables real-time monitoring of network status including information.

In addition, the user can adjust the priority of the network traffic for the status information and control information through the user terminal 500, and the priority adjustment information adjusted by the priority to the network controller 300 through the REST API By providing, it is possible to apply the adjusted priority to the status information and control information.

FIG. 2 is a diagram illustrating a method for performing imitation reinforcement learning for remote control of a device through a device remote control apparatus and an imitation reinforcement learning unit according to an embodiment of the present invention.

As illustrated in FIG. 2, the process of performing imitation reinforcement learning for remote control of a device through the imitation reinforcement learning unit 200 controls the control of the device 400 located in the physical environment according to a preset episode. By performing, the plurality of transition information is stored in the experience memory 210 of the FIFO structure, and when the set episode ends, the transition information stored in the experience memory 210 is applied to update the reinforcement learning network. By doing so.

Meanwhile, the reinforcement learning network is a basis for a device control learning model that extracts control information for remotely controlling the device 400, and the weight of the initial reinforcement learning network is randomly or randomly set according to a user's selection do.

Accordingly, the learning model for device control that is initially generated may have a very small number of reinforcement learning and a small number of transfer information used for reinforcement learning, so that the accuracy may be deteriorated.

To solve this problem, the device remote control apparatus 100 refers to the experience memory 210, and the transition information stored in the experience memory 210 is greater than a preset number (for example, 20,000). In the case of less, by controlling the device 400 using the PID controller 110, it is possible to stably and quickly generate initial transition information and store it in the experience memory 210.

That is, the device remote control apparatus 100, by controlling the device 400 through the PID controller 110, by quickly collecting the initial lack of transition information, thereby enabling the reinforcement learning based on this , The reinforcement learning model for controlling the device can be quickly adapted to the physical environment of the device 400 to precisely control the device 400.

In addition, when the device remote control apparatus 100 controls the device 400 according to a preset episode, by generating control information for initializing the device 400 and transmitting it to the device 400, Initialize the device.

At this time, the device remote control apparatus 100, if the transition information stored in the experience memory 210 is less than or equal to a preset number, allows the device 300 to be initialized through the PID controller 110 and , After the device 400 performs the initialization, device status information is provided to the PID controller 110.

In addition, when the device status information received from the device 400 is device status information based on control information for initialization, the PID controller 110 generates control information for the status information according to a previously stored PID control model. To the device 400 to receive device status information.

At this time, the PID controller 110 generates reward information for the device status information by calculating a reward for the device status information based on the received device status information.

Thereafter, the PID controller 110 generates transition information for the device including the generated reward information, and stores the generated transition information in the experience memory 210.

The above process for storing the generated transition information in the experience memory 210 is repeatedly performed until the set episode ends.

In addition, when the episode ends, the reinforcement learning unit 220 of the imitation reinforcement learning unit 200 performs a random mini-batch on the transition information stored in the experience memory 210 to transfer the preset number of times. Information is randomly selected and extracted from the experience memory 210, and reinforced learning for the reinforcement learning network is performed using the extracted transfer information.

On the other hand, when the episode ends and the transition information stored in the experience memory 210 is less than or equal to a preset number, the reinforcement learning is transition information generated by the PID controller 110.

Since the transition information generated by the PID controller 110 is defined in advance and is generated based on a result of controlling through a PID control model, which is a complex mathematical control model for precisely controlling the device 400, reliability is obtained. It is high and very stable data.

Therefore, since the transition information initially stored in the preset number of times in the experience memory 210 is the transition information generated through the PID controller 110, the reinforcement learning unit 220 rapidly reinforces based on the initial stable transition information. Learning can be performed.

That is, performing reinforcement learning for the reinforcement learning network through the reinforcement learning unit 220 is performed based on the transfer information generated through the PID controller 110, which is performed by the PID controller 110. Reinforcement learning is performed by imitating experience (that is, a result of controlling the device according to a predefined mathematical model), which is defined as mimic reinforcement learning in the present invention.

Next, when the transfer information stored in the experience memory 210 is a preset number, the device remote control apparatus 100 controls the device 120 through the reinforcement learning agent 120 to transfer the transition By generating information, it is possible to perform reinforcement learning for the reinforcement learning network.

That is, when the transfer information stored in the experience memory 210 is a preset number, control through the PID controller 110 is excluded, and from this point, only control through the reinforcement learning agent 120 is performed. .

This is, the reinforcement learning model for device control stored in the memory (not shown) of the reinforcement learning agent 120 performs reinforcement learning based on the transfer information (ie, the transfer information stored by the PID controller) stored in the experience memory 210. Since it is in one state, reliable transition information about the device 400 can be secured through the reinforcement learning agent 120.

Therefore, when the reinforcement learning agent 120 receives device status information from the device 400 according to the control, the reinforcement learning agent 120 enters the received device status information into the reinforcement learning model for device control stored in the memory, and the device Control information for controlling the 300 is generated and transmitted to the device 300 so that an operation according to the corresponding control information can be performed through the device 300.

Subsequently, the device 400 operates the corresponding device 400 based on the control information received from the reinforcement learning agent 120, generates device status information for this, and transmits the device status information to the reinforcement learning agent 120. .

In addition, the reinforcement learning agent 120, by calculating the reward for the received device status information, generates the reward information for the status information of the corresponding device, and of the device 400 including the generated reward information Transition information is stored in the experience memory 210.

Thereafter, when the episode is over, the reinforcement learning unit 220 of the imitation reinforcement learning unit 200 transfers a plurality of transition information stored in the experience memory 210 as a preset number through a random mini-batch. Information is extracted, and reinforcement learning is performed on the reinforcement learning network using the extracted plurality of transfer information.

In addition, when the reinforcement learning unit 220 performs the reinforcement learning, the reinforcement learning agent 120 provides learning parameters that are the result of the reinforcement learning to the reinforcement learning agent 120, and the reinforcement learning agent 120 receives the received learning By applying the parameter to the reinforcement learning model stored in the memory, the corresponding reinforcement learning model is updated, thereby enhancing the reinforcement learning model for device remote control for remote control of the device.

As described above, the transition information for the device 400 indicates a plurality of results of controlling the device according to the preset episode, and the experience memory (FIFO method) until the episode ends 210).

That is, the transition information is always stored in the experience memory 210 as many as a preset number, and when the transition information is input in excess of the preset number, the transition information stored first in the current experience memory 210 As is deleted, transition information that is subsequently input is sequentially stored.

Meanwhile, the episode refers to a series and process of performing a specific task according to a unique function for each device 400. For example, the device 400 is a rotary inverted pendulum. In the case of, a series of processes of establishing a pendulum becomes an episode, and when the device 400 is a robot arm for moving a specific object to a preset position, a series of processes for moving the specific object to a preset position It becomes an episode.

In addition, the transition information is generated for each control information for controlling the device 400 step by step to perform the preset episode, the current state information, control information, and the device of the device 400 It comprises a reward information for the next state information of the 400 and the next state information of the device 400.

In addition, the current status information of the device 400 refers to status information before the corresponding device 400 is controlled based on the control information, and the next status information of the device 400 is the corresponding device based on the control information It means the status information after 400 is controlled.

In addition, the reward information refers to a compensation value that is given based on a preset threshold range for the next state information after being controlled based on the control information, which is a result of operating the next state information within a preset threshold range. In the case of a positive compensation value (for example, +1), when the next state information is a result of operating beyond a preset threshold range, it can be assigned as a negative compensation value (for example, -100).

In addition, in the reinforcement learning, when extracting control information for controlling the device 400 based on the current state information of the device 400 based on the reward information, the entire reward information until the episode ends It is performed so that control information for controlling with high next state information is extracted.

That is, in the reinforcement learning, when the device 400 transitions from the current state to the next state according to a specific control command, it does not extract the control command having the highest reward information for the control command to transition to the next state. From the current state, until the end of the episode, the control command for the next state information with high total reward information is extracted.

In other words, the reinforcement learning unit 220, based on the transition information stored in the experience memory 210, when specific status information of the device 400 is input, the entire reward information to output high control information to the The reinforcement learning for the device control learning network will be performed.

On the other hand, the learning network for device control is preferably constructed using a convolutonal neural network (CNN), but can be constructed through various machine learning networks such as an artificial neural network (ANN). That is, the structure of the learning network for device control according to an embodiment of the present invention is not limited.

Such reinforcement learning is repeatedly performed each time the episode ends, and by continuously updating the learning model through such a process, each device 400 exposed to various environments can be precisely controlled by adapting to the environment. Make it possible.

Therefore, the reinforcement learning agent 120 recognizes the current state of the device 300 located in an arbitrary environment, and selects control information having the maximum sum of reward information in all episodes among the selectable control information, thereby making the device It is possible to precisely control the 300.

Also, the device status information and control information may be transmitted and received through an arbitrary communication method.

Accordingly, the device status information and control information are converted into a data format suitable for the specific communication method, and transmitted and received.

Meanwhile, the communication method refers to a lightweight publishing and subscription messaging transport protocol capable of communicating between communication devices, and is optimized for machine-to-machine (M2M), internet of things (IoT), and low. And capable of performing communication at high speed in a power and low bandwidth environment.

3 is a diagram showing the structure of a reinforcement learning network according to an embodiment of the present invention.

As shown in FIG. 3, the reinforcement learning network according to an embodiment of the present invention uses the input layer for receiving transition information stored in the experience memory 210 and a kernel of a preset size. And a first convolution layer, a second convolution layer, two fully connected layers, and an output layer that convolution a specific portion of the reinforcement learning data.

The input layer receives a plurality of transition information stored in the experience memory 210 whenever a preset episode is ended for each device 400. At this time, the transition information is converted into an image suitable for the reinforcement learning network and input to the input layer.

At this time, the transition information is generated by the PID controller 110 or the reinforcement learning agent 120, and not all the transition information stored in the experience memory 210 is input to the input layer, but performs random mini-batch. Then, it is randomly selected and arranged as many as a preset number and input to the input layer.

On the other hand, the reason for performing the random mini-batch for the transition information, because the current state information and the next state information included in the transition information includes a process of sequentially transitioning according to the control information, correlation between the respective transition information Correlation may cause a problem that the learning speed for the reinforcement learning becomes very slow, ignoring the correlation, and until the episode ends in the specific state information (that is, the final state of the device) This is to enable reinforcement learning to be performed quickly so that the control command with the highest sum of the reward information of can be extracted.

In addition, the first convolution layer, the image for the plurality of transition information while moving according to the size of the preset stride (eg 1) of the first kernel having a predetermined weight (eg 4x4) and a specific weight Each of the weights of the specific portion and the first kernel is convolved, and a first feature map for a specific portion of the image is generated and output. Here, the size of the stride means the moving unit of the kernel.

At this time, the result of outputting the first convolution layer through convolution is a feature map of a total of 32 (ie, 7x1x32) having a size of 7x1, and the second convolution is the first convolution layer. The first feature map outputted through the preset size (eg 4x1) and the second kernel having a specific weight are moved according to the preset stride size (eg 1) and output from the first convolution layer. The second feature map is generated and output from the first feature map. When the size of the second kernel of the second convolution layer is 4x1, the second feature map output through the second convolution layer may be composed of a total of 64 feature maps having a size of 4x1.

Next, the two fully connected layers connect the output feature map to output at least one control command for transitioning from a specific state of the device 400 to a next state through an output layer.

At this time, the control command, when transferring the device from the specific state to the next state, outputs at least one or more in the order of the sum of the reward information for each state up to the end state in which the episode set in the device 400 ends. do.

Through this process, the reinforcement learning unit 220 may gradually update the reinforcement learning network by applying the transition information generated each time the device 400 is controlled according to the episode to the reinforcement learning network. .

Through this, the reinforcement learning unit 220 allows the device to continuously enhance the reinforcement learning model for controlling the device, and adapts the device 400 located in various environments to the environment to precisely control it remotely. .

4 is a block diagram showing the configuration of a device remote control apparatus according to an embodiment of the present invention.

As shown in FIG. 4, the device remote control apparatus 100 according to an embodiment of the present invention uses a PID controller 110 for controlling the device and a reinforcement learning model for device control using a PID control model It comprises a reinforcement learning agent 120 for controlling the device.

Also, the device remote control apparatus 100 refers to the experience memory 210 of the imitation reinforcement learning unit 200, and when the transition information of a preset number or less is stored in the experience memory 210, the PID The device 400 is controlled through the controller 110 to generate transition information for the device 400 and store it in the experience memory 210.

In this case, the PID controller 110 controls the device 400 according to a preset episode using the PID control model stored in a memory (not shown) of the PID controller 110 to generate the transition information. And, it is possible to store the generated transition information in the experience memory of the imitation reinforcement learning unit 210.

Since the PID controller 110 has been described with reference to FIG. 2, further detailed description will be omitted.

Meanwhile, in FIGS. 1 to 3, although the PID controller 110 is illustrated as being integrated in the device remote control device 100, the device remote control device is configured separately from the device remote control device 100 ( It is natural that it can be implemented to perform the function of the PID controller 110 through interworking with 100).

In addition, the device remote control apparatus 100, if the transition information is stored in a preset number in the corresponding experience memory 210 with reference to the experience memory 210 of the imitation reinforcement learning unit 200, the reinforcement The device 400 is controlled through the learning agent 120 to generate transfer information for the device 400 and store it in the experience memory 210.

The reinforcement learning agent 120 generates control information for remotely controlling the device 400 according to a preset episode, and transmits the generated control information to the device 400, thereby allowing the device 400 to ), a control information processing unit 121 for remote control, a device status information receiving unit 122 for receiving device status information of the corresponding device 400 controlled according to the control information from the device 400, the Reward information generation unit 123 for generating reward information for the received device status information, a transition for generating transition information for the device 400 based on the generated reward information, and storing it in the experience memory 121 It comprises an information generation unit 124, a reinforcement learning model update unit 125 and memory 126 for receiving learning parameters from the imitation reinforcement learning unit 220 and updating the reinforcement learning model for controlling the device.

The control information processing unit 121 generates control information for controlling the device 400 according to a preset episode using the reinforcement learning model for device control provided by the imitation reinforcement learning unit 200 and generates the control information. By transmitting one control information to the device 400, it is possible to control the device 400.

That is, the control information processing unit 121 loads the device status information received from the device status information receiving unit 122 from the memory 126 for the device learning reinforcement learning model for the corresponding device 400, and loads the device status information. The received device status information is input to the reinforcement learning model for device control, and control information for controlling the corresponding device 400 step by step is generated.

At this time, the reinforcement learning model for device control outputs at least one or more control information in the order of the highest sum of the reward information up to the final state according to the episode of the device 400 preset, and the control information processing unit 121 , By selecting control information having the highest reward information, control information for controlling the device 400 is generated.

Meanwhile, the control information processing unit 121 further includes a function for generating control information for initializing the device 400 and transmitting it to the device 400 when the device 400 is to be controlled.

In addition, the device status information receiving unit 121, the device 400 is controlled according to the control information for the initialization or control information generated through the reinforcement learning model for controlling the device to receive the status information of the device 400 To perform the function.

Meanwhile, the control information and the device status information are transmitted and received using an arbitrary communication method as described above.

Also, the reward information generating unit 123 generates a reward information for the device status information by calculating a reward for the device status information based on a preset threshold range based on the received device status information. To perform.

The reward information generation unit 123 applies a positive compensation value when the received device status information is controlled within the threshold range, and the received device status information is operated beyond a mirror-set threshold range. In this case, by applying a negative compensation value, reward information for the received device status information is generated.

In addition, the transition information generation unit 124, based on the received device status information and the generated reward information, generates the transition information of the device 400 according to the control information, the imitation reinforcement learning unit 200 It performs the function of storing in the experience memory 210.

The transition information is configured to include current status information, control information, reward information, and next status information of the device 400.

That is, the transition information generation unit 124 sets device status information and reward information received from the device 400 as next status information and reward information, and sets previous status information of the next status information as current status information. And, by setting the control information generated for the previous state information as control information of the corresponding transition information, the transition information is generated.

The transition information is sequentially stored in the experience memory 210 until the set episode ends. On the other hand, when the set episode ends, it indicates a final state in which control of the device 400 is finally completed, and means when the device 400 finally transitions to a preset end state according to the control. do.

Meanwhile, the transition information is stored in the memory 126 provided in the reinforcement learning agent 120 until the episode ends, while the transition is transferred to the experience memory 210 when the episode ends. Information can be stored at once in the order in which it was created.

Thereafter, when the episode ends, the imitation reinforcement learning unit 200 randomly selects a predetermined number of a plurality of transition information stored in the experience memory 200 to strengthen the reinforcement learning network for controlling the device. Make learning possible.

In addition, the reinforcement learning model update unit 125 receives learning parameters that are a result of performing reinforcement learning in the imitation reinforcement learning unit 200 from the imitation reinforcement learning unit 200, and memory the provided learning parameters. By applying the reinforcement learning model for device control stored in (126), the corresponding learning model is updated.

That is, the reinforcement learning agent 120 controls the device 400 according to a preset episode, generates transition information through the control, and provides it to the imitation reinforcement learning unit 200 and strengthens the imitation By receiving a learning parameter that is a result of performing reinforcement learning from the learning unit 200 and repeatedly performing the process of updating the reinforcement learning model for device control, the device control reinforcement learning model is gradually upgraded and the device 400 ) Can be precisely controlled.

5 is a view showing the configuration of the imitation reinforcement learning unit of the imitation reinforcement learning system for remote control of a device according to an embodiment of the present invention.

As shown in FIG. 5, the imitation reinforcement learning unit 200 of the imitation reinforcement learning system 10 for remote control of a device according to an embodiment of the present invention enhances device control for controlling a specific device 400 A learning model is generated, and the reinforcement learning model for controlling the generated device is provided to the reinforcement learning agent 120 to remotely control the corresponding device 400 located in a physical environment through the reinforcement learning model for controlling the device. To make.

In addition, the imitation reinforcement learning unit 200, the experience memory 210 for storing a predetermined number of transition information of the device 400 generated based on the control of the device 400 in a FIFO format, the stored Reinforcement of the reinforcement learning agent (120) for performing reinforcement learning on the reinforcement learning network of the reinforcement learning model for device control using transition information, and reinforcement learning agent (120) as a result of the reinforcement learning It comprises a learning parameter providing unit 230 provided to the learning model update unit 125.

The transition information is data that is the basis for performing the reinforcement learning, and is generated by the PID controller 110 and the reinforcement learning agent 120 and stored in the experience memory 210.

Meanwhile, the experience memory 210 is configured with a FIFO structure, and stores the transition information in the order in which the transition information is generated as many as a preset number.

In addition, the transition information is stored in order according to a result of controlling the device 400 during a preset episode, and stored at one time each time the episode ends, or whenever the transition information is generated. Can be.

On the other hand, since the reinforcement learning model for controlling the device is initially generated, the weight for the reinforcement learning model for controlling the device is randomly or randomly set according to the user's selection, and the number of transition information for reinforcement learning is insufficient, so the transition information is It is generated by the PID controller 110. At this time, the experience memory 210 stores a predetermined number of transition information generated by the PID controller 110, and reinforcement learning for the reinforcement learning network of the reinforcement learning model for controlling the device is the PID controller 110 ). This is to perform reinforcement learning by imitating the PID control model, which is a traditional control model of the PID controller 110, and the present invention is referred to as imitation reinforcement learning.

Thereafter, the transition information is generated through the reinforcement learning agent 120 and stored in the FIFO format in the experience memory 210, and the reinforcement learning unit 220 transmits the transition information stored in the experience memory 210. By performing reinforcement learning by using, it is possible to precisely control the device 400 located in the physical environment.

In addition, the learning parameter providing unit 230, as a result of performing reinforcement learning on the reinforcement learning network, provides learning parameters of the reinforcement learning network to the reinforcement learning model update unit 125 of the reinforcement learning agent 120 By doing so, by applying the learning parameter to the reinforcement learning model for controlling the device, it is possible to update the reinforcement learning model for controlling the device.

That is, the reinforcement learning model for device control is gradually advanced through learning parameters that are the result of performing reinforcement learning, so that the device 400 can be quickly and precisely controlled.

Meanwhile, when the reinforcement learning network is configured based on a convolutional neural nework (CNN), transition information stored in the experience memory 210 is imaged so that reinforcement learning can be performed on the reinforcement learning network.

However, the reinforcement learning network in the present invention may be implemented based on various machine learning networks such as an artificial neural network (ANN), and accordingly, the transfer information may be converted into various formats according to the various machine learning networks. .

6 is a flowchart illustrating a procedure for performing reinforcement learning according to an embodiment of the present invention.

As illustrated in FIG. 6, the device 300 located in a physical environment is quickly adapted to a corresponding physical environment through the imitation reinforcement learning system 10 for remote control of the device according to an embodiment of the present invention. 300) The procedure for performing reinforcement learning in order to precisely control, first, the device remote control apparatus 100 of the imitation reinforcement learning system 10 for remote control of the device is set in advance to perform the reinforcement learning When the device 400 is to be remotely controlled according to an episode (S110), the device 400 is initialized, and status information of the device 400 is received (S120).

In this case, when the device remote control apparatus 100 refers to the experience memory 210 of the imitation reinforcement learning unit 200, when the stored transition information of the experience memory 210 is less than or equal to a preset number, The device 400 is controlled by the PID controller 110 to generate the transition information, and in the case of abnormality, the device 400 is controlled by the reinforcement learning agent 120 to control the device 400. Create transition information.

In addition, in the process of generating the transition information by controlling the device 400 through the PID controller 110, the PID controller 110 generates control information according to the device state information using a PID control model. (S131).

The PID control model calculates an error value by comparing the state information of the device 400 with a preset reference value or a set value, and then calculates a value necessary for controlling the device using the calculated error value, By controlling the device 400, it means a predefined mathematical model to stably control the device 400 according to a preset episode.

Next, the PID controller 110 transmits the generated control information to the device 400 to control the device 400 (S141), and then, the PID controller 110, the device Device status information for a control result according to the control information is received from 400 to generate transfer information for the result, and the generated transfer information is stored in the experience memory 210 (S151).

Thereafter, when the episode ends (S160), the imitation reinforcement learning unit 200 uses the transfer information stored in the experience memory 210 to perform reinforcement learning on the reinforcement learning network, thereby controlling the device It is necessary to advance the reinforcement learning model.

Meanwhile, generating the transition information through the PID controller 110 and storing it in the experience memory 210 quickly collects initial reinforcement learning data (ie, transition information) for the reinforcement learning model for device control. In order to enable reinforcement learning by imitating the initial stable data, the reinforcement learning model for controlling the device can be quickly adapted to the physical environment of the device 400.

Also, whenever the transition information is generated through the PID controller 110 and stored in the experience memory 210, whenever the transition information stored through the PID controller 110 becomes the preset number, or When the episode ends, the imitation reinforcement learning unit 200 may be implemented to perform the imitation strong learning based on the transfer information stored through the PID controller 110.

At this time, the imitation reinforcement learning unit 200 provides the learning parameter that is the result of performing the reinforcement learning to the reinforcement learning agent 120 so that it can be applied to the reinforcement learning model for controlling the device, thereby controlling the device The reinforcement learning model can be quickly adapted to the physical environment of the device 400.

In addition, when the transition information of the experience memory 210 is stored in a preset number, when the device 400 is controlled through the reinforcement learning agent 110 to generate and store the transition information for the reinforcement learning, the In the reinforcement learning agent 110, the received state information is input to the reinforcement learning model for device control, and control information for controlling the device is generated (S130).

Next, the reinforcement learning agent 120 controls the device 400 by transmitting the generated control information to the device 400, and is controlled according to the control information from the device 400 The device status information of the corresponding device 400 is received, the transfer information is generated, and the generated transfer information is stored in the experience memory 210 (S150).

Thereafter, when control of the device 400 is terminated according to the preset episode (S160), the imitation reinforcement learning unit 200 uses the transition information stored in the experience memory 210 to enhance the Learning is performed (S170).

At this time, the reinforcing learning unit 200 provides the learning parameter that is the result of performing the reinforcement learning to the reinforcement learning agent 120 so that it can be applied to the reinforcement learning model for device control, thereby controlling the device By enhancing the reinforcement learning model, the device 400 can be precisely controlled.

Meanwhile, reinforcement learning using transition information stored through the reinforcement learning agent 120 is performed whenever the episode ends, and the reinforcement learning is repeatedly performed to gradually enhance the reinforcement learning model for device control. By upgrading, it is possible to precisely control the device 400 remotely.

In addition, the reinforcement learning is performed so that at least one or more of the sum of the reward information for all states may be output in a high order from the specific state information of the device 400 based on the transition information until the episode ends. In addition, the reinforcement learning agent 120 generates the control information by selecting the control information having the highest sum of the reward information.

In addition, the device status information and the control information are transmitted and received through any specific communication method, and are controlled to be transmitted and received in real time by the network controller 300 as described above.

In addition, the experience memory 210 is formed of a FIFO-structured memory, and stores the transition information as many as a preset number, and stores the transition information in the experience memory 210 in excess of the preset number. If possible, the transfer information exceeding the preset number is stored in the experience memory 210 by deleting the transfer information stored at the earliest.

Through this, when the reinforcement learning is performed, it is possible to perform efficient reinforcement learning by preventing the transfer of the repetition information repeatedly.

As described above, the imitation reinforcement learning system and method for remote control of a device according to an embodiment of the present invention utilize a traditional machine control model for controlling a specific device to quickly transfer transition information for the specific device. It is possible to quickly adapt to the physical environment of the device and to precisely control the device by allowing the device to perform reinforcement learning on the reinforcement learning model for device control for controlling the device by imitating the generated transition information. It has the effect of making it possible.

In the above, the preferred embodiment according to the present invention has been mainly described, but the technical spirit of the present invention is not limited thereto, and each component of the present invention is modified or modified within the technical scope of the present invention in order to achieve the same purpose and effect. It could be.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. In addition, various modifications may be implemented by a person having ordinary knowledge in the art, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

10: Imitation reinforcement learning system for device remote control
100: device remote control device 110: PID controller
120: reinforcement learning agent 121: control information processing unit
122: device status information receiving unit 123: reward information generating unit
124: transfer information generation unit 125: reinforcement learning model update unit
126: memory 200: imitation reinforcement learning department
210: experience memory 220: reinforcement learning department
230: learning parameter providing unit 300: network controller
400: device 500: user terminal

Claims (10)

An experience memory storing transition information generated based on a control result for at least one device located in a physical environment according to a preset episode in a first in first out (FIFO) method; And
Includes; a mimic reinforcement learning unit for performing reinforcement learning on a reinforcement learning model for device control for controlling the device using the transition information stored in the experience memory.
The transition information includes current status information of the device, control information for controlling the device, next status information of the device controlled according to the control information, and reward information for the next status information. Imitation reinforcement learning system for device remote control. The method according to claim 1,
The imitation reinforcement learning system,
When the transition information is stored in the experience memory in a number less than a preset number, the device is controlled according to the episode through a PID controller using a PID control model, and the transition information is generated and stored in the experience memory And
The imitation reinforcement learning unit, whenever the transition information generated and stored through the PID controller becomes the preset number, or whenever the transition information is stored in the experience memory through the PID controller, or the episode ends. At this time, the device characterized in that it further comprises performing the reinforcement learning by mimicking the experience of the PID controller controlling the device by performing the reinforcement learning based on the transfer information generated and stored through the PID controller. Imitation reinforcement learning system for remote control. The method according to claim 2,
The imitation reinforcement learning system,
When the transition information is stored in the experience memory in a preset number, by inputting the state information of the device into the reinforcement learning model for controlling the device performing the reinforcement learning, extracting control information for controlling the device, Further comprising; reinforcement learning agent for controlling the device step by step according to the episode,
The reinforcement learning agent receives device status information from the device controlled based on the control information, generates transition information for the device, and stores it in the FIFO method in the experience memory for remote control of the device. Imitation reinforcement learning system. The method according to claim 1,
The reinforcement learning,
It is performed to extract control information in which the sum of the reward information is the maximum until the final state of the device, which is set in advance according to device state information for a specific state of the device,
The experience memory,
A imitation reinforcement learning system for remote control of a device, characterized in that the transition information is stored in a FIFO manner to prevent the transition information from being duplicated and applied during the reinforcement learning. The method according to claim 1,
The reward information,
It is characterized in that it is generated by applying a positive compensation value when the state information of the device is controlled within a preset threshold range and applying a negative compensation value when the result is controlled beyond a preset threshold range. Imitation reinforcement learning system for remote control of devices. A storage step of storing transition information generated based on a control result for at least one device located in a physical environment according to a preset episode in an experience memory in a first in first out (FIFO) method; And
Includes; imitation reinforcement learning step of performing reinforcement learning on the reinforcement learning model for device control for controlling the device by using the transition information stored in the experience memory.
The transition information includes current status information of the device, control information for controlling the device, next status information of the device controlled according to the control information, and reward information for the next status information. Imitation reinforcement learning method for device remote control. The method according to claim 6,
The imitation reinforcement learning method,
When the transition information is stored in the experience memory in less than a preset number, the device is controlled according to the episode through a PID controller using a PID control model, and the transition information is generated and stored in the experience memory And
In the imitation reinforcement learning step, whenever the transition information generated and stored through the PID controller becomes the preset number, or whenever the transition information is stored in the experience memory of the PID controller, or the episode ends. At this time, the device characterized in that it further comprises performing the reinforcement learning by mimicking the experience of the PID controller controlling the device by performing the reinforcement learning based on the transfer information generated and stored through the PID controller. Imitation reinforcement learning method for remote control. The method according to claim 7,
The imitation reinforcement learning method,
When the transition information is stored in the experience memory in a preset number, through the reinforcement learning agent, control for controlling the device by inputting the state information of the device into the reinforcement learning model for controlling the device performing the reinforcement learning And extracting information to control the device stepwise according to the episode.
The reinforcement learning agent receives device status information from the device controlled based on the control information, generates transition information for the device, and stores it in the FIFO method in the experience memory for remote control of the device. Imitation reinforcement learning method. The method according to claim 6,
The reinforcement learning,
It is performed to extract control information in which the sum of the reward information is the maximum until the final state of the device, which is set in advance according to device state information for a specific state of the device,
The experience memory,
By storing the transition information in a FIFO manner, when the reinforcement learning, the mitigation reinforcement learning method for remote control of a device, characterized in that the transition information is prevented from being applied repeatedly. The method according to claim 6,
The reward information,
It is characterized in that it is generated by applying a positive compensation value when the state information of the device is controlled within a preset threshold range and applying a negative compensation value when the result is controlled beyond a preset threshold range. Reinforcement learning method for remote control of a device.

Images