KR20230151736A - Method and system for selecting movement routs of mobile robot performing multiple tasks based on deep reinforcement learning - Google Patents

Abstract

The reinforcement learning-based movement path selection method for a mobile robot that performs a variety of tasks involves inputting parameters of DQN artificial intelligence, including the task to be performed by the mobile robot and the work environment, and selecting nodes for multiple zones for the input work environment. Designing a simulation environment by allocating and displaying it as a graph, defining environmental states and observable states for the task, and designing a goal for the task and a reward function according to the goal, the DQN in the simulated environment It includes a step of reinforcement learning by having artificial intelligence select a movement path that maximizes the action value function value, and a step of controlling autonomous driving of the mobile robot in a real environment or simulation environment using the learned DQN artificial intelligence. It is composed by:

Description

Reinforcement learning-based movement path selection method and system for mobile robots performing multiple tasks {METHOD AND SYSTEM FOR SELECTING MOVEMENT ROUTS OF MOBILE ROBOT PERFORMING MULTIPLE TASKS BASED ON DEEP REINFORCEMENT LEARNING}

The following description relates to a method and system for selecting a movement path that allows a mobile robot to optimally perform a selected task among various tasks in a given work environment.

Reinforcement learning is a machine learning methodology that defines a reward function that represents mission achievement when a random task environment is given and makes a random agent learn actions that maximize reward within that environment. With the development of computing device performance and deep learning technology, reinforcement learning has developed into the form of deep reinforcement learning and has proven effective performance in various problems that were previously difficult to solve due to complex modeling. . For example, in 2016, AlphaGo, a reinforcement learning-based Go artificial intelligence, won a Go match against Korean professional Go player Lee Sedol, thereby impressing upon the public the effectiveness of deep reinforcement learning technology. The scope of application of these deep reinforcement learning technologies is expanding to research into autonomous task performance of mobile robots.

Research to secure autonomous task performance capabilities of mobile robots has been continuously conducted due to its advantages such as cost reduction and reduced risk to life. Among them, research on service robots that can perform a variety of tasks including patrolling, security deployment, fire surveillance, delivery of goods, and cleaning has been actively conducted to increase the scope and efficiency of existing unmanned service systems.

Existing studies on deep reinforcement learning technology have been limited to applications to mobile robots that perform only single tasks such as patrolling, security deployment, fire surveillance, product delivery, and cleaning. However, a reinforcement learning simulation environment was constructed to apply to various tasks including patrolling, security deployment, fire surveillance, product delivery, and cleaning in the same environment, and there were no agents learned in the constructed simulated environment.

In addition, the existing deep reinforcement learning technology for application to mobile robots receives video information captured in real time by the mobile robot or map images of the work environment, and is applied to an artificial intelligence model that estimates the control signal or movement coordinates of the mobile robot. Research is mainly conducted on However, since the method of using video data requires a large amount of calculation, there is a problem that performance is limited when operated on a computing device that can be mounted on a mobile robot.

In order to reduce the amount of computation, research is being conducted on applying reinforcement learning to data abstracted to a lower level than video. However, there are cases where single tasks such as patrolling, delivery, and cleaning are performed, but there are also cases where single tasks such as patrolling, security deployment, and fire are performed. There has been no case in which an abstract reinforcement learning simulation environment was constructed and an agent was trained for various tasks including surveillance, product delivery, and cleaning.

The above-mentioned background technology is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before the present application.

The purpose of the embodiment is to provide a method and system for selecting a movement path to enable a mobile robot to optimally perform selected tasks among various tasks such as patrolling, security deployment, fire surveillance, product delivery, and cleaning in a given work environment. .

The problems to be solved in the embodiments are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

The reinforcement learning-based movement path selection method for a mobile robot performing multiple tasks according to an embodiment includes the steps of receiving parameters of DQN artificial intelligence including the task to be performed by the mobile robot and the work environment, and multiple steps for the input work environment. Designing a simulation environment by allocating nodes for each zone and displaying it in a graph, defining environmental states and observable states for the task, and designing a goal for the task and a reward function according to the goal, the simulation Reinforcement learning in the environment by having the DQN artificial intelligence select a movement path that maximizes the action value function value, and controlling autonomous driving of the mobile robot in a real or simulated environment using the learned DQN artificial intelligence. It is composed of steps including:

In addition, the movement path selection system that controls the autonomous driving of a mobile robot that performs various tasks according to an embodiment includes an input unit that receives parameters of DQN artificial intelligence, including the task to be performed by the mobile robot and the work environment, and the DQN artificial intelligence. A learning unit that designs a simulated environment for reinforcement learning of intelligence, strengthens and learns the DQN artificial intelligence in the designed simulated environment, and controls the autonomous driving of a mobile robot in a real or simulated environment using the learned DQN artificial intelligence. It consists of a driving control unit.

In addition, the mobile robot that performs various tasks according to the embodiment designs a simulated environment for the input task and work environment, selects a movement path that maximizes the action value function value in the simulated environment, and uses DQN artificial intelligence for reinforcement learning. and is operated based on learned DQN artificial intelligence.

According to embodiments, in a given work environment, a simulated environment is designed for deep reinforcement learning of a mobile robot that can perform a variety of tasks including one or more of patrolling, security deployment, fire surveillance, product delivery, and cleaning. , By learning an artificial intelligence agent through this, the mobile robot can perform calculations efficiently and select a movement path that performs the optimal task.

The effects of the reinforcement learning-based movement path selection method and system for a mobile robot that performs various tasks according to the embodiment are not limited to those mentioned above, and other effects not mentioned will become clear to those skilled in the art from the description below. It will be understandable.

Figure 1 is a block diagram illustrating a reinforcement learning-based movement path selection system for a mobile robot that performs various tasks according to an embodiment.
Figure 2 is a diagram illustrating a method of representing a work environment as a graph according to an embodiment.
Figure 3 is a block diagram explaining the configuration of DQN artificial intelligence according to an embodiment.
Figure 4 is a diagram explaining a DQN artificial intelligence learning method according to an embodiment.
FIG. 5 is a block diagram illustrating the configuration of the travel control unit of FIG. 1.
FIG. 6 is a diagram explaining the operation of the travel control unit of FIG. 5.
Figures 7A to 7E are graphs showing the task success rate according to the number of training sessions of DQN artificial intelligence according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes can be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

Additionally, in describing the components of the embodiment, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. When a component is described as being "connected," "coupled," or "connected" to another component, that component may be directly connected or connected to that other component, but there is no need for another component between each component. It should be understood that may be “connected,” “combined,” or “connected.”

Components included in one embodiment and components including common functions will be described using the same names in other embodiments. Unless stated to the contrary, the description given in one embodiment may be applied to other embodiments, and detailed description will be omitted to the extent of overlap.

Hereinafter, with reference to FIGS. 1 to 7E, a reinforcement learning-based movement path selection method and system for a mobile robot that performs various tasks according to an embodiment will be described. For reference, FIG. 1 is a block diagram illustrating a reinforcement learning-based movement path selection system (hereinafter referred to as “movement path selection system”) 100 for a mobile robot that performs various tasks in one embodiment. FIG. 2 is a diagram illustrating a method of representing a work environment as a graph for designing a simulated environment in the movement path selection system 100 according to an embodiment. And Figure 3 is a block diagram explaining the configuration of the DQN artificial intelligence 151 according to an embodiment, and Figure 4 is a diagram explaining the reinforcement learning method of the DQN artificial intelligence 151. And FIG. 5 is a block diagram explaining the configuration of the driving control unit 160 according to an embodiment, and FIG. 6 is a diagram explaining the operation of the driving control unit 160.

Referring to FIG. 1, the movement path selection system 100 for controlling the autonomous driving of a mobile robot that performs various tasks includes an input unit 110, a processor 120, a memory 130, an output unit 140, It may include a learning unit 150 and a driving control unit 160.

The movement path selection system 100 applies autonomous driving technology that combines location recognition technology and driving control technology using sensors such as cameras and LiDAR (Light Detection And Ranging: LiDAR), and DQN (Deep Q Network) Based on artificial intelligence (151), mobile robots that perform various tasks such as patrol (security), delivery of goods, cleaning, security deployment, and fire audit can efficiently perform calculations and select the optimal movement path. Let it happen.

First, reinforcement learning will be explained.

Reinforcement learning is one of the machine learning methodologies that solves cases where the theoretical model for a given problem is complex by simplifying the problem into the form of interaction between the environment and the agent. It is mainly used to learn artificial intelligence that must perform specific tasks with optimal efficiency in dynamically changing situations.

Reinforcement learning is where an agent in a random environment recognizes the current state through observation and takes action according to the action policy for the recognized current state. If you do this, you will receive a reward. Here, reward refers to the degree to which the action taken by the agent is close to the optimal solution of the problem, and the closer the action taken by the agent is to the optimal solution, the higher the reward is received.

Reinforcement learning, for example, can model the environment as a POMDP (Partially Observable Markov Decision Process) to solve a problem. POMDP consists of 7-tuples (S, A, P, R, Ω, O, γ). Here, S is the state space, A is the action space, P is the transition probability, R is the reward function, Ω is the observation space, and O is the observation. Observation probability, γ is the discount factor.

State space S is a set of information representing the current state of the environment. For example, the state space S may include the current location of the mobile robot, the location of the work object, the location of dynamic obstacles, the location of the congestion area, etc.

Action space A is the set of actions that an agent can take in the current environment. For example, action space A may include the coordinates of a point where the mobile robot can move from its current location.

The transition probability P represents the probability that the agent will transition from the current state s _t to the state s _t+1 according to the selected action a _t . For example, the transition probability P may represent the possibility of movement from the current location of the mobile robot to the center coordinate of each area in the environment.

The reward function R refers to the reward given as the agent selects action a _t in the current state s _t and transitions to state s _t+1 . For example, the reward function R may be configured to provide a reward of -10 whenever the mobile robot attempts to move to an impossible coordinate during each task, and to provide a reward of +10 when the mobile robot completes the task.

Observation probability O represents the probability that the agent will observe information in the current state. For example, the observation probability O may represent the observability of a dynamic obstacle according to the current positions of each of the mobile robot and the dynamic obstacle, at the current location of the mobile robot.

The discount coefficient γ∈[0, 1] is a constant that indicates how much more important the current reward is than the reward that will be obtained in the future. For example, if γ = 0, the mobile robot will learn an action policy that maximizes the current reward, and if γ = 1, the mobile robot will learn an action policy that maximizes the sum of future rewards. You can.

In other words, the agent learns the action policy π ^* to maximize the reward r _t through a series of processes that repeat observation o _t , state s _t recognition, and action a _t in the work environment modeled by POMDP as above. You can find the optimal solution to a given problem.

In reinforcement learning, the state value function V _π (s _t ) can be defined as the expected value of reward that the agent will receive when she follows the action policy π in any state s _t . And the action value function Q _π (s _t , a _t ) can be defined as the expected value of reward received when the agent takes action a _t according to the action policy π in a random state s _t .

Reinforcement learning trains the agent in the direction of maximizing the reward of the state value function V _π (s _t ) or action value function Q _π (s _t , a _t ) for each state, thereby determining the optimal behavior for the agent to maximize the reward. We can find the policy π ^* (s _t ).

The state value function V _π (s _t ) or the action value function Q _π (s _t , a _t ) can be expressed as Equation 1 according to the Bellman expectation equation.

At this time, the optimal state value function V ^* (s _t ) and the optimal action value function Q ^* (s _t , a _t ) can be defined by Equation 2.

The optimal state value function V ^* (s _t ) and the optimal action value function Q ^* (s _t , a _t ) can be expressed as Equation 3 from Equation 1 and Equation 2 as follows.

Meanwhile, in the Q-Learning method, the optimal action policy π ^* to maximize reward by learning the action value function Q _π (s _t , a _t ) for all states in the environment using the Bellman expectation equation. (s _t ) was found. However, when the number of states in the environment is infinite, it is difficult to define and learn the action value function Q _π (s _t , a _t ) for all states S and all actions A.

On the other hand, in the DQN (Deep Q Network) method, by constructing a deep neural network that can approximate the action value function Q _π (s _t , a _t ) for all states S and all actions A, Calculations can be performed efficiently.

For example, as shown in FIG. 3, when the DQN artificial intelligence 151 receives the current state s _t , it sets the action value function value Q(s _t , a _t ) for the action that can be taken in that state. You can select an action by outputting and taking the action a _t ^* =argmax _at Q(s _t , a _t ) that maximizes the output action value function value Q(s _t , a _t ).

And the DQN artificial intelligence 151 is trained to find a parameter that minimizes the loss function L designed by applying Equation 3. The loss function L can be defined as Equation 4.

Returning to Figure 1, the input unit 110 receives parameters from the user, such as the task to be performed by the mobile robot, the work environment for performing the task, and algorithm settings.

The processor 120 controls the memory 130, output unit 140, learning unit 150, and driving control unit 160 using the input parameters.

The memory 130 stores data such as parameters, tasks, and environmental settings of the artificial intelligence algorithm.

The output unit 140 can output system status and simulation status on the screen.

The learning unit 150 designs a simulation environment for deep reinforcement learning of an artificial intelligence agent according to the task and work environment selected by the user input from the input unit 110 and the results selected by the mobile robot, and creates an agent in the designed simulation environment. DQN (Deep Q Network) artificial intelligence (151) trains the agent to maximize rewards according to the selected movement path.

In detail, the method of designing a simulated environment in the learning unit 150 is to represent the work environment in a graph by allocating nodes for each zone, define environmental states and observable states appropriate for each task, and It may include designing goals and reward functions.

First, in the step of representing the work environment in a graph, as shown in FIG. 2, a map of the work environment is created (S11), a plurality of zones are divided in the created map, and a node is assigned to each zone (S12). , It includes the step of creating a graph (S13) showing the possibility of movement from each zone to another zone and the observability of dynamic obstacles. For example, each node is set to the center coordinates representing each area, and a graph can be created in the form of connecting movable nodes and observable nodes at each node with lines. Here, the movable area and the observable area may appear different from each other or may coincide. For example, node 2 can move to nodes 0, 1, 3, 4, 19, 23, 24, and 25, and nodes 0, 1, 3, 4, 23, 24, and 25 can be observed. .Next, the input unit 110 defines an environmental state suitable for the task selected by the user and an observable state that can be observed by the mobile robot according to the selected task. The environmental state may include all information necessary for the mobile robot to perform the selected task within the work environment. And the observable state may include a state that is accessible while the mobile robot is performing a selected task among defined environmental states. For example, environmental states and observable states can be defined as shown in Table 1.

target action environmental status observable state patrol · The robot's movement area for each location · The robot's field of view for each location
· Robot position
· Robot vision
· Intruder location changes randomly · Area where the robot can move from the current location
· Robot position for the last 3 hour frame
· Robot field of view and location of intruders captured within the field of view during the last 3 hour frame delivery of goods · Area where the robot can move to each location · Target location for delivery of goods
· Robot position
· Congestion in each area changes arbitrarily · Area where the robot can move from the current location
· Delivery target point
· Robot position for the last 3 hour frame
· Location of congested areas during the last 3 hour frame cleaning ·Robot movement area for each location·Cleaning completion area
· Robot position
· Congestion in each area changes arbitrarily · Area where the robot can move from the current location
· Cleaned area
· Robot position for the last 3 hour frame
· Location of congested areas during the last 3 hour frame Guard deployment · The robot's movement area for each location · The robot's field of view for each location
· Target location of intruders
· Robot position
· Location of intruders moving toward the target point · Area where the robot can move from the current location
· Target location of intruders
· Robot position for the last 3 hour frame
· Robot field of view and location of intruders captured within the field of view during the last 3 hour frame fire surveillance ·Robot's movement area for each location ·Robot's field of view for each location
· Robot position
· Robot vision
· Fire area · Area where the robot can move from the current location
· Robot position for the last 3 hour frame
· Location of fire areas captured within the robot’s field of view and field of view during the last three hour frame.

Next, the input unit 110 designs a reward function that is set as a goal appropriate for the task selected by the user and a reward condition according to the goal. By defining the goal, it is possible to define the optimal action policy that the mobile robot should take in the work environment. And by designing the reward function to correspond to the goal, the mobile robot can learn the optimal action policy. For example, goals and reward conditions can be defined as shown in Table 2.

target action target Compensation Conditions patrol Robots patrol the work environment to capture intruders wandering around the work environment. · Robot Crash: -10
· Normal: -1
· Intruder detection: -1~0 (increases as the distance becomes closer)
· Intruder Capture: +10 (Operation Ended) delivery of goods Reach the delivery target point by bypassing congested areas that change locations arbitrarily within the work environment · Robot Crash: -10
· Normal: -1
· Near target point: -1~0 (increases as distance becomes closer)
· Goal reached: +10 (task completed) cleaning Visit all areas within the work environment, bypassing congested areas that change location arbitrarily within the work environment · Robot Crash: -10
· Normal: -1
· Increase in cleaned area: -0.5
· Clean all areas: +10 (job finished) Guard deployment Detecting intruders waiting at a specific location and moving toward a target within the work environment · Robot Crash: -10
· Change robot position: -5
· Normal: -1
· Detect intruder: +10
· At the end of work
▷ Number of captured intruders > Number of missed intruders: +10
▷ Number of captured intruders < Number of missed intruders: -10 fire surveillance The robot patrols the work environment and reaches areas where fires occur randomly within the work environment. · Robot Crash: -10
· Normal: -1
· Fire risk detection: -1~0 (increases as the distance increases)
· Fire zone reached: +10 (Operation ends)

And the learning unit 150 trains the DQN artificial intelligence 151 in a designed simulation environment.

Referring to FIG. 3, the simulation environment designed by the learning unit 150 receives a state s _t with C pieces of information in a graph represented by N nodes, and the action value function value Q(s _t for the N nodes , a _t ) is configured to output. And the DQN artificial intelligence 151 selects the action a _t ^* =argmax _at Q(s _t , a _t ) by taking the movement path that maximizes the output action value function value Q(s _t , a _t ). The action value function value is the expected value of the reward received when a movement path is selected for the input state.

For example, the DQN artificial intelligence 151 may be composed of a CNN (Convolutional Neural Network) that applies 1-Dimensional Convolution to the input state.

Referring to FIG. 4, the deep reinforcement learning method of DQN artificial intelligence 151 is an action selection step of DQN artificial intelligence 151, a simulated environment reflection step, a loss function calculation step, and DQN artificial intelligence according to the loss function value. It may consist of calculating the parameters of (151).

In detail, the action selection of the agent (DQN artificial intelligence 151) is, as described in Figure 3, the action that maximizes the action value function value Q(s _t , a _t ) for the state s _t (i.e. Movement path) is selected as a _t ^* =argmax _at Q(s _t , a _t ).

Here, the environmental state is input from the designed simulated environment, an action is selected by the DQN artificial intelligence 151, and the observable state according to the selected action is transmitted to the simulated environment.

Next, the loss function value is calculated from the environmental state of the simulated environment, the action selection of the DQN artificial intelligence 151, and the action value function value, which is a reward according to the selected action. For example, the loss function can be defined using Equation 4.

Next, the DQN artificial intelligence 151 selects an action to maximize the reward given according to the movement path and calculates a parameter that minimizes the loss function value from this. Then, the calculated parameters are updated to the DQN artificial intelligence (151) (agent). For example, the parameters are the movement path, which is an action policy that maximizes the selected task and action value function value in the current environment (i.e., simulated environment).

By repeating the above steps and updating the parameters that minimize the loss function value, the DQN artificial intelligence 151 is subjected to deep reinforcement learning.

Referring to Figures 5 and 6, the driving control unit 160 transmits the selected movement path to the mobile robot in a real environment or a 3D simulation environment through the learned DQN artificial intelligence 151 and enables autonomous driving to perform tasks. Run it.

Referring to FIG. 5, the travel control unit 160 may be configured to include a movement path selection unit 610, an autonomous driving unit 620, a robot driving unit 630, and a simulation unit 640.

Referring to FIG. 6, the movement path selection unit 610 uses the learned DQN artificial intelligence 151 to select a movement path to optimally perform the task in the current state within the work environment (i.e., simulated environment). is transmitted to the autonomous driving unit 620.

The autonomous driving unit 620 may include a control module 621 and an environment recognition module 622.

The environmental recognition module 622 estimates the current position and posture within the mobile robot's working environment, generates an obstacle map in the selected area, and detects targets in the image. Here, the environment recognition module 622 uses sensor measurements input from the robot driving unit 631 and the simulation unit 632, which will be described later, to determine the mobile robot's current location (i.e., current location in the real environment or simulation environment) and The posture can be estimated. And the environment recognition module 622 transmits the recognized environment recognition result to the movement path control module 621.

The control module 621 receives a movement path selection command transmitted from the movement path selection unit 610, and based on the environment recognition result transmitted from the environment recognition module 622, from the current location of the mobile robot to the center coordinates of the selected area. Design a global path, generate a local trajectory to follow the designed global path, and calculate control commands to perform autonomous driving of the mobile robot. And the control module 621 transmits the generated control command to the robot driving unit 630 or the simulation unit 640.

The robot driving unit 630 operates a mobile robot in a real environment and may include a motor control module 631 and a sensing module 632 of the mobile robot.

The motor control module 631 can receive control command signals from the autonomous driving unit 620 and drive and operate the motor of the mobile robot in a real environment.

The sensing module 632 receives sensor measurement values from a plurality of sensors, for example, sensors such as cameras and lidar mounted on the mobile robot, and transmits them to the autonomous driving unit 620.

The simulation unit 640 receives control command signals from the autonomous driving unit 620 and generates and visualizes them as 3D simulations. In addition, the simulation unit 640 is for operating a mobile robot in a 3D simulation environment. Similar to the robot driving unit 630, it operates by driving the motor of the mobile robot in the 3D simulation environment and operates the mobile robot in the 3D simulation environment. Measured values from sensors mounted on the mobile robot can be transmitted to the autonomous driving unit 620.

Meanwhile, Figures 7A to 7E are graphs showing the success rate of each task according to the number of learning episodes when training DQN artificial intelligence according to an embodiment. For reference, Figure 7a is a patrol, Figure 7b is a delivery of goods, Figure 7c is a cleaning, Figure 7d is a guard arrangement, and Figure 7e is a graph showing the learning rate of the reinforcement learning agent for fire surveillance, respectively, at the end of each episode. The task success rate is evaluated and displayed for each task. Referring to the drawing, it can be seen that the task success rate of DQN artificial intelligence increases as the number of learning episodes increases. In addition, from these results, it can be seen that DQN artificial intelligence is learning behavioral policies to optimally perform tasks.

Additionally, the learning rate graph of DQN artificial intelligence shows different aspects depending on the complexity of the task and the settings of the learning parameters. For example, according to the learning rate graph for patrol in Figure 7a, the task success rate is close to 1 when the number of learning episodes is approximately 1,000, and the learning rates for item delivery in Figure 7b, cleaning in Figure 7c, and fire surveillance in Figure 7e According to the graph, you can see that when the number of learning episodes is more than 3,000, the task success rate approaches 1. However, according to the learning rate graph for guard placement in Figure 7d, it can be seen that the task success rate reaches 0.9 even when the number of learning episodes is more than 12,500. From these results, it can be seen that when the task is complex, learning is difficult and the task success rate decreases. In addition, it can be seen from this that among the tasks illustrated, the guard placement task is more complex and difficult to learn than other tasks.

According to this embodiment, the reinforcement learning-based movement path selection system 100 of a mobile robot that performs various tasks uses autonomous driving technology that combines location recognition technology and driving control technology using sensors such as cameras and lidar. By applying it, it is possible to select the optimal movement path to perform the selected task in a mobile robot that performs various tasks such as patrolling and delivering goods. In addition, through deep reinforcement learning of DQN artificial intelligence, the optimal movement path of the mobile robot can be selected to perform tasks efficiently.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

100: Reinforcement learning-based movement path selection system for mobile robots that perform multiple tasks
110: input unit
120: processor
130: memory
140: output unit
150: Learning Department
151: DQN artificial intelligence (agent)
160: Driving control unit

Claims (15)

A step of receiving input parameters of DQN artificial intelligence, including tasks to be performed by the mobile robot and the work environment;
For the input work environment, nodes are assigned to each zone and displayed in a graph, environmental states and observable states for the work are defined, and a goal for the work and a reward function according to the goal are designed to simulate the environment. designing;
Reinforcement learning by having the DQN artificial intelligence select a movement path that maximizes the action value function value in the simulated environment; and
Controlling autonomous driving of the mobile robot in a real environment or simulation environment using learned DQN artificial intelligence;
Reinforcement learning-based movement path selection method for mobile robots that perform various tasks including.
According to paragraph 1,
The reinforcement learning step is,
Calculating a loss function value by reflecting the environmental state of the simulated environment, an action selected by the DQN artificial intelligence, and an observable state according to the selected action;
calculating a parameter by which the loss function value is minimized; and
Updating the DQN artificial intelligence with the calculated parameters;
Including,
A reinforcement learning-based movement path selection method for a mobile robot that performs various tasks including repeatedly performing the step of calculating the loss function value and the step of updating.
According to paragraph 2,
The DQN artificial intelligence is reinforcement learning for mobile robots that perform various tasks consisting of a CNN (Convolutional Neural Network) that applies one-dimensional convolution to a state with multiple information in a graph represented by multiple nodes in the simulated environment. How to select a base movement path.
According to paragraph 1,
The step of designing the simulated environment is,
Create a map of the input work environment,
Divide the work environment into a plurality of zones and assign nodes to each zone,
Displaying the work environment graphically by displaying the possibility of movement from each zone to another zone and the observability of dynamic obstacles,
Reinforcement learning-based movement path selection method for mobile robots performing multiple tasks.
According to paragraph 4,
The environmental state includes information for the mobile robot to perform the task within the work environment,
The observable state includes a state that is accessible while the mobile robot is performing the task in the environmental state,
Reinforcement learning-based movement path selection method for mobile robots performing multiple tasks.
According to paragraph 1,
The step of controlling the autonomous driving of the mobile robot is,
Using the learned DQN artificial intelligence, a movement path selection command is generated to select a movement route to perform a task within the simulated environment,
Generate control commands for autonomous driving of the mobile robot based on the movement path selection command and the recognition result of the work environment,
Operating the mobile robot in a real environment or simulation environment according to the control command,
Reinforcement learning-based movement path selection method for mobile robots performing multiple tasks.
According to clause 6,
The step of controlling the autonomous driving of the mobile robot is,
In a real environment or simulation environment in which the mobile robot is operated, sensor measurement values measured through a plurality of sensors provided in the mobile robot are input, and the current location and posture of the mobile robot are estimated.
Reinforcement learning-based movement path selection method for mobile robots performing multiple tasks.
In the movement path selection system that controls the autonomous driving of a mobile robot that performs various tasks,
An input unit that receives parameters of DQN artificial intelligence, including the tasks to be performed by the mobile robot and the work environment;
A learning unit that designs a simulation environment for reinforcement learning of the DQN artificial intelligence and performs reinforcement learning of the DQN artificial intelligence in the designed simulation environment; and
A driving control unit that controls autonomous driving of a mobile robot in a real or simulated environment using learned DQN artificial intelligence;
Reinforcement learning-based movement path selection system for mobile robots that perform various tasks including.
According to clause 8,
The learning department,
The DQN artificial intelligence selects an action by taking a movement path that maximizes the action value function value,
Calculate parameters that minimize the loss function value by reflecting the behavior and simulated environment selected by the DQN artificial intelligence,
Reinforcement learning of the DQN artificial intelligence by updating the DQN artificial intelligence with the calculated parameters,
Reinforcement learning-based movement path selection system for mobile robots that perform multiple tasks.
According to clause 8,
The simulated environment is,
For the input work environment, a graph displayed by assigning a plurality of nodes to each zone,
An environmental state including information for the mobile robot to perform the task within the work environment and an observable state including an accessible state while the mobile robot is performing the task in the environmental state;
Containing a reward function set as a goal appropriate for the task and a reward condition according to the goal,
Reinforcement learning-based movement path selection system for mobile robots that perform multiple tasks.
According to clause 10,
The graph is a reinforcement learning-based movement path selection system for a mobile robot that performs various tasks, indicating the possibility of movement from each area to another area and the observability of dynamic obstacles.
According to clause 8,
The driving control unit,
A movement path selection unit that generates a movement path selection command to select a movement path for performing a task within the simulated environment using learned DQN artificial intelligence;
an autonomous driving unit that generates a control command for autonomous driving of the mobile robot based on the movement path selection command and a recognition result of the work environment;
a robot driving unit that operates the mobile robot in a real environment according to the control command; and
a simulation unit that operates the mobile robot in a simulation environment according to the control command;
Reinforcement learning-based movement path selection system for mobile robots that perform various tasks including.
According to clause 12,
The autonomous driving department,
A control module that designs a global path from the current location of the mobile robot to a selected area, generates a local trajectory to follow the global path, and generates the control command, and
Comprising an environment recognition module that estimates the position and posture of the mobile robot at the current location within the work environment, creates an obstacle map, and detects a target,
Reinforcement learning-based movement path selection system for mobile robots that perform multiple tasks.
According to clause 13,
The environment recognition module is,
Receiving sensor measurement values measured through a plurality of sensors provided in the mobile robot from the robot driving unit and the simulation unit,
Reinforcement learning-based movement path selection system for mobile robots that perform multiple tasks.
Design a simulated environment for the input task and work environment, select a movement path that maximizes the action value function value in the simulated environment, reinforce DQN artificial intelligence, and operate various types of AI based on the learned DQN artificial intelligence. Mobile robots performing tasks.

Images