KR102303432B1 - System for mapless navigation based on dqn and slam considering characteristic of obstacle and processing method thereof - Google Patents

Abstract

The present invention relates to a mapless navigation system based on DQN and SLAM in consideration of the characteristics of obstacles and a processing method thereof. By reducing the learning time by gradually reducing the size and applying the weight of the safety distance to SLAM differentially according to the characteristics of the obstacles confirmed through image recognition and distance measurement without human intervention, there are several obstacles in the unmanned moving object. And it relates to a DQN and SLAM-based mapless navigation system and its processing method that enable optimal route driving in consideration of the safe distance from people or obstacles in a smart factory environment that collaborates with people.

Description

DQN and SLAM-based mapless navigation system considering the characteristics of obstacles and their processing method

The present invention relates to a mapless navigation system based on DQN and SLAM considering the characteristics of obstacles and a processing method therefor, and more particularly, to a Deep Q Network (DQN)-based for autonomous driving of unmanned moving objects such as robots, vehicles, and drones. Simultaneously SLAM (SLAM) by implementing mapless navigation of Localization and Mapping), considering the characteristics of obstacles that allow unmanned moving objects to perform optimal route driving by considering the safe distance from people or obstacles in a smart factory environment where multiple obstacles exist and collaborate with people. It relates to a mapless navigation system based on DQN and SLAM and a processing method therefor.

In recent years, with the rapid development of high-performance GPUs and CPUs, as the computing speed has rapidly developed, control systems and data analysis using artificial intelligence technology are being actively conducted. Research and development of smart factories grafted onto existing factories and warehouse facilities is rapidly expanding.

The smart factory performs facility control and fault diagnosis through an artificial intelligence-based control system based on data acquired through a plurality of IoT sensors. One of the important elements of such a smart factory is the establishment of a production system for flexible logistics transfer that is different from the existing one-way conveyor belt-based production system. We are responding to the production of a wide variety of products.

To this end, the robot first acquires information about obstacles in the space to be driven in the form of a map through SLAM, synchronizes the positions, and then creates a path to the target and moves.

However, in order to perform SLAM, it is necessary to train by manipulating the location of obstacles and the movable space in the initial stage. This is necessary, thus reducing efficiency.

In addition, since robots used in factories and warehouses are premised on cooperating in the same space as humans, measures to recognize and avoid people and important equipment or increase the safety distance are necessary when performing autonomous driving. For example, if you bump into a computer or expensive equipment, or hit a hard desk, the damage can be greater than just hitting a box. Therefore, there arises a need for the robot to inform the user of the location information of these risk factors while autonomously driving, or to mark it on a map.

That is, conventional robots have a problem in that after manually creating a map through SLAM, characteristics of obstacles, such as locations of important equipment, must be marked on the map through additional work.

Therefore, in the present invention, in order to reduce the learning time by implementing mapless navigation based on DQN for an unmanned moving object, the size of the target is initially doubled and then gradually reduced, and the characteristics of obstacles are identified. This study proposes a method that can be displayed on the map by differentially applying the weight of the safety distance to the SLAM according to the characteristics of each obstacle.

Through this, according to the present invention, an unmanned moving object can autonomously perform SLAM, and in a smart factory environment in which people and various obstacles exist, it is possible to drive on an optimal path to a target point without colliding with people or obstacles.

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters that the present invention intends to achieve differently compared to the prior art will be described.

First, Korean Patent Application Laid-Open No. 2019-0029524 (2019.03.20.) relates to a system and method for training a robot to autonomously navigate a path, detecting the initial placement of the robot at the initializing position, and starting from the initializing position. While demonstrating the navigable route to the robot, generating a map of the navigable route and surrounding environment, detecting the next placement of the robot at the initial position, and causing the robot to map the navigable route from the initial position It is a technical feature to allow at least some autonomous navigation.

That is, the prior art describes a method of enabling a user to train a robot to navigate an environment that the user did not expect in advance by training the robot to travel a route by way of a demonstration.

In addition, Korean Patent No. 2008367 (2019.08.07.) relates to an autonomous driving type mobile robot using artificial intelligence planning and a smart indoor work management system and method using the same. Smart indoor work management system and method for moving the robot to the required place in the home and business place using the app and the robot and the app, taking the necessary image, recognizing the image, and taking the necessary action according to the recognized result is about In addition, instead of the indoor positioning technology, the robot creates an indoor map by itself and uses this map to count the number of wheel rotations of the robot, etc. will be.

That is, in the prior art, in the management of indoor work in a home, building, or factory, etc., even if there is no wireless communication-based control system set in advance indoors, it is possible to reach the object by determining a path to find the object to be controlled by itself, and Describes a system and method capable of recognizing the state of the object and delivering the resulting object recognition information to the controller's smartphone app.

As a result of examining the prior art in the above, the prior art suggests a configuration for training a robot to travel a path by a demonstration, a configuration for reaching the object by determining a path to find an object to be controlled by itself, and the like, and the present invention Compared to , it is somewhat similar in that it trains path movement through artificial intelligence, but uses a gradually decreasing target size to shorten the learning time by implementing DQN-based mapless navigation for autonomous driving of unmanned moving objects. , presents a technical feature of differentially applying the weight of the safety distance to the SLAM according to the characteristics of the obstacle and displaying it on the map. There is a clear technical difference between the technology and the present invention.

In particular, in order to reduce the learning time by implementing DQN-based mapless navigation, the size of the target is initially doubled and then gradually decreased (decrease target size). The weight of the safety distance is differentially adjusted according to the characteristics of obstacles. The configuration of offsetting LiDAR data in order to be applied to SLAM and displayed on a map is a characteristic technical configuration of the present invention that is not presented at all in the prior art.

The present invention was created to solve the above problems, and implements DQN-based mapless navigation for autonomous driving of an unmanned moving object, so as to autonomously perform SLAM of the unmanned moving object, and a method for processing the same aims to provide

In addition, the present invention implements DQN-based mapless navigation to reduce the learning time by reducing the size of the target by doubling initially and then identifying the characteristics such as the type and speed of the obstacle through image recognition and distance measurement. It is another object of the present invention to provide a system and a processing method for differentially applying the weight of the safety distance to the SLAM according to the characteristics of the obstacle and displaying it on the map.

In addition, the present invention provides a system and its processing that allow the unmanned moving object to travel on an optimal path to a target point in consideration of a safety distance without colliding with people or obstacles in a smart factory environment where several obstacles exist and collaborate with people Another object is to provide a method.

A mapless navigation system based on DQN and SLAM considering the characteristics of obstacles according to an embodiment of the present invention is an unmanned moving object from a current state to a next state in a predetermined space where an obstacle including a person, an object, or a combination thereof is located. a learning module for learning angle and distance data between the unmanned moving object and a target point, and lidar data for each predetermined angle measured by the unmanned moving object; In the process of performing the learning, when the obstacle is detected, the rider data offset that offsets the rider data by differentially giving a weight of the safety distance set in advance according to the characteristics including the type, speed, or combination of the obstacle module; and a map generating module that receives the offset rider data and updates and displays the boundary surface for the obstacle on the map.

In addition, the learning module is characterized in that when performing the learning according to each episode, referring to the rider data measured by the unmanned moving object at each step time to select an action.

In this case, the episode is set as one episode when the unmanned moving object arrives at the target point, collides with a wall, an obstacle, or a person, or fails to arrive within a preset time at the target point, and the collision is , characterized in that the collision is determined when the unmanned moving object approaches within a preset range of a wall, an obstacle, and a person.

In addition, the learning module sets the radius of the target point by an integer multiple of an original size when performing the episode for the first time, and whenever the unmanned moving object arrives at the target point according to the execution of the episode, the target point The next episode is performed by reducing the radius of the target point to a preset range, and each episode is performed until the radius of the target point becomes the original size, thereby reducing the learning time by narrowing the search range.

In addition, the lidar data offset module detects the obstacle based on the depth data of the obstacle photographed by the unmanned moving object and the bounding box data according to image recognition to confirm the type of the obstacle, and when the obstacle is detected, the unmanned moving object Calculate the difference between the rider data between the current state and the previous state measured in It is characterized in that a safety distance is differentially given, and an offset of the rider data with respect to the obstacle and the differentially assigned safety distance is performed.

In addition, the method for processing mapless navigation based on DQN and SLAM in consideration of the characteristics of obstacles according to an embodiment of the present invention provides that, in a mapless navigation system based on DQN and SLAM, an obstacle including a person, an object, or a combination thereof is located Learning to learn angle and distance data between the unmanned moving object and a target point, and lidar data for each predetermined angle measured by the unmanned moving object, in order to output an action for the unmanned moving object from the current state to the next state in a predetermined space step; In the process of performing the learning, when the obstacle is detected, the rider data offset that offsets the rider data by differentially giving a weight of the safety distance set in advance according to the characteristics including the type, speed, or combination of the obstacle step; and a map generating step of receiving the offset rider data and updating and displaying the boundary surface for the obstacle on the map.

Also, in the learning step, in the DQN and SLAM-based mapless navigation system, when learning according to each episode is performed, the action is selected by referring to the rider data measured by the unmanned moving object at each step time. .

In this case, the episode is set as one episode when the unmanned moving object arrives at the target point, collides with a wall, an obstacle, or a person, or fails to arrive within a preset time at the target point, and the collision is , characterized in that the collision is determined when the unmanned moving object approaches within a preset range of a wall, an obstacle, and a person.

In addition, the learning step may include setting, in the DQN and SLAM-based mapless navigation system, expanding the radius of the target point to an integer multiple of an original size when the episode is initially performed; performing the next episode by reducing the radius of the target point to a preset range whenever the unmanned moving object arrives at the target point according to the execution of the episode; and performing each episode until the radius of the target point becomes the original size.

In the lidar data offset step, in the DQN and SLAM-based mapless navigation system, the obstacle is detected based on the depth data of the obstacle photographed by the unmanned moving object and the bounding box data according to image recognition, and the type of the obstacle to confirm; calculating a difference in lidar data between the current state and a previous state measured by the unmanned moving object when the obstacle is detected, and confirming the position of the obstacle through the difference in the calculated lidar data; and differentially granting a safety distance to be set from the position of the identified obstacle according to the type of the identified obstacle, and performing an offset of the rider data with respect to the obstacle and the differentially assigned safety distance; further characterized by including.
In the present specification, the DQN- and SLAM-based mapless navigation system refers to a DQN-based mapless navigation system in which the weight of the safety distance is differentially applied to the SLAM according to obstacle characteristics.
Also, the map generation module and step refers to the module and step that are updated and displayed on the map.

As described above, according to the DQN and SLAM-based mapless navigation system and its processing method considering the characteristics of obstacles of the present invention, the size of the target is initially doubled by implementing DQN-based mapless navigation for an unmanned moving object. Then, the learning time is reduced by gradually reducing the learning time, and by identifying characteristics such as the type and speed of the obstacle, the weight of the safety distance is differentially applied to the SLAM according to the characteristics of each obstacle and displayed on the map. In a smart factory environment where people and obstacles exist, it has the effect of reaching the target point in the shortest way without colliding with people or obstacles.

1 is a diagram showing the configuration of a mapless navigation system based on DQN and SLAM considering the characteristics of obstacles according to an embodiment of the present invention.
2 is a diagram for explaining in detail a DQN and SLAM-based mapless navigation process in consideration of the characteristics of obstacles to which the present invention is applied.
3 is a diagram for explaining an experimental environment of mapless navigation based on DQN and SLAM according to an embodiment of the present invention.
4 is a diagram for explaining a target size applied to mapless navigation based on DQN and SLAM according to an embodiment of the present invention.
5 is a diagram for explaining a reward type and a value when performing DQN-based learning applied to the present invention.
6 is a diagram for explaining the structure of a DQN applied to the present invention.
7 is a view for explaining all output actions of the unmanned moving object applied to the present invention.
8 and 9 are diagrams for explaining a method of reducing the target size applied to the present invention and a result of performing DQN-based learning with a general target size.
10 is a diagram for explaining a result of performing DQN-based learning in an environment in which a person to which the present invention is applied stands or walks.
11 is a view for explaining a viewing angle of a depth camera applied to a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
12 is a diagram for explaining a difference in a lidar offset according to a speed applied to a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
13 is a diagram for explaining a lidar offset algorithm applied to a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
14 is a diagram for explaining image recognition and an offset of lidar data by a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
15 is a diagram illustrating results of standing people and objects by a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
16 is a diagram illustrating results of people and objects moving by a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
17 is a diagram illustrating a result of performing autonomous SLAM on a space in which an unmanned moving object is moving by the mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.
18 is a flowchart illustrating in detail an operation process of a method for processing mapless navigation based on DQN and SLAM in consideration of the characteristics of obstacles according to an embodiment of the present invention.

Hereinafter, a preferred embodiment of a DQN- and SLAM-based mapless navigation system in consideration of the characteristics of an obstacle according to the present invention and a processing method thereof will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements. In addition, specific structural or functional descriptions for the embodiments of the present invention are only exemplified for the purpose of describing the embodiments according to the present invention, and unless otherwise defined, all terms used herein, including technical or scientific terms They have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. It is preferable not to

1 is a diagram showing the configuration of a mapless navigation system based on DQN and SLAM considering the characteristics of obstacles according to an embodiment of the present invention.

1, the DQN and SLAM-based mapless navigation system 100 of the present invention is a configuration applied to unmanned moving objects such as robots, vehicles, and drones for autonomous driving in a smart factory, and a virtual environment. It is configured to include a building module 110 , a sensor 120 , a learning module 130 , a lidar data offset module 140 , a map generation module 150 , a control module 160 , a memory 170 , and the like.

In addition, although not shown in the drawing, the DQN and SLAM-based mapless navigation system 100 is a power supply supplying operating power to each component, an input unit for data input for various functions, and managing updates of various operation programs. An update management unit and the like may be additionally included.

Since the virtual environment building module 110 has problems of safety and efficiency when learning is performed in an environment of an actual unmanned moving object, for example, ROS (Robot Operating System) used for robot programming and a physics engine-based This is the part that configures the simulation using a virtual simulation tool (eg GAZEBO).

That is, the virtual environment building module 110 creates a virtual space for the real space of an obstacle including at least one or more people, at least one or more objects, or a combination thereof, which is stationary or moving, and creates an image on the created virtual space. It is to set the start point and the target point of the unmanned moving object equipped with the sensor 120 for recognition and distance measurement (see FIG. 3).

The sensor 120 includes a depth camera 121 and a lidar meter 122 , and outputs image capturing information and distance measurement information to the learning module 130 . At this time, the distance measurement information is output to the lidar data offset module 140 .

In the case of the depth camera 121, the viewing angle is set to 60 degrees and the frame size is set to 640x480, and in the case of the lidar meter 122, the number of samples is set to 60, and the resolution is set to 6 degrees. At this time, the reason for setting the resolution to 6 degrees is to increase versatility by making it possible to respond even to low-performance riders.

In addition, the LiDAR measuring device 122 is a remote sensing device that measures the distance by irradiating an object with a laser and analyzing the reflected light.

The learning module 130 uses the DQN configured as shown in FIG. 6 for learning according to the surrounding environment of the unmanned moving object, a standing or moving person, at least one object ( Example: In a virtual space including equipment with hard or soft surfaces, expensive or inexpensive equipment, etc.) or a combination thereof, learning is performed to output the action value from the current state to the next state of the unmanned moving object. do. That is, when learning according to each episode is performed, at each step time, the data scanned around through the lidar meter 122 provided in the unmanned moving object is collected and stacked by referring to the stack (that is, distance data by the rider) by referring to the action is to choose

More specifically, in order to output the action of the unmanned moving object from the current state to the next state, the learning module 130 includes angle and distance data between the unmanned moving object and a target point, and the data provided in the unmanned moving object. It learns the lidar data for each predetermined angle measured by the lidar meter 122 , and outputs the learned result to the control module 160 .

For example, the learning module 130 sets the radius of the target point by extending the radius of the target point to an integer multiple (preferably twice) of the original size when performing the specific episode for the first time, and according to the execution of the episode, the Whenever an unmanned moving object arrives at the target point, the next episode is performed by reducing the radius of the target point to a preset range (preferably 1%), and each episode is played until the radius of the target point becomes the original size. By doing so, it is possible to shorten the learning time by narrowing the search range.

In this way, if the size of the initial target point is increased by an integer multiple and learning is performed while the size is reduced, the search range of the unmanned moving object is narrowed. This will help avoid situations that take too much time.

In this case, the episode is set as one episode when the unmanned moving object arrives at the target point, collides with a wall, an obstacle, or a person, or does not arrive within a preset time (eg, 200 seconds) at the target point do. In addition, the collision is determined as a collision when the unmanned moving object approaches within a preset range of a wall, an obstacle, and a person.

In the process of performing learning through the learning module 130, the lidar data offset module 140 detects the obstacle, based on image recognition and distance measurement information, characteristics including the type, speed, or a combination thereof. to offset the rider data by differentially assigning a weight of a safety distance set in advance according to the checked characteristics of the obstacle, and output the offset rider data to the map generation module 150 .

That is, the lidar data offset module 140 detects an obstacle including a person or several objects, and offsets the lidar data by changing the weight of the safety distance according to the type of the detected obstacle, and based on this, the map generating module 150 ) to update and display the boundary surface for the obstacle on the map, thereby helping the learning module 130 learn to safely move the unmanned moving object to the optimal path.

More specifically, the lidar data offset module 140 detects the obstacle based on the depth data of the obstacle photographed by the depth camera 121 of the unmanned moving object and the bounding box data according to image recognition. Check the type. At this time, in the present invention, a You Only Look Once (YOLO) network is used as the algorithm for image recognition.

And when the obstacle is detected, the lidar data offset module 140 calculates the difference in lidar data between the current state and the previous state measured by the lidar meter 122 of the unmanned moving object, and then calculates the difference between the calculated lidar data Check the location of the obstacle through

And the lidar data offset module 140 differentially gives a safety distance to be set from the position of the identified obstacle according to the type of the identified obstacle, and rider data for the obstacle and the differentially assigned safety distance perform an offset of

At this time, the offset lidar data is converted into a data message form for ROS and output to the map generation module 150 .

The map generation module 150 receives the lidar data offset from the lidar data offset module 140 and updates and displays the boundary surface for the obstacle on the map.

That is, the information obtained from the lidar data offset module 140 and offset by the distance of the boundary to be set from the position of the obstacle based on the point at which the obstacle is detected is applied to SLAM so that the boundary surface for the obstacle is displayed on the map. to make it displayed.

In this case, the updated map of the boundary surface for the obstacle is used for learning to output an action from the current state to the next state of the unmanned moving object when the next episode is performed in the learning module 130 .

The control module 160 is a part that collectively controls the operations of the virtual environment building module 110 , the sensor 120 , the learning module 130 , the lidar data offset module 140 , and the map generation module 150 . .

The memory 170 stores various operation programs used in the unmanned moving object.

In addition, the memory 170 stores the learning model performed by the learning module 130 , and stores SLAM information implemented through the map generation module 150 .

The DQN and SLAM-based mapless navigation process of the present invention configured as described above will be described in detail with reference to FIG. 2 as follows.

2 is a diagram for explaining in detail a process of mapless navigation based on DQN and SLAM to which the present invention is applied.

As shown in Figure 2, the learning module 130 for each episode, angle and distance information to the target point in a predetermined space containing obstacles including people, objects, or combinations thereof, and rider data for each predetermined angle. Learning is performed by input, and an action value from the current state to the next state of the unmanned moving object is output according to the learning result.

In addition, the lidar data offset module 140, when an obstacle is detected in the process of learning by the learning module 130, the depth data (depth_raw) and raw image data (depth_raw) taken with the depth camera 121 ( image_raw), the bounding box data on which image recognition is performed, and lidar data for each predetermined angle measured by the lidar meter 122 are input to check the characteristics of the detected obstacle, such as the type and speed.

In addition, the lidar data offset module 140 offsets the lidar data by differentially giving a weight of the safety distance set in advance according to the identified obstacle characteristics, and converts the offset data into ROS to generate the map. (150) is output.

At this time, the preset safety distance is set to gradually decrease in the order of a moving person, a standing person, a high-priced device, and a low-cost device.

Accordingly, the map generating module 150 applies the offset information to the SLAM to display the boundary surface for the obstacle on the map.

As described above, a final learning model and a map in which various obstacle characteristics are considered are completed based on the learning for outputting the action of the unmanned moving object and the differential application of the safety distance weight according to the characteristics of the obstacle.

Next, mapless navigation based on DQN and SLAM considering the characteristics of obstacles according to the present invention configured as described above will be described in more detail with reference to FIGS. 3 to 10 .

FIG. 3 is a diagram for explaining an experimental environment of mapless navigation based on DQN and SLAM according to an embodiment of the present invention. In the present invention, as shown in FIG. An unmanned moving body of the real world provided with the measuring device 122 is implemented in a virtual space.

In addition, as shown in FIG. 3B , two fixed cylindrical obstacles are arranged inside a rectangular virtual space of a predetermined size (eg, 6m x 6m). In the experiment of the present invention, the size of the obstacle is set to a radius of 0.3 m, installed at an interval of about 2 m, and the coordinates of the target position are randomly set at eight predetermined coordinates around the obstacle.

In addition, since it is not easy to implement various obstacles in a virtual environment other than a standing person or a moving person, in the present invention, as shown in FIG. After that, place

Based on the virtual environment implemented in FIG. 3 , the DQN applied to the learning module 130 will be described as follows.

Reinforcement learning is based on Bellman Equation created by Bellman based on Markov Decision Process (MDP) for the optimization control problem of the agent, which is the subject of learning and control. First, the MDP is expressed that the first time t = surface starting from any state S ₀ from 0 to as Chance in the immediately previous state S _t-1 probability to get to the current state S _t to the current state S _t which Markov and can be expressed as Equation 1 below

[Equation 1]

Here, the action A _t represents the action taken by the agent, and the reward R _t numerically represents how much the agent moved as intended considering the environment after the action was taken, that is, the surrounding environment. Reinforcement learning generally assumes that the state S _t has a Markov relationship with the previous state S _t-1. A value function for DQN learning is derived using the relationship of When the agent takes an action at each step _{, it probabilistically reaches the state S t due} to disturbance, etc., and immediately obtains a reward according to this state.

At the end of an episode, the higher the sum of the rewards received at each time step, the better the learning.

The State Value Function can be expressed as Equation (2).

[Equation 2]

The reason why it is expressed as an expected value is because it depends on the probability that the _{agent reaches the state S t , and G t} represents the sum of the rewards obtained from the environment and can be expressed as in Equation (3).

[Equation 3]

는 디스카운트 팩터(Discount Factor)를 나타내며 리워드 시간에 따른 차이를 부여한다. 단순히 각 시간마다 얻은 리워드를 더할 경우 시간이 무한대로 감에 따라 리워드의 크기를 다르게 받더라도 반드시 무한대로 수렴하기 때문에 해당 리워드를 받은 때가 시작하자마자 받은 것인지, 미래에 받은 것인지 비교가 불가능하다. DQN은 MDP를 기반으로 하여 미래의 리워드를 확률적으로 얻을 수 있기 때문에 미래의 리워드의 경우 실제로 받을 수 있을지 확실한 것이 아니다. 따라서 Discounting Factor를 사용해 시간에 따른 리워드를 차등 지급함으로써 미래에 받을 것이라 예상되는 리워드에 대한 영향력을 감소시킨다. Discounting Factor는 0에서 1사이로 책정되는데 지나치게 낮을 경우 미래의 리워드를 아예 반영하지 못하므로 본 발명에서는 0.9로 설정한다.In this case, in the above expression

denotes a discount factor and gives a difference according to the reward time. If you simply add the rewards obtained for each time, even if you receive a different size of the reward as time goes to infinity, it necessarily converges to infinity, so it is impossible to compare whether the reward was received immediately after the start or in the future. Because DQN can obtain future rewards probabilistically based on MDP, it is not certain whether future rewards will actually be received. Therefore, by using the Discounting Factor to differentiate rewards over time, we reduce the influence of rewards that are expected to be received in the future. The Discounting Factor is set between 0 and 1, but if it is too low, future rewards cannot be reflected at all, so it is set to 0.9 in the present invention.

In the State Value Function of Equation 2, only the relationship between the state and the reward is considered, but the policy for determining the action of the agent must also be considered. This is due to indicate the probability to take action A _t when the policy π agent at the time t have a state S _t, as represented by " can be expressed as Equation 4 below.

[Equation 4]

를 가질 때 이를 고려한 State Value Function은 수학식 5와 같이 π를 표기하여 나타낼 수 있다.agent policy

When having , the State Value Function taking this into account can be expressed by notating π as in Equation 5.

[Equation 5]

Bellman Equation, which is the basis of Q-learning, is a formula that defines the state value function relationship between _{the state S t+1} at the next time and the current state S _t. have. Substituting Equation 3 into Equation 5, it can be expressed as Equation 6 below.

[Equation 6]

은 수학식 3의 관계를 이용하면 다음의 수학식 7과 같이 만들 수 있다.In Equation 6 above

can be made as in Equation 7 below using the relation of Equation 3.

[Equation 7]

로 나타낼 수 있으므로 이를 대입하면 Bellman Equation은 최종적으로 수학식 8과 같이 정리가 가능하다.In Equation 7, G _t+1 is

Since it can be expressed as , by substituting it, the Bellman Equation can be finally arranged as in Equation 8.

[Equation 8]

는 시간 t + 1에서 상태 S_t+1이 주어졌을 때 받을 수 있는 Discounted State Value Function이다.In Equation 8, R _t+1 is a reward immediately received from the environment when the current state S _{t is given,}

Is Discounted State Value Function can be, given the state S _{t + 1} at time t + 1.

Here, Expectation can be said to arise because it depends on the policy indicating the probability of taking some action a. According to the definition of Expectation, which is the sum of the product of probability and expected gain, Equation 8 can be expressed as Equation 9 below.

[Equation 9]

는 에이전트가 환경에 대하여 확률적으로 움직이는 것을 뜻하며 이는 환경에 대한 모델링을 반드시 필요로 한다. 다시 말하자면 에이전트는 다음의 시간 t+1에서 가능한 모든 상태에 대한 리워드 값을 알아야 하며 동시에 그 상태로 이동하기 위한 액션 및 확률 또한 알아야 한다는 것을 뜻한다. 이것은 모델에 의한 결정론적인 방법이라고 할 수 있는데 실제 환경에 대한 모델링을 하는 것은 매우 어려운 일이다.While performing DQN, the agent must use this value function to move from the state S _{t at the time to the state S t+1} where the highest reward can be obtained. However, in Equation 9 above,

means that the agent moves probabilistically with respect to the environment, which requires modeling of the environment. In other words, the agent must know the reward values for all possible states at the next time t+1, and at the same time also know the actions and probabilities to move to that state. This can be said to be a deterministic method by model, but it is very difficult to model the real environment.

Therefore, DQN uses Action Value Function for a model-free method that does not require modeling of the environment. Finding the Value Function after taking an action in the current state S _{t eliminates the need to find all Value Functions for all possible states S t+1} in the next time, so the expression can be simplified, which is called Action Value Function or Q Function. and is expressed as Equation 10.

[Equation 10]

The Action Value Function is also arranged as in Equation 11 in the same manner as the State Value Function above.

[Equation 11]

는 다음의 수학식 12로 정의할 수 있다.The Action Value Function is _{determined according to the state S t} and the action A _t , and since the Model Free method used in reinforcement learning is based on experience rather than probability, it is summarized as in Equation 11. Optimal Action Value Function, which is the maximum value that can be received from the equation

can be defined by the following Equation 12.

[Equation 12]

는 상기 수학식 12에 따라 현재 상태 S_t를 알 때 에이전트가 취할 수 있는 액션 중

를 최대로 하는 액션 A_t를 통해 구할 수 있으며 이 Optimal Action Value Function을 얻을 수 있는 정책을 최적 정책(Optimal Policy)이라고 한다. 최적 정책은 다음의 수학식 13과 같이 표현할 수 있다.here

is among the actions that the agent can take when the current state S _{t is known according to Equation 12}

It can be obtained through the action A _t that maximizes , and the policy that can obtain this Optimal Action Value Function is called the Optimal Policy. The optimal policy can be expressed as Equation 13 below.

[Equation 13]

_{Here, the optimal policy selects the action A t} that maximizes the Action Value Function, which is called the greedy method. To summarize the above description, DQN implements the Action Value Function as an artificial neural network and selects _{the action A t} having the maximum Q Value or reward according to the _{current state St and the optimal policy.} Therefore, it can be said that the purpose of DQN is to construct and train an artificial neural network representing this Q function.

This Q function can be obtained through Policy Iteration, an iterative process of Evaluation that evaluates the reward of an action, and Improvement, which indicates the update of the neural network.

방법이 사용된다. 이 Policy Iteration에는 크게 두 가지 방법이 사용되는데, 먼저 Dynamic Programming으로 대표되는 Model 기반의 방법은 한 에피소드 내의 상태에 따른 Bellman Equation을 먼저 전부 계산 후에 Optimal Value를 구하게 되며, 이에 따라 타임 스텝마다 마다 업데이트 할 수 있다는 장점이 있다. 그러나 Dynamic Programming의 경우 처음부터 모든 환경에 대한 상태들을 알아야 하므로 모르는 지점을 찾아가기 위해 본 발명에서는 Model Free 방법 중 하나인 Temporal Difference Method를 사용한다. 다른 Model Free 방법 중 하나인 Monte Carlo Method 방법에서는 Value Function을 구하기 위해 우선 한 에피소드를 진행하면서 각 시간에서의 상태 및 리워드를 시계열 데이터로써 메모리에 모두 저장하고 그 후에 메모리에 저장된 데이터로 Value Function을 계산하고 Policy를 업데이트한다. 하지만 에피소드의 종료까지 시간이 과도하게 소요될 경우 업데이트가 너무 늦게 되어 학습이 힘들다는 단점이 존재하며 학습을 이제 막 시작하는 무인 이동체에서는 타겟 지점에 도착하기까지 시간이 오래 소요될 가능성이 있다.Value evaluation means calculating the predicted value of the reward that can be obtained for a given policy and state at time t, and updates the Q function using this value. This process is called improvement, and in DQN,

method is used Two major methods are used for this policy iteration. First, the model-based method represented by dynamic programming calculates the Bellman Equation according to the state in one episode first, then obtains the optimal value. It has the advantage of being able to However, in the case of Dynamic Programming, the Temporal Difference Method, which is one of the Model Free methods, is used in the present invention to find an unknown point because the states of all environments must be known from the beginning. In the Monte Carlo method, which is one of the other model-free methods, in order to obtain the value function, the state and reward at each time are stored in the memory as time series data while proceeding with one episode first, and then the value function is calculated using the data stored in the memory. and update the policy. However, if it takes too much time until the end of the episode, the update is too late and learning is difficult, and there is a possibility that it may take a long time to arrive at the target point in an unmanned moving vehicle that is just starting to learn.

Conversely, the Temporal Difference Method has the advantage that it can be updated every time step like Dynamic Programming. The Monte Carlo Method starts updating only when an episode ends, that is, when it arrives at the correct target point, so it is possible to learn the correct answer, but it has a relatively high variance because it arrives late. Conversely, since the Temporal Difference continues to learn even before reaching the target, the variance is not large, but since it needs to learn on the way, the exact target cannot be used, so the bias is relatively high. Therefore, when using Temporal Difference, trade off considering variance and bias through appropriate selection of interval rather than every time step.

On the other hand, there are two main methods for updating and using the Q function in the Temporal Difference Method: On-Policy and Off-Policy. In On-Policy, the policy used to control the agent and the policy to update are not used separately. In this case, an action is performed according to one policy and learning about the result is repeated. As a result, it is difficult to perform exploration, and for this reason, there is a high risk of falling into the local minimum, and since updates and actions are performed using the same neural network in the early stages of learning, stable learning is difficult because the weight changes due to updates are severe.

In order to improve this shortcoming, one policy is divided into two off-Policy, which means a method to update the target policy in addition to the behavior policy for agent control. This is the general structure of Q Learning. can

The advantage of this structure is that, while updating the target policy as a goal, which is different from the behavior policy, the agent can arbitrarily perform actions, gain experience, and explore the environment.

First, in Q Learning, the target policy uses a greedy method that selects and updates the value with the largest Q Value. The target value of Q to which this is applied can be expressed as in Equation 14 below.

[Equation 14]

Substituting this into Equation 11, it can be summarized as Equation 15 below.

[Equation 15]

방법을 통해 좀 더 넓은 환경을 탐색하고 업데이트를 수행한다.According to the definition of Equation 11 and the role of the target policy, Equation 15 becomes the target value of the artificial neural network. However, when using this method, because the neural network always trains only the Max Q Value in front, it is impossible to search for an action that can receive a higher reward in the future, and it is difficult to apply it in various environments. Therefore, in Q Learning, it is not directly used for updating.

The method explores a wider environment and performs updates.

는 Target Policy에 의한 액션의 선택 외에 임의의 상수와 1 이하의 상수인

을 이용하여 확률적으로 임의의 행동을 취함으로써 greedy 방법으로는 관찰할 수 없는 주변 환경을 탐험하는 것을 뜻한다. 이때 얻은 Observed States로 Policy의 업데이트를 수행하는 것이다. 이것은 개념적으로는 에이전트가 액션을 취하였으나 외란 등에 의하여 의도한 곳과 다른 곳에 도착했음을 표현한다. 에피소드가 지나감에 따라

은 1에서 시작해 0이 아닌 양의 실수까지 특정한 비율로 감소하게 되며 이를 통한

방법은 수학식 16 및 17로 표현할 수 있다.

is an arbitrary constant and a constant less than or equal to 1 other than the selection of an action by the target policy.

It means to explore the surrounding environment that cannot be observed by the greedy method by taking random actions in a probabilistic way using Policy update is performed with the Observed States obtained at this time. This conceptually expresses that the agent has taken an action but has arrived at a different place than intended due to disturbance or the like. As the episodes go by

starts at 1 and decreases at a certain rate until a non-zero positive real number.

The method can be expressed by Equations 16 and 17.

[Equation 16]

[Equation 17]

는 시간에 따른

의 변화를 나타내며 이는 에피소드의 진행에 따라 최소 0.05까지 감소된다. 따라서 에이전트의 액션은 기본적으로 Q Value를 최대로 하는 것을 선택하되 확률적으로

에 따라 임의의 액션을 취하여 탐험을 수행하게 된다.In Equation 16 above

is over time

, which decreases to at least 0.05 as the episode progresses. Therefore, the agent's action basically chooses the one that maximizes the Q Value, but probabilistically

Depending on the action, the exploration is performed.

방법을 통해 에이전트의 액션이 정해지면 수학식 15에 대입하여

를 구하고 다음의 수학식 18에 의해 Policy를 업데이트 한다.remind

When the action of the agent is determined through the method, it is substituted into Equation 15

, and update the Policy according to Equation 18 below.

[Equation 18]

은 Q Learning을 위한 Target이 되며 그에 대한 Error는

와 같이 나타낼 수 있다.In Equation 18 above

is the target for Q Learning, and the error for it is

can be expressed as

Meanwhile, in DQN, several algorithms are added to improve performance. Experience Replay is one of the methods that allowed DQN to develop. Instead of updating the data at each time directly to the neural network, it first stores it in memory, and then selects a random sample as much as the batch size when the data is accumulated to some extent. learn This is because, when the time series data obtained while driving an unmanned vehicle is used as an input to the neural network for learning, similar values and types of data are continuously input and learned, and there is a risk that the network falls into the local minimum. Therefore, this was followed when implementing DQN in the present invention, and the batch size is set to 64. And in order to shorten the learning time, the method of changing the target size shown in FIG. 4 is used.

4 is a diagram for explaining a target size applied to mapless navigation based on DQN and SLAM according to an embodiment of the present invention.

As shown in FIG. 4 , in DQN, an Optimal Policy that can receive the most rewards is found through exploration and search as described above. However, there is a possibility that excessive time is required due to the nature of reinforcement learning, which repeats Trial and Error (learning by direct encountering) in this process. Therefore, the present invention proposes a method of narrowing the range searched by the agent by expanding the range of the target point.

Similar to the Epsilon Greedy method, this method narrows the search range of the unmanned moving object by doubling the size of the initial target point as shown in FIG. 4 at the beginning of training. In addition, as the episode ends due to the arrival of the unmanned moving object to the target point, the size of the target point is gradually reduced so that it is eventually the same as the size of the existing target point. For example, in the present invention, the size of the initial target point is doubled in the virtual space implemented as shown in FIG. 3, and each time the unmanned moving object arrives at the target point according to the learning by each episode, it is reduced by 1%. proceed with learning. The learning proceeds until the original target size is reached. This is to prevent a situation in which an unmanned moving object cannot find a target point and takes an excessively long time when the target point is in a difficult location while performing DQN.

In addition, in the experimental environment of the present invention, a reward according to the environment is set as shown in FIG. 5 .

5 is a view for explaining a reward type and value when performing DQN-based learning applied to the present invention. In the present invention, a driving reward, a target arrival reward, and a collision reward are shown. separate

, 상기 무인 이동체와 타겟 지점간의 거리를

이라할 때 주행 리워드는 다음의 수학식 19 내지 21로 나타낼 수 있다.The driving reward is a reward received at every time t, and is calculated by adding Heading Reward and Distance Reward according to the heading direction of the target point and the unmanned moving object. The angle between the unmanned moving object and the target point at time t

, the distance between the unmanned moving object and the target point

In this case, the driving reward may be expressed by Equations 19 to 21 below.

[Equation 19]

[Equation 20]

[Equation 21]

은 상기 무인 이동체와 타겟 지점이 이루는 각도를 의미하고, 수학식 20에서

는 에피소드가 시작하였을 때 상기 무인 이동체와 타겟 지점간의 거리를 나타낸다.In Equation 19 above

denotes an angle between the unmanned moving object and the target point, and in Equation 20

denotes the distance between the unmanned moving object and the target point when the episode starts.

Meanwhile, in the present invention, the network of FIG. 6 is configured to learn driving according to the surrounding environment of the unmanned moving object.

6 is a diagram for explaining the structure of the DQN applied to the present invention, and the input of the network is the scan data of the rider and the distance and direction to the target point.

The artificial neural network of the present invention is constructed using a Fully Connected Network and a Dropout Layer that directly connect the neural networks to each other as shown in FIG. 6 .

The Drop Out Layer intentionally blocks data propagation from some neurons to the next when training an artificial neural network. This prevents the neural network from being overfitted with respect to the Learning Data Set used for learning, and is set to 20% in the present invention. Each neural network is initialized with a Gaussian distribution, and the activation function of the input layer and the hidden layer is ReLU, and the output layer uses the Linear function as the activation function to output the result.

The Observed States used as the input of the DQN are the measured distance at 6-degree intervals obtained through LiDAR, and the angle and distance between the unmanned vehicle and the target point, and 62 pieces of data are input. The output layer determines the action of the unmanned moving object, and the speed of the unmanned moving object according to the determined action is shown in FIG. 7 .

7 is a view for explaining all output actions of the unmanned moving object applied to the present invention, in which linear velocity and angular velocity for each of five actions of 0 to 4 are set.

The optimizer that determines the update method uses RMSprop, a kind of stochastic gradient descent. Unlike the existing gradient descent that calculates all data, stochastic gradient descent uses a mini batch size to speed up the update rate, and it originates from the method proposed by Geoffrey Hinton.

, 입력을

, 출력 레이어의 출력을

그리고 E는 Cost Function이라 할 때 다음의 수학식 22 내지 24를 이용하여 인공 신경망의 가중치를 업데이트한다.In the case of general gradient descent, the weight of the i-th hidden layer is

, input

, the output of the output layer

And, when E is a cost function, the weight of the artificial neural network is updated using Equations 22 to 24 below.

[Equation 22]

[Equation 23]

[Equation 24]

는 Learning Rate를 뜻하는데 이때 각각의 가중치를 업데이트하면서 가중치의 변화량에 따라 이를 변화시키는 것이 RMSprop이다. 기본적인 개념은 많이 변화한 가중치는 Optimal Value에 상대적으로 가까울 것이므로 Learning Rate를 감소시켜 Local Minimum에 빠지는 것을 방지하고 그렇지 않은 것은 많이 변화시켜 Optimal Value에 가까이 이동하도록 하는 것이다. RMSprop를 수학식으로 나타내면 다음의 수학식 25와 같다.Equation 22 represents the output of the i-th hidden layer, and Equation 23 represents the correlation between the cost function and the weight. This is due to the fact that an optimal weight can be found if the weight is updated in a direction in which the change amount of the cost function with respect to the weight continues to decrease, which is expressed as Equation (24). in Equation 24

is the learning rate, and RMSprop is to update each weight and change it according to the amount of weight change. The basic concept is to reduce the learning rate to prevent falling into the local minimum because the weights that change a lot will be relatively close to the optimal value, and to change the weights that do not change a lot to move closer to the optimal value. RMSprop is expressed as an equation as shown in Equation 25 below.

[Equation 25]

In Equation 25, the magnitude of change between weights can be compared through _{G t ,} which can be expressed as a sum of square by continuously accumulating a predetermined ratio of the squared value of the cost function with respect to the weight.

Using this, if Equation 24, which is the equation of the existing gradient descent update, is changed, it can be expressed as Equation 26 below.

[Equation 26]

는 10^-4 이하의 작은 숫자로 설정하며 Learning Rate가 0으로 나눠지는 것을 방지한다.In Equation 26 above

is ^{set to a small number less than 10 -4} and prevents the learning rate from being divided by zero.

8 and 9 are diagrams for explaining a method of reducing the target size applied to the present invention and a result of performing DQN-based learning with a general target size.

In FIGS. 8 and 9, No Decreasing represents the results of experiments without increasing or decreasing the target point size, and Decreasing represents the results of performing simulations while changing the size of the target point. (a) is the sum of rewards obtained from each episode, (b) is Max Q Value, and (c) is accuracy. The size of the target point decreases by 1% and finally decreases to the original size. When the target point arrives 70 times, the size of the target point has the same size as the original size. In the case of FIG. 8, the target point arrived 70 times after 600 episodes, and when comparing the two cases in (c), the accuracy shows a difference of about 27% in 1000 episodes.

In addition, when the accuracy of FIG. 8(c) was compared with the result of the additional experiment in FIG. 9, the accuracy increased rapidly to about 40% in FIG. 8 and episode 1500 or less, and thereafter, a difference of 20% or more was observed. visibility can be verified.

10 is a diagram for explaining a result of performing DQN-based learning in an environment in which a person to which the present invention is applied stands or walks.

As shown in FIG. 10, in the present invention, a test for mapless navigation was performed in an environment in which a person is standing or walking, and the results of performing 1000 episodes in each case are shown in (a) and (b). as shown. Here, all cases of arriving at the target point were expressed as goal and optimal path driving as optimal, and optimal path driving means driving the shortest distance while avoiding obstacles as shown in FIG. It was classified based on the time required and the size of the reward obtained.

Next, the offset algorithm according to the characteristics of the obstacle according to the present invention configured as described above will be described in more detail with reference to FIGS. 11 to 16 .

First, in the present invention, obstacles including people and objects are detected and the weight of the safety distance is changed according to the type to learn so that an unmanned moving object can safely move to an optimal path. Therefore, the YOLO algorithm is used to recognize each obstacle as an image.

For image recognition, 640x480 raw image data received from a virtual depth camera is used as an input of the YOLO network for image recognition. The feature of YOLO is that it can quickly classify many and diverse objects based on CNN. In addition, it is possible to obtain bounding box data having data on the type and location of the obstacle.

In the conventional SLAM, since a map is created using only the rider's distance data, there is a need to manually mark or prevent the user from entering the restricted area of the space, the type of obstacle, and the like. Therefore, after classifying the types of obstacles using the YOLO network, weights are given to the safety distances according to the types, and the safety distances are expressed differently on the map according to the importance of the obstacles in advance when the unmanned vehicle performs SLAM. can have

To this end, according to the type and location of the obstacle detected by the YOLO network, the size of the obstacle should be virtually increased by offsetting the lidar data for use in SLAM. In the present invention, an algorithm using a depth camera and a lidar meter is presented. A depth camera and a lidar are mounted on the front of the unmanned vehicle, and the coordinates of the bounding box follow the coordinate system shown in FIG. 11 and have values between (0, 0) and (640, 480).

11 is a view for explaining a viewing angle of a depth camera applied to a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention. The frame of view of the depth camera is 60 degrees, and a lidar for SLAM The resolution of the data was set to 1 degree. Therefore, depth and image data are overlapped within 30 degrees of the front left and right angles of the rider, and the distance to the obstacle in front can be obtained through sensor fusion.

를 다음의 수학식 27과 같이 구할 수 있다.The data of FIG. 11 can be obtained through this method, d _h is the distance from the unmanned moving object to the obstacle, and P _x is the x-axis coordinate of the obstacle at the viewing angle. And the angle between the center of the unmanned moving object and the obstacle using _{P x}

can be obtained as in Equation 27 below.

[Equation 27]

는 화면을 y축을 기준으로 반으로 나누었을 때 화면상 x축의 좌표는 320, 각도는 30도이기 때문에 위와 같이 나타내었으며 x 좌표의 변화량은 다음의 수학식 28과 같이 나타낼 수 있다. 그리고 수학식 27의 관계에서 각도의 변화량은 수학식 29와 같이 구할 수 있다.here

When the screen is divided in half based on the y-axis, the coordinates of the x-axis on the screen are 320 and the angle is 30 degrees, so it is expressed as above, and the amount of change in the x-coordinate can be expressed as Equation 28 below. And in the relation of Equation 27, the amount of change of the angle can be obtained as Equation 29.

[Equation 28]

[Equation 29]

When the location of the obstacle at time t-1 and time t is known from the unmanned moving object, the distance moved per unit time of the obstacle within the screen, that is, the speed, can be obtained by using the triangle rule of Equation 30 below.

[Equation 30]

Through this, the dynamic characteristics of the position and speed of the obstacle in front of the unmanned moving object can be checked, and the weight of the safety distance is differentially set accordingly.

, i는 안전거리 경계면의 길이라 할 때, 장애물의 위치로부터 설정해야 할 경계면의 거리

는 수학식 31과 같이 표현된다.weights according to the characteristics of obstacles

, i is the length of the safety distance boundary, the distance of the boundary to be set from the position of the obstacle

is expressed as in Equation 31.

[Equation 31]

에 차등적으로 결정되는데, 도 12와 같이 각각 [10, 12, 14, 16]로 설정되는 4단계의 vel 값으로 나눈다.The length of the offset is the speed of the obstacle

is determined differentially, and divided by vel values of 4 steps set to [10, 12, 14, 16], respectively, as shown in FIG. 12 .

12 is a diagram for explaining the difference in the rider offset depending on the speed applied to the mapless navigation system based on DQN and SLAM according to an embodiment of the invention, and the offset of the rider data increases left and right as the speed increases. The higher the obstacle, the greater the safety distance, the wider the boundary line for the obstacle.

In order to apply this to SLAM, it is necessary to change the lidar data of the position where the obstacle is detected. The resolution of the lidar for SLAM is 1 degree, and the position is estimated through the algorithm of FIG.

13 is a diagram for explaining a lidar offset algorithm applied to a mapless navigation system based on DQN and SLAM according to an embodiment of the present invention.

및 속력

을 계산한다. 그 후 장애물의 위치를 파악하기 위하여 라이더 데이터의 차이(difference) 값을 사용한다. 라이더 데이터는 장애물이 존재하는 구간에서 급변하기 때문에 이를 이용해 영상을 통해 측정된 위치

에서의 장애물의 존재 여부를 판단하고 보조 역할을 수행한다.As shown in FIG. 13 , for example, when the y coordinate corresponding to the height of the unmanned moving object among the PointCloud data of the depth camera is 440, the Point Cloud Data P _x of all corresponding y-axis coordinates is obtained, and through this, the unmanned Angle between moving object and obstacle

and speed

to calculate Then, the difference value of the rider data is used to determine the position of the obstacle. Since the lidar data changes rapidly in the section where there are obstacles, the position measured through the video

It judges the presence of obstacles in the system and plays an auxiliary role.

만큼 오프셋을 수행한다. 그리고 SLAM 패키지에 적용될 수 있도록 하여 장애물에 대한 경계면이 SLAM의 화면 및 맵 상에도 적용될 수 있도록 한다.If the location of the point by the video and the location of the obstacle discovered by the rider are similar, it can be determined that there is an obstacle at the location, and based on this point

Offset as much as And by making it applicable to the SLAM package, the boundary surface for obstacles can also be applied to the screen and map of the SLAM.

를 계산하고 라이더 데이터의 오프셋을 수행한 결과는 도 14와 같다.like this

The result of calculating and performing the offset of the rider data is shown in FIG. 14 .

14 is a diagram for explaining image recognition and an offset of lidar data by a mapless navigation system based on DQN and SLAM according to an embodiment of the invention. When an unmanned moving object recognizes a person through an image, the lidar data is actually It can be seen that the boundary line is created by the safety distance in front of the position of the obstacle.

In addition, the performance results in an environment in which a person stands or moves through FIG. 14 are the same as in FIGS. 15 and 16 .

15 is a diagram showing the results of a person and an object standing by a mapless navigation system based on DQN and SLAM according to an embodiment of the invention, and FIG. 16 is a mapless navigation based on DQN and SLAM according to an embodiment of the invention. It is a diagram showing the results of people and objects moving by the system.

In FIGS. 15 and 16 , green marks indicate rider data currently recognized by the SLAM program, and black marks indicate places recognized as obstacles by SLAM. Through this, it can be confirmed that the boundary line indicating the safety distance is applied to the map in front of the location where the actual obstacle exists.

In addition, when comparing the rider data, it can be seen that Fig. 15 in which a person is stationary is shorter than the boundary line of Fig. 16, which is an environment in which a person is walking, and thus it can be confirmed that the offset of the rider data is applied.

In addition, it can be seen that the two obstacles recognized through YOLO are recognized simultaneously, a walking human and a monitor, and the safety distance is applied at the same time.

17 is a diagram showing the result of performing autonomous SLAM on a space in which an unmanned moving object is moving by the mapless navigation system based on DQN and SLAM according to an embodiment of the present invention. It is the result of performing autonomous SLAM on the space in which the unmanned moving object is moving simultaneously.

Next, an embodiment of the DQN and SLAM-based mapless navigation processing method in consideration of the characteristics of obstacles according to the present invention configured as described above will be described in detail with reference to FIG. 18 . In this case, the order of each step according to the method of the present invention may be changed by the environment of use or by those skilled in the art.

18 is a flowchart illustrating in detail an operation process of a method for processing mapless navigation based on DQN and SLAM according to an embodiment of the present invention.

First, an obstacle including an unmanned moving object and a person, an object, or a combination thereof is implemented as a virtual environment through the virtual environment building module 110 (S100). That is, when learning an unmanned moving object in a real environment, safety and efficiency are lowered, so a virtual simulation is constituted.

Thereafter, the learning module 130 is configured to output an action for the unmanned moving object from the current state to the next state in a predetermined space in which an obstacle including a person, an object, or a combination thereof is located, between the unmanned moving object and the target point. A learning step of learning the angle and distance data of , and lidar data for each predetermined angle measured by the unmanned moving object is performed.

When the learning step is described in detail, the learning module 130 performs the step of setting the radius of the target point by expanding the radius of the target point to an integer multiple (eg, 2 times) of the original size when the episode is first performed ( S200). At this time, in the present invention, one episode is set when the unmanned moving object arrives at the target point, collides with a wall, an obstacle, or a person, or does not arrive at the target point within a preset time.

After expanding the size of the target point through the step S200, the learning module 130 inputs the angle and distance between the unmanned moving object and the target point according to the episode, and rider data for each predetermined angle measured by the unmanned moving object, Learning to output the action of the next state proceeds (S300).

At this time, when the learning module 130 performs learning according to each episode, the stack (that is, by the rider distance data) to select an action.

After performing the learning of a specific episode through the steps S200 and S300, the rider data offset module 140 detects the obstacle in the process of performing the learning. Characteristics including the type, speed, or combination thereof. The rider data offset step of offsetting the rider data by differentially giving the weight of the safety distance set in advance is performed (S400).

If the lidar data offset step of the S400 is described in more detail, the lidar data offset module 140 detects the obstacle based on the depth data of the obstacle photographed by the unmanned moving object and the bounding box data according to image recognition. Check the type of obstacle. And when the obstacle is detected, a difference in lidar data between the current state and the previous state measured by the unmanned moving object is calculated, and the position of the obstacle is confirmed through the difference in the calculated lidar data. And, according to the type of the identified obstacle, a safety distance to be set from the position of the identified obstacle is differentially given, and the rider data is offset with the obstacle and the differentially assigned safety distance.

In this way, after the rider data is offset by differentially weighting the safety distance according to the characteristics of the obstacle through the step S400, the map generation module 150 receives the offset rider data and creates a boundary surface for the obstacle. A map generation step of updating and displaying the map is performed (S500).

Thereafter, the learning module 130 determines whether the episode ends when the unmanned moving object arrives at the target point according to the execution of the episode (S600).

As a result of the determination in step S600, when the episode is ended because the unmanned moving object arrives at the target point, the learning module 130 reduces the radius of the target point to a preset range (eg, 1%), and then Perform the episode (S700).

However, as a result of the determination in step S600, if the unmanned moving object does not arrive at the target point, the unmanned moving object collides with an obstacle or fails to arrive at the target point within a preset time, and thus the episode ends, the learning module 130 ) ends the current episode and proceeds to the next episode (S800), and repeats after step S300.

When the radius of the target point is reduced to a preset range (eg, 1%) through step S700 and the next episode is proceeded, the learning module 130 determines that the size of the target point is the original size (ie, all episodes). It is determined whether the size is the same as or most similar to (S900), and if the size of the target point is larger than the original size as a result of the determination, the radius of the target point is the original size for each episode in the S300 Repeat after step.

However, when the size of the target point becomes the original size as a result of the determination in step S900, the learning module 130 ends the learning according to each episode, and finally completes the SLAM in which weights for each obstacle are differentially assigned ( S1000).

As such, the present invention implements DQN-based mapless navigation for an unmanned moving object to reduce the learning time by initially doubling the size of the target and then gradually reducing it, and identifying the characteristics such as the type and speed of the obstacle to identify each obstacle. By differentially applying the weight of the safety distance to SLAM and displaying it on the map according to the characteristics of It is possible to reach the target point by the shortest path without collision.

As described above, the present invention has been described with reference to the embodiments shown in the drawings, which are merely exemplary, and those of ordinary skill in the art can make various modifications and equivalent other embodiments therefrom. You will understand that it is possible. Therefore, the technical protection scope of the present invention should be determined by the following claims.

100 : DQN and SLAM based mapless navigation system
110: virtual environment building module 120: sensor
121: depth camera 122: rider meter
130: learning module 140: rider data offset module
150: map generation module 160: control module
170: memory

Claims (10)

In order to output an action for the unmanned moving object from the current state to the next state in a predetermined space where an obstacle including a person, an object, or a combination thereof is located, angle and distance data between the unmanned moving object and a target point, the unmanned moving object a learning module for generating a learning model by performing DQN-based learning by inputting the rider data for each predetermined angle measured in ; and
In the process of performing the learning in the learning module, when the obstacle is detected, the weight of the safety distance set in advance is differentially applied to the rider data according to the characteristics including the type, speed, or a combination thereof. It includes; a lidar data offset module that determines the offset to be performed.
The learning module is
When learning is first performed for an arbitrary episode, the radius of the target point is set by expanding an integer multiple of the original size,
Whenever the unmanned moving object arrives at the target point according to the DQN-based learning, the radius of the target point is reduced to a preset range to perform learning for the next episode,
It further comprises reducing the learning time by performing learning for each episode by narrowing the search range until the radius of the target point becomes the original size,
A DQN-based mapless navigation system in consideration of the characteristics of the obstacle that receives the rider data to which the determined offset is applied and applies it to the SLAM so that the boundary surface for the obstacle is updated and displayed on the map. The method according to claim 1,
A DQN-based mapless navigation system, characterized in that when learning according to each episode is performed, an action is selected by referring to the rider data measured by the unmanned moving object at each step time. The method according to claim 1,
The episode is
or the unmanned moving object arrives at the target point; collide with a wall, obstacle or person; Alternatively, a case in which the target point does not arrive within a preset time is set as one episode,
The collision is
A DQN-based mapless navigation system, characterized in that it is determined as a collision when the unmanned moving object approaches within a preset range of a wall, an obstacle, or a person. delete The method according to claim 1,
The lidar data offset module,
Detecting the obstacle based on the depth data of the obstacle photographed by the unmanned moving object and the bounding box data according to image recognition, and confirming the type of the obstacle,
Calculates the difference in lidar data between the current state and the previous state measured by the unmanned moving object, and checks the position of the detected obstacle through the difference in the calculated lidar data,
DQN-based mapless navigation system, characterized in that the offset to be applied to the rider data is determined by differentially giving a safety distance to be set from the position of the detected obstacle according to the type of the identified obstacle. In a DQN-based mapless navigation system, in a predetermined space where an obstacle including a person, an object, or a combination thereof is located, in order to output an action for an unmanned moving object from a current state to a next state, between the unmanned moving object and a target point A learning step of generating a learning model by performing DQN-based learning by inputting the angle and distance data of , and rider data for each predetermined angle measured by the unmanned moving object; and
In the process of performing the learning in the learning step, when the obstacle is detected, the weight of the safety distance set in advance according to the characteristics including the type, speed, or a combination thereof is differentially applied to the rider data Including; a rider data offset step of determining the offset to be taken;
The learning step is
When learning is first performed for an arbitrary episode, the radius of the target point is set by expanding an integer multiple of the original size,
Whenever the unmanned moving object arrives at the target point according to the DQN-based learning, the radius of the target point is reduced to a preset range to perform learning for the next episode,
It further comprises reducing the learning time by performing learning for each episode by narrowing the search range until the radius of the target point becomes the original size,
A DQN-based mapless navigation processing method in consideration of the characteristics of the obstacle, which receives the rider data to which the determined offset is applied and applies it to the SLAM so that the boundary surface for the obstacle is updated and displayed on the map. 7. The method of claim 6,
The learning step is
In the DQN-based mapless navigation system, when learning according to each episode is performed, at each step time, an action is selected by referring to the rider data measured by the unmanned moving object. processing method. 7. The method of claim 6,
The episode is
or the unmanned moving object arrives at the target point; collide with a wall, obstacle or person; Alternatively, a case in which the target point does not arrive within a preset time is set as one episode,
The collision is
DQN-based mapless navigation processing method, characterized in that it is determined as a collision when the unmanned moving object approaches within a preset range of a wall, an obstacle, or a person. delete 7. The method of claim 6,
The lidar data offset step is,
in the DQN-based mapless navigation system, detecting the obstacle based on the depth data of the obstacle photographed by the unmanned moving object and the bounding box data according to image recognition to confirm the type of the obstacle;
calculating the difference in lidar data between the current state and the previous state measured by the unmanned moving object, and confirming the position of the detected obstacle through the difference in the calculated lidar data; and
DQN-based mapless, characterized in that it further comprises the step of determining an offset to be applied to the rider data by differentially giving a safety distance to be set from the position of the detected obstacle according to the type of the identified obstacle How to handle navigation.

Images