KR20230113475A - 3D lidar obstacle detection system using super resolution and reflection intensity - Google Patents

Abstract

The present invention relates to a three-dimensional lidar obstacle detection system using super-resolution and reflection intensity, and relates to a moving object formed to be able to move through autonomous driving, and an object formed on the moving object and existing around the moving object and in the driving direction. A lidar sensor formed to collect object information from the mobile body, a control unit formed on the moving body, storing and editing the object information collected through the lidar sensor, and controlling the movement of the moving body, and formed on the control unit. and a neural network for estimating the shape of the object through super-resolution of the object information collected through the LIDAR sensor and distinguishing the location of the object using reflection intensity.

Description

3D lidar obstacle detection system using super resolution and reflection intensity

The present invention relates to a three-dimensional lidar obstacle detection system using super-resolution and reflection intensity, and more particularly, a mobile vehicle capable of autonomous driving can precisely detect the location and shape of surrounding objects to prevent autonomous driving errors. It is about a 3D lidar obstacle detection system using super-resolution and reflection intensity.

The self-driving vehicle refers to a vehicle having a function of driving itself without a user's manipulation.

An autonomous vehicle may perform autonomous driving without user manipulation by transmitting information obtained from at least one sensor mounted in the vehicle to a server and receiving information about surrounding vehicles and road conditions from the server.

An autonomous vehicle may detect an object around the vehicle using a lidar (light detection and ranging) sensor.

The LiDAR sensor can sense the distance, direction, speed, temperature, material distribution and concentration characteristics of an object by shining a light source called a laser on the target.

Based on the information collected through these LIDAR sensors, the driving direction of the vehicle is determined, and the location of obstacles on the road or other adjacent vehicles is recognized in real time to assist the vehicle in autonomous driving.

However, in the case of lidar sensors, the shape or position of an object is determined based on information received after being reflected on the object.

In order to reduce these errors, cameras or detection sensors are attached to various vehicles and used as auxiliary means, but there is a problem that malfunctions caused by these auxiliary means may stop or cause accidents during autonomous driving.

Korean Patent Registration No. 10-1834124

An object of the present invention for solving the above problems is to change the object information collected through the lidar sensor to a high resolution through a deep learning algorithm when it is determined to be low resolution, so that the object shape and location can be clearly recognized. To provide a 3D LIDAR obstacle detection system using fire and reflection intensity.

Another object of the present invention is to use super-resolution and reflection intensity to clearly recognize two separated objects using reflection intensity when there are objects with different positions among object information collected by reflected light sources. It is to provide a 3D LIDAR obstacle detection system.

In addition, another object of the present invention is a 3D LiDAR obstacle detection system using super-resolution and reflection intensity to increase accuracy by securing data necessary for deep learning without collecting data in the field using a virtual simulation environment. is to provide

The three-dimensional lidar obstacle detection system using super-resolution and reflection intensity of the present invention to solve the above problems is a moving body formed to move through autonomous driving, formed on the moving body, and around the moving body and in the driving direction. A lidar sensor formed to collect object information from an existing object, and a control unit formed on the mobile body to store and edit the object information collected through the lidar sensor and control the movement of the mobile body; It is formed in the controller and characterized in that it includes a neural network for estimating the object shape through super-resolution of the object information collected through the lidar sensor and distinguishing the object position using reflection strength.

In addition, the control unit of the 3D lidar obstacle detection system using super-resolution and reflection intensity of the present invention converts the 3D low-resolution point cloud of the object collected by the lidar sensor into a 2D low-resolution distance image, The shape of the object included in the 2D low-resolution distance image is estimated by a deep learning method through the neural network, the object is converted into a 2D high-resolution distance image by increasing the resolution of the object, and the 2D high-resolution distance image is converted into a 3D high-resolution distance image. It is characterized in that it is converted into a point cloud and used for autonomous driving.

In addition, the neural network of the 3D lidar obstacle detection system using super-resolution and reflection intensity of the present invention is formed to estimate the shape of the object by deep learning, and the object through deep learning using a virtual simulation environment. Characterized in that it is possible to collect data for estimating.

In addition, the neural network of the 3D lidar obstacle detection system using super-resolution and reflection intensity of the present invention uses the pulse intensity that changes according to the surface state of the object among the object information collected from the lidar sensor to detect the object It is characterized in that the reflection intensity of is measured, and each independent object can be distinguished based on the reflection intensity.

In addition, the control unit of the 3D LIDAR obstacle detection system using super-resolution and reflection intensity of the present invention is formed to communicate with an external server, which is necessary for updating the data of the neural network or for deep learning or object estimation. It is characterized in that it can transmit and receive data.

As described above, according to the three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention, when the object information collected through the lidar sensor is determined to be low resolution, it is changed to high resolution through a deep learning algorithm. This has the effect of clearly recognizing the shape and location of the object.

In addition, according to the 3D LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention, when objects with different positions exist among the object information collected by the reflected light source, two separated objects are separated using reflection intensity. It has the effect of clearly recognizing objects.

In addition, according to the three-dimensional LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention, it is possible to secure data necessary for deep learning without collecting data in the field using a virtual simulation environment to increase accuracy. has the effect of

1 is a configuration diagram showing the overall configuration of a three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention.
Figure 2 is an exemplary view showing the order in which the resolution of the three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention is changed.
Figure 3 is an exemplary view showing how the three-dimensional LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention collects data in a virtual simulation environment.
Figure 4 is an exemplary view showing the neural network structure of the three-dimensional LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention.
Figure 5 is an exemplary view showing how the neural network of the three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention removes prediction with high uncertainty.
6 is an exemplary view showing how the lidar sensor of the three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention distinguishes objects through a neural network.
7 is an exemplary view showing reflection intensity collected from a 2-dimensional lidar of a 3-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention.

Specific features and advantages of the present invention will be described in detail below with reference to the accompanying drawings. Prior to this, if it is determined that the detailed description of functions and configurations related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The present invention relates to a three-dimensional lidar obstacle detection system using super-resolution and reflection intensity, and more particularly, a mobile vehicle capable of autonomous driving can precisely detect the location and shape of surrounding objects to prevent autonomous driving errors. It is about a 3D lidar obstacle detection system using super-resolution and reflection intensity.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram showing the overall configuration of a three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention.

As shown in FIG. 1, the 3D LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention is formed on a moving body 100 and a moving body 100 formed to move through autonomous driving. A lidar sensor 200 formed to collect object information from objects existing around the moving body 100 and in the driving direction, and object information formed on the moving body 100 and collected through the lidar sensor 200 The control unit 140 that stores and edits and controls the movement of the moving object 100, and the object information formed in the control unit 140 and collected through the lidar sensor 200 is super-resolved to estimate the object shape, It is characterized in that it includes a neural network 300 that classifies the object location using the reflection intensity.

The movable body 100 may be configured as a vehicle capable of carrying a person or carrying an object, or may be configured to operate unattended without a human being on board, and may be configured to transport an object or operate for a special purpose.

In the present invention, the mobile body 100 is described based on a vehicle, but in practice, the mobile body 100 may be formed as an unmanned drone capable of autonomous flight, and the type of mobile body 100 may vary depending on the purpose of use. be able to

The movable body 100 is formed to recognize lanes along the road for autonomous driving, and can recognize the presence of a vehicle existing around it, thereby preventing an accident from colliding with a vehicle.

In addition, the movable body 100 may be largely composed of a driving device 110, a braking device 120, a steering device 130, and a control unit 140, and the driving device 110 generates power by a motor or an engine. The mobile body 100 is allowed to move, and the braking device 120 is used to decelerate or stop the mobile body 100 during autonomous driving.

The steering device 130 is used to control the moving direction of the mobile body 100, and the control unit 140 controls the autonomous driving speed, direction, location, It is used to overall control the moving body 100 to ensure safety.

At this time, the control unit 140 acquires information existing around the moving body 100 using various GPS, lidar sensors 200, proximity sensors, and cameras formed on the moving body 100 to perform autonomous driving based on safety. be able to

In addition, the lidar sensor 200 is formed in the moving body 100 so that the control unit 140 collects surrounding information, and the lidar sensor 200 emits a light source around the moving body 100 and then reflects it by the object. It is possible to acquire object information existing around the moving object 100 by receiving a light source.

The information collected through the lidar sensor 200 is transmitted to the control unit 140, and the control unit 140 uses the object information collected through the lidar sensor 200 to enable people present on the road and around during autonomous driving. , the presence or absence of objects and animals can be judged.

In addition, the control unit 140 is configured to process object information collected through the lidar sensor 200, and a method of processing the object information by the control unit 140 will be described later with reference to FIG. 2 .

The neural network 300 is formed in the controller 140, and when object information is received by the controller 140 through the lidar sensor 200, the resolution of the object is increased based on the received object information and the shape and location of the object are determined. used to estimate

At this time, the neural network 300 converts the object information collected at low resolution to super-resolution by comparing it with previously collected data using a deep learning method, thereby converting it to high resolution and estimating and predicting information lost due to light source reflection.

In addition, the neural network 300 can determine whether each object is connected or separated through the reflection intensity generated when the light source emitted from the lidar sensor 200 is reflected on the object, and through this, independently separated objects and A unified object can be separated and grasped.

Figure 2 is an exemplary view showing the order in which the resolution of the three-dimensional LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention is changed.

As shown in FIG. 2, the controller 140 of the 3D lidar obstacle detection system using super-resolution and reflection intensity according to the present invention is a 3D low-resolution point cloud of objects collected by the lidar sensor 200. 210 is converted into a 2D low-resolution distance image 220, and the shape of an object included in the 2D low-resolution distance image 220 is estimated through a neural network 300 using a deep learning method to increase the resolution of the object. It is characterized in that it is converted into a 2D high resolution distance image 230, and the 2D high resolution distance image 230 is converted into a 3D high resolution point cloud 240 and used for autonomous driving.

When the light source from the lidar sensor 200 is reflected on the object and returns, the control unit 140 receives it to form a point cloud representing the shape and location of the object. Depending on the performance of the lidar sensor 200 or environmental conditions Errors may occur in the collected information.

Since these errors cause errors in the actual object shape or location, the controller 140 can process object information so that the object shape and location can be clearly detected.

First, when object information is collected through the lidar sensor 200, a 3D low-resolution point cloud composed of points is formed. The controller 140 creates a 3D low-resolution point cloud so that the neural network 300 can analyze the object. It is converted into a 2D low-resolution distance image 220 .

The 2D low-resolution distance image 220 is used to allow the neural network 300 to estimate the shape of an object, and the 3D low-resolution point cloud becomes a panoramic picture when converted to 2D. When converted into dimensions, it is desirable to maintain the shape by applying a correction value so that distortion in which the object is bent or deformed does not occur.

The neural network 300 estimates the shape of an object from the low-resolution distance image 220 that has been converted into two dimensions, and converts it into a high-resolution distance image 230 so that the estimated shape of the object can be clearly expressed.

That is, by estimating the shape of an object based on existing data learned by a deep learning algorithm for an object expressed as a two-dimensional image, it is possible to correct the image to match the actual shape of the object.

Here, the objects may include objects, obstacles, people, animals, plants, traffic lights, streets, roads, buildings, and structures, and the neural network 300 makes the low-resolution image collected through the lidar sensor 200 clear. It is possible to increase the resolution by limiting the estimated shape and increase the DPI in the 2D image.

In addition, the neural network 300 may expand the size of the image when correcting the image according to the shape of the estimated object, and through this, the neural network 300 estimates to a wider range than the detection range of the lidar sensor 200 be able to display

When the low-resolution distance image 220 is converted into the high-resolution distance image 230 by the neural network 300, the controller 140 changes the 2-dimensional high-resolution distance image 230 into a 3-dimensional high-resolution point cloud 240, In autonomous driving, the position and shape of an object are converted so that they can be clearly expressed.

The high-resolution point cloud 240 is compared to the low-resolution point cloud 210 formed through the lidar sensor 200 because the shape of the object is estimated by the neural network 300 and the high-resolution distance image 230 with high resolution is converted. Objects can be clearly displayed, and distances or locations between objects can be clearly displayed, so that they can be used as data for autonomous driving.

3 is an exemplary view showing how the 3D lidar obstacle detection system using super-resolution and reflection intensity according to the present invention collects data in a virtual simulation environment.

As shown in FIG. 3, the neural network 300 of the 3D lidar obstacle detection system using super-resolution and reflection intensity according to the present invention is formed to estimate the shape of an object by a deep learning method, and a virtual simulation environment It is characterized in that data for estimating an object can be collected through deep learning using

The neural network 300 super-resolutions low-resolution object information to high-resolution through a deep learning algorithm. At this time, the data necessary for deep learning is supplied through the virtual simulator 400 and deep learning based on virtual data without actual data. It is possible to secure the data necessary for the algorithm.

In general, for deep learning training, after providing real data, massive real data is used to increase the probability of estimating an object of the neural network 300 by repeating whether the data estimated through the neural network 300 and the actual data match. It is necessary, and there is a problem that it takes a long time to learn it.

In order to solve this problem, the control unit 140 of the present invention enables data to be secured using virtual objects around the moving object 100 in a virtual space through the virtual simulator 400, and inferred through the neural network 300. The object information assists the neural network 300 to learn by determining whether the object information inferred by the virtual simulator 400 matches a virtual object.

In addition, the virtual simulator 400 provides data similar to the information collected through the lidar sensor 200 in high resolution and low resolution, thereby inducing inference learning through data without actually driving the lidar sensor 200. be able to

This iterative process is to be repeatedly performed and learned while the moving object 100 is not traveling, or when the user manually updates the control unit 140, the neural network 300 is controlled to learn through the virtual simulator 400. be able to

In addition, the controller 140 is configured to communicate with the server 500 formed externally, so that it can update data of the neural network 300 or send and receive data necessary for deep learning or object estimation.

The control unit 140 is formed to communicate with the server 500 in real time through wireless communication, and receives traffic conditions through the server 500 to optimize an autonomous driving route.

At this time, since a plurality of 5G antennas are formed on the road, real-time wireless communication with the server 500 is possible through the 5G antennas, and traffic information generated around the 5G antenna, autonomous vehicle operation information, and route information are transmitted through the server 500. Through this sharing, safe autonomous driving becomes possible.

In addition, the control unit 140 connected to the server 500 automatically updates the data of the neural network 300 so as to be up to date, and corrects information about object estimation errors or misjudgments of deep learning algorithms generated in other self-driving vehicles. Data is received and can be used as data.

In addition, object information data used in the virtual simulator 400 can also be downloaded through the server 500. Through data for expressing the shape, standard, and movement of a person or animal of an object in the virtual simulator 400, the virtual The simulator 400 implements various virtual environments so that the neural network 300 can be trained.

4 is an exemplary diagram showing the structure of a neural network 300 of a 3D LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention, and FIG. It is an exemplary view showing how the neural network 300 of the 3D lidar obstacle detection system removes prediction with high uncertainty, and FIG. It is an exemplary view showing how the Ida sensor 200 classifies objects through the neural network 300, and FIG. 7 is a lidar sensor of a 3D lidar obstacle detection system using super-resolution and reflection intensity according to the present invention ( 200) is an exemplary view showing the reflection intensity collected through.

As shown in FIGS. 4 to 7, the neural network 300 of the 3D LIDAR obstacle detection system using super-resolution and reflection intensity according to the present invention applies Monte-Carlo dropout to uncertainty It is characterized by removing this high prediction to generate a point cloud with high accuracy.

When the controller 140 converts the 3D low-resolution point cloud 210 collected from the lidar sensor 200 into a 2D low-resolution distance image 220, the neural network 300 converts the 2D low-resolution distance image In (220), the shape of an object existing in a pixel form is predicted and estimated.

At this time, the neural network 300 is composed of an input layer 310, a hidden layer 320, and an output layer 330, synapses are formed between each layer, and the hidden layer 320 is formed of a plurality of layers.

First, in the input layer 310, a 2D low-resolution distance image 220 is input in units of pixels, and object information and coordinate information according to the input pixels may be provided.

The hidden layer 320 is a layer in which logic and calculation are performed for object estimation, and is configured to estimate the shape or state of an object using a function.

The output layer 330 outputs the object information estimated through the hidden layer 320 in 2D high resolution and outputs the high resolution distance image 230 .

As shown in FIG. 4 , information input to the input layer 310 is connected to each hidden layer 320, so that an object shape is output through the output layer 330 based on each input pixel information.

However, in this case, since point clouds incorrectly received from the lidar sensor 200 or distorted information due to nearby animals and objects moving at high speed may be included, it is necessary to remove unnecessary uncertain points 340 to determine the object shape. there is

To this end, Monte-Carlo dropout is applied to the neural network 300, as shown in FIG. By not reflecting, the object shape can be inferred through clear data.

When two independent first sensing objects 10 and second sensing objects 20 as shown in FIG. 6 are formed to be spaced apart from each other, the neural network 300 to which Monte-Carlo dropout is not applied When used, the first sensing object 10 and the second sensing object 20 can be recognized as connected to each other by the red uncertain point 340 .

However, when using Monte-Carlo dropout, the uncertainty point 340 between the first sensing object 10 and the second sensing object 20 is removed, so that they can be recognized as two independent objects. .

In addition, the neural network 300 measures the reflection intensity of the object using the pulse intensity that changes according to the surface state of the object among the object information collected by the lidar sensor 200, and distinguishes each independent object based on the reflection intensity. characterized by being able to

In addition, the object information collected by the lidar sensor 200 includes the reflection intensity of the light source reflected from the object. The reflection intensity is relatively different depending on the object surface condition, material, and reflection angle, and the intensity of the received pulse Based on this, the reflection intensity can be measured.

In this case, when the reflection intensity is used, it is possible to clearly distinguish whether the object is a separate object or a unified object by comparing the reflection intensity of two or more objects existing at the same location.

In particular, when the uncertainty point 340 between the first sensing object 10 and the second sensing object 20 is removed using Monte-Carlo dropout, reflection intensity information is obtained from the uncertainty point 340. It can be used to determine if two objects are independent objects.

Through this, the first sensing object 10 and the second sensing object 20 can be determined more clearly, and the object shape can be determined by estimating data excluded through the uncertainty point 340 through reflection intensity. there will be

As described above, according to the three-dimensional lidar obstacle detection system using super-resolution and reflection intensity according to the present invention, when the object information collected through the lidar sensor is determined to be low resolution, it is changed to high resolution through a deep learning algorithm. It is possible to clearly recognize the object shape and position, and if there are objects with different positions among the object information collected by the reflected light source, the two separated objects can be clearly recognized using the reflection intensity, and the virtual It has the effect of increasing the accuracy by securing the data necessary for deep learning without having to collect data in the field using the simulation environment.

As described above, the present invention has been described with a focus on preferred embodiments, but those skilled in the art can make the present invention various within the range that does not deviate from the technical spirit and scope described in the claims of the present invention. It can be implemented by modifying or transforming accordingly. Accordingly, the scope of the present invention should be construed by the claims which are written to include examples of these many variations.

10: first sensing object
20: second sensing object
100: mobile body
110: driving device
120: braking device
130: steering device
140: control unit
200: lidar sensor
210: low resolution point cloud
220: low-resolution street image
230: high resolution distance image
240: high resolution point cloud
300: neural network
310: input layer
320: hidden layer
330: output layer
340: Uncertain point
400: virtual simulator
500: server

Claims (5)

A moving body formed to be able to move through autonomous driving;
a LiDAR sensor formed on the mobile body and configured to collect object information from objects existing around the mobile body and in a traveling direction;
a control unit formed on the mobile body to store and edit the object information collected through the lidar sensor and to control the movement of the mobile body;
A neural network formed in the controller and estimating the shape of the object through super-resolution of the object information collected through the LIDAR sensor and distinguishing the location of the object using reflection strength; characterized in that it comprises a
3D lidar obstacle detection system using super-resolution and reflection intensity.
According to claim 1,
The control unit
Converting the 3D low-resolution point cloud of the object collected by the lidar sensor into a 2D low-resolution distance image;
Estimating the shape of the object included in the 2D low-resolution distance image through the neural network using a deep learning method and converting the object into a 2D high-resolution distance image by increasing the resolution of the object;
Characterized in that the 2-dimensional high-resolution distance image is converted into a 3-dimensional high-resolution point cloud and used for autonomous driving
3D lidar obstacle detection system using super-resolution and reflection intensity.
According to claim 1,
The neural network
It is formed to estimate the shape of the object by a deep learning method, and it is characterized in that data for estimating the object can be collected through deep learning using a virtual simulation environment
3D lidar obstacle detection system using super-resolution and reflection intensity.
According to claim 1,
The neural network
Among the object information collected by the lidar sensor, the reflected intensity of the object is measured using a pulse intensity that changes according to the surface state of the object,
Characterized in that each independent object can be distinguished based on the reflection intensity
3D lidar obstacle detection system using super-resolution and reflection intensity.
According to claim 1,
The control unit
Characterized in that it is formed to communicate with an externally formed server so that data of the neural network can be updated or data necessary for deep learning or object estimation can be transmitted and received
3D lidar obstacle detection system using super-resolution and reflection intensity.