KR20210090384A - Method and Apparatus for Detecting 3D Object Using Camera and Lidar Sensor - Google Patents

Abstract

The present embodiment relates to a method and device for detecting a 3D object using a camera and a lidar sensor. The method generates matching data by matching an image collected using a camera and point cloud information collected using the lidar sensor, and detects a three-dimensional object region by using the generated matching data as an input of a learning model, so as to provide more accurate and improved object recognition results compared to the prior art in relation to the lidar sensor.

Description

Method and Apparatus for Detecting 3D Object Using Camera and Lidar Sensor

This embodiment relates to a method and apparatus for detecting a 3D object using a camera and a lidar sensor. More particularly, it relates to a camera and a 3D object detection method and apparatus using a lidar sensor that improve the recognition rate of a lidar sensor by using image information of a camera in an autonomous vehicle.

The content described in this section merely provides background information for the present embodiment and does not constitute the prior art.

As vehicle intelligence progresses, studies on advanced driver assistance systems (ADAS) and autonomous vehicles are being actively conducted.

Among them, many research achievements have already been made on the problem of recognition and classification of 2D images. However, in the case of many autonomous driving controllers being developed recently, 3D recognition of objects is essential. This is because there is a limitation in that it is difficult to control driving only by image recognition of a vehicle ahead, and in particular, if the location of a pedestrian is not accurately estimated, the risk of an accident greatly increases.

In the conventional case, various sensors have been used for three-dimensional recognition of an object. In particular, in the case of lidar, since it has the advantage of obtaining accurate distance information compared to other sensors, most autonomous or semi-autonomous driving vehicles have a lidar device mounted on the vehicle and used.

However, in the case of a 3D object recognition algorithm using such a lidar sensor, there is a problem that recognition performance is somewhat inferior compared to a 2D object recognition algorithm using an image.

In order to solve this problem, a method of improving the performance of a 3D object recognition algorithm using deep learning has been proposed, but the recognition result is still satisfactory in that 3D labeling data used as learning data is very limited. There is a limit that cannot be obtained.

In this embodiment, matching data is generated by matching an image collected using a camera and point cloud information collected using a lidar sensor, and a three-dimensional object region is generated by using the generated matching data as an input of a learning model. The main purpose is to provide an object recognition result that is more accurate and improved than the prior art in relation to the lidar sensor by detecting it.

The present embodiment includes: a data acquisition unit for acquiring images and lidar data collected from a camera and a lidar sensor for an area around the vehicle; a color information calculating unit for detecting an object region in which an object in the image exists by applying the image to a first learning model, and calculating RGB color information for the object region; a data matching unit that calculates point cloud information corresponding to the object region among the lidar data, and generates matching data by matching the point cloud information and the RGB color information; and a detector configured to detect a three-dimensional object region corresponding to the object region by applying the matching data to a second learning model.

In addition, according to another aspect of the present embodiment, the process of acquiring images and lidar data collected for the surrounding area of the vehicle from the camera and lidar sensor; detecting an object region in which an object in the image exists by applying the image to a first learning model, and calculating RGB color information for the object region; calculating point cloud information corresponding to the object region among the lidar data, and generating matching data by matching the point cloud information and the RGB color information; and detecting a 3D object area corresponding to the object area by applying the matching data to a second learning model.

As described above, according to this embodiment, the matching data is generated by matching the image collected using the camera and the point cloud information collected using the lidar sensor, and the generated matching data is used as an input of the learning model. By detecting a three-dimensional object region, there is an effect of providing more accurate and improved object recognition results compared to the prior art in relation to the lidar sensor.

In addition, the performance of a 3D object recognition algorithm based on deep learning can be improved by generating 3D labeling data based on the detected 3D object area and providing the generated 3D labeling data as learning data for 3D object detection. It works.

1 is an exemplary diagram for explaining a related configuration for a 3D object detection method according to the present embodiment.
2 is an exemplary diagram illustrating the structure of a deep learning network for a 3D object detection method according to the present embodiment.
3 is a block diagram schematically illustrating a 3D object detecting apparatus according to the present embodiment.
4 is a flowchart illustrating a 3D object detection method according to the present embodiment.
5 is an exemplary diagram for explaining matching data according to the present embodiment.
6 is an exemplary view for explaining a method of calculating loss information for re-evaluation of a 3D object region according to the present embodiment.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated in different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

Recently, deep learning techniques have been applied to many 3D object detection algorithms, showing improved performance in object detection. However, there is a limit that 3D labeling data is very limited compared to 2D image dataset. Although the performance of the network is evaluated using some limited 3D labeling data, there is a problem in that it operates only in a specific scene or only in a specific dataset.

To solve this problem, in the case of this embodiment, a 3D object detection network that is much more powerful than the existing recognition algorithm based on deep learning is proposed. In more detail, matching data is generated by matching an image collected using a camera and point cloud information collected using a lidar sensor, and a three-dimensional object area is detected by using the generated matching data as an input of a learning model. By doing so, we propose a new network model that can provide more accurate and improved object recognition results compared to the prior art in relation to the lidar sensor. In addition, a method for generating 3D labeling data using the acquired data and a method for using the same are proposed.

On the other hand, this embodiment describes a 3D object detection method in an autonomous or semi-autonomous vehicle equipped with a camera and a lidar sensor as a sensor for object detection, but this is for convenience of description, and the camera and It can be applied to various transportation means equipped with Ida sensor.

1 is an exemplary diagram for explaining a related configuration for a 3D object detection method according to this embodiment, and FIG. 2 is an exemplary diagram illustrating a structure of a deep learning network for a 3D object detection method according to the present embodiment.

Hereinafter, the 3D object detecting apparatus 130 and related configurations according to the present embodiment will be described with reference to FIGS. 1 and 2 together.

The 3D object detection apparatus 110 may be implemented in a form directly installed on the vehicle 100 as shown in FIG. 1 .

The 3D object detection apparatus 110 receives the lidar data collected by using the lidar sensor 120 and the image collected by using the camera 130 , and performs 3D object detection based thereon.

The 3D object detection apparatus 130 generates 3D labeling data based on the detected data, and provides the generated 3D labeling data as learning data for 3D object detection.

In this embodiment, the 3D object detection device 130 generates matching data by matching the image collected using the camera 130 and the point cloud information collected using the lidar sensor 120 , and the generated It is implemented to detect a three-dimensional object region by using the matching data as an input of the learning model.

To this end, the 3D object detection apparatus 130 first performs 2D object detection by applying a deep learning technique based on an image collected using the camera 130 .

Based on the 2D object detection result, the 3D object detection apparatus 130 calculates point cloud information corresponding to the object area, and performs 3D object detection based thereon. Here, since the 3D object detection apparatus 130 already has a hint about what kind of object the object is by using the previous 2D object detection result, it can perform more detailed segmentation using the information. .

In this embodiment, the 3D object detection apparatus 130 does not simply use the point cloud information, but provides a 3D object detection algorithm that is much more powerful than the existing recognition algorithm by using the relationship between the image and the point cloud information. That is, the 3D object detection device 130 generates matching data by matching the RGB color information of the point cloud and the image, and detects the three-dimensional object region by using this as an input of the learning model to detect a problem in the existing algorithm. It allows the probability to be reduced, which leads to optimal performance.

For example, referring to FIG. 2 , the structure of the deep learning network for the 3D object detection method according to the present embodiment may be largely divided into two stages.

First, the proposal ensemble network corresponds to the first stage of generating matching data by matching the image collected using the camera 130 and the point cloud information collected using the lidar sensor 120 .

In the first stage, 2D object detection is performed on an image using various pre-trained networks, and RGB color information corresponding to the detected object region is calculated based on this.

In the first stage, point cloud information corresponding to the previously detected object area among the lidar data is calculated, and matched data is generated by matching the calculated point cloud information and RGB color information.

The segmentation section corresponds to the second stage in which 3D object detection is performed by inputting the matching data generated in the first stage as an input.

In the second stage, segmentation is performed again on the point cloud information by additionally using the RGB color information obtained from the image, and based on this, a three-dimensional object region corresponding to the object region detected in the first stage is generated. detect

In the second stage, re-evaluation is performed on the detected 3D object area in consideration of the relationship with the 2D object area, and an error range is minimized based on the calculated loss information.

In the second stage, 3D labeling data is generated based on the detected 3D object area and provided as learning data for 3D object detection.

The lidar sensor 120 is mounted on one side of the vehicle, and emits a laser toward the periphery of the vehicle, for example, the front.

The lidar sensor 120 emits light of a laser wavelength to the outside, measures the time it takes for the emitted light to be reflected by an external object and returns to generate information (LIDAR data) on the position of the external object. do.

The lidar sensor 120 provides distance information measured using a laser in the form of a set of points in a 3D space (point cloud). For example, each point in the point cloud may include distance information of how many m away from the target and the robot coordinate system in X, Y, and Z directions from the position of the sensor and the reflection coefficient value of the target.

The camera 130 is mounted on one side of the vehicle and provides an image of a surrounding area of the vehicle. In this embodiment, the camera 130 may preferably be a photographing device capable of acquiring an RGB image.

The camera 130 additionally provides information about the RGB coordinate system measured for the image along with the captured image.

Meanwhile, in the present embodiment, the positions of the lidar sensor 120 and the camera 130 mounted on the vehicle 100 are not limited as specific positions.

3 is a block diagram schematically illustrating a 3D object detecting apparatus according to the present embodiment.

As shown in FIG. 3 , the 3D object detection apparatus 110 according to the present embodiment includes a data acquisition unit 300 , a color information calculation unit 310 , a data matching unit 320 , and a detection unit 330 . . Here, the components included in the 3D object detecting apparatus 110 are not necessarily limited thereto.

Meanwhile, in FIG. 3 , the data acquisition unit 300 , the color information calculation unit 310 , and the data matching unit 320 correspond to the proposal ensemble network of FIG. 2 , and the detection unit 330 may correspond to the segmentation section. .

The data acquisition unit 300 performs a function of acquiring data for 3D object detection in connection with an external device.

The data acquisition unit 300 is linked with the lidar sensor 120 to acquire lidar data collected by the lidar sensor 120 for the surrounding area of the vehicle 100 . Such lidar data may be point cloud information in which distance information measured by the lidar sensor 120 using a laser is expressed in the form of a set of dots in 3D space.

The data acquisition unit 300 is linked with the camera 130 to acquire an image of the surrounding area of the vehicle 100 photographed by the camera 130 .

The data acquisition unit 300 may additionally receive, from the camera 130 , calibration information, for example, RGB coordinate system information, measured with respect to the image, along with the image of the surrounding area. The RGB coordinate system information refers to three-dimensional coordinate system information in which an RGB element corresponding to each pixel in an image is used as one axis.

The color information calculating unit 310 detects an area in which an object exists in the image of the camera 130 obtained by using the data obtaining unit 300 and calculates RGB color information for the detected object area.

In this embodiment, an image of the color information calculating unit 310 may be applied to the first learning model to detect a region in which an object exists in the image. Here, the first learning model is configured in a form including at least one model (ex: VENET, PENET) trained in advance according to the propensity of each object, and based on this, it can be implemented to have strong image-based recognition performance. have. For example, the first learning model may have at least one or more network structures, and each network may be pre-trained to detect a two-dimensional object region in an image using a two-dimensional recognition algorithm based on a 2D image.

In more detail, the first learning model according to this embodiment is implemented based on the Tensorflow Object Detection API, and in order to increase performance, it can be implemented by improving it with PANet based on the Faster rcnn with inception resnet v2 model. PANet consists of a structure in which Augmented bottom-up structure, Adaptive Feature Pooling, and Fully Connected Fusion layer are added with FPN as a backbone to the existing Mask RCNN model.

The color information calculator 310 may detect an object area defined as a 2D bounding box on a 2D image by applying the image to the first learning model and learning it.

The color information calculating unit 310 collects RGB color information corresponding to the object region by using the RGB coordinate system information obtained using the data obtaining unit 300 . For example, the color information calculating unit 310 calculates RGB coordinate system information corresponding to an object region in the image based on the RGB coordinate system information collected for the image, and decomposes the color value of the calculated RGB coordinate system information into RGB to obtain an RGB color. information can be collected.

The data matching unit 320 calculates point cloud information corresponding to the detected object area using the color information calculation unit 310 among the lidar data, and matches the calculated point cloud information with the previously collected RGB color information. Generate matching data (=XYZRGB channel).

In this embodiment, the data matching unit 320 projects the calculated point cloud information on two-dimensional image coordinates, and based on the corresponding image coordinates and RGB coordinate system information collected from the camera, point cloud information and RGB color information can be matched. For example, the data matching unit 320 generates a UV map by projecting the calculated point cloud information on two-dimensional image coordinates, and maps RGB color information on the coordinates corresponding to the UV map based on the RGB coordinate system information. to generate matching data.

The data matching unit 320 may perform crop processing on the lidar data before calculating the point cloud information corresponding to the object region detected by using the color information calculating unit 310 among the lidar data. have. For example, the data matching unit 320 may check field of view (FOV) angle information of the camera 130 and perform crop processing on the lidar data based on the field of view angle information.

For example, referring to FIG. 5 , the form of matching data calculated by the data matching unit 320 may be checked. Looking at the matching data, it can be seen that the RGB color information is matched and displayed on the point cloud information, and the size is also cropped according to the field of view (FOV) angle information of the camera 130 .

The detection unit 330 detects a 3D object area corresponding to the previously detected object area on the image based on the matching data.

In the present embodiment, the detector 330 may detect a region in which an object exists in the matching data by applying the matching data to the second learning model. Here, the second learning model may have at least one network structure and may be pre-trained to detect a 3D object region in the matching data using a 3D recognition algorithm based on matching data corresponding to a 2D image.

In this embodiment, the second learning model is not limited as a specific learning model. For example, the second learning model may be implemented based on various conventional learning models for detecting a 3D object region based on lidar data.

The detection unit 330 may detect an object region defined as a 3D bounding box in a 3D space by applying the matching data to the second learning model and learning it.

On the other hand, in the case of the prior art, it was frequent that the 3D bounding box and the 2D bounding box have different positional information on an image plane despite being the same object. Due to this, the detector 330 according to the present embodiment may re-evaluate the detected 3D object region.

That is, the detection unit 330 compares the 3D bounding box with the 2D bounding box calculated using the color information calculating unit 310, and loses the error values of the size and position between the two bounding boxes calculated according to the comparison result. ) as information.

For example, referring to FIG. 6 , it can be confirmed that the detector 330 projects the 2D bounding box and the 3D bounding box on the image plane, and compares the size and position of each bounding box based on the corner points of each bounding box. have.

In general, the vehicle 100 is in close contact with the ground under the influence of gravity. For this reason, in the case of an object detected through the 3D object detection device 110 in the vehicle 100, it is also generally detected in a form in close contact with the ground, but in some cases, it may be detected in a form separated from the ground due to a recognition error. . Due to this point, the detection unit 330 may calculate a distance difference from the ground to the 3D bounding box based on a ground plane extracted from the point cloud information, and additionally utilize this as loss information.

The detection unit 330 generates 3D labeling data based on the detected 3D object region, and provides the generated 3D labeling data as learning data for 3D object detection.

In the present embodiment, the detection unit 330 may be provided with a labeling tool for directly editing and visualizing the 3D object region, and may generate 3D labeling data by using the labeling tool.

The detection unit 330 may use the generated 3D labeling data as training data using the second learning model provided in the detection unit 330 . In the present embodiment, as the second learning model is continuously learned using the 3D labeling data generated by the detection unit 330 as learning data, there is an effect of having higher recognition performance in relation to 3D object detection.

4 is a flowchart illustrating a 3D object detection method according to the present embodiment.

The 3D object detection apparatus 110 obtains images and lidar data collected for the surrounding area of the vehicle 100 from the camera 130 and the lidar sensor 120 ( S402 ).

The 3D object detection apparatus 110 applies the image obtained in step S402 to the first learning model to detect a region in which an object exists in the image, and calculates RGB color information for the detected object region (S404). In step S402, the first learning model consists of at least one network structure, and each network is pre-trained to detect a two-dimensional object region in an image using a two-dimensional recognition algorithm based on a 2D image.

The 3D object detection apparatus 110 may detect an object area defined as a 2D bounding box on a 2D image by applying the image to the first learning model and learning it.

The 3D object detection apparatus 110 collects RGB color information corresponding to the object region by using the RGB coordinate system information additionally acquired from the camera 130 .

The 3D object detection apparatus 110 calculates point cloud information corresponding to the object area of step S404 among the lidar data, and generates matching data by matching the calculated point cloud information with the RGB color information of step S404 (S406) . In step S406, the 3D object detection apparatus 110 projects the calculated point cloud information onto the two-dimensional image coordinates to generate a UV map, and based on the RGB coordinate system information, the RGB color information is projected onto the coordinates corresponding to the UV map. to create matching data.

The 3D object detection apparatus 110 detects a 3D object area corresponding to the object area of step S404 by applying the matching data of step S408 to the second learning model (S408). In step S408, the second learning model has at least one network structure and is pre-trained to detect a 3D object region in the registration data using a 3D recognition algorithm based on matching data corresponding to the 2D image.

The 3D object detection apparatus 110 detects an object area defined as a 3D bounding box in a 3D space by applying the matching data to the second learning model and learning it.

The 3D object detection apparatus 110 compares the 3D bounding box with the 2D bounding box calculated in step S404, and uses the error value of the size and position between the two bounding boxes calculated according to the comparison result as loss information.

The 3D object detection apparatus 110 calculates a distance difference from the ground to the 3D bounding box based on the reference plane extracted from the point cloud information, and additionally uses this as loss information.

Here, since steps S402 to S408 correspond to the operation of each component of the 3D object detecting apparatus 110 described above, further detailed description will be omitted.

Although it is described that each process is sequentially executed in FIG. 4 , it is not necessarily limited thereto. In other words, since it may be applicable to changing and executing the process described in FIG. 4 or executing one or more processes in parallel, FIG. 4 is not limited to a time series sequence.

As described above, the 3D object detection method described in FIG. 4 is implemented as a program and is a readable recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) can be recorded.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are for explanation rather than limiting the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

110: 3D object detection device 120: lidar sensor
130: camera 300: data acquisition unit
310: color information calculation unit 320: data matching unit
330: detection unit

Claims (12)

a data acquisition unit configured to acquire images and lidar data collected from a camera and a lidar sensor for an area around the vehicle;
a color information calculating unit for detecting an object region in which an object in the image exists by applying the image to a first learning model, and calculating RGB color information for the object region;
a data matching unit that calculates point cloud information corresponding to the object region among the lidar data, and generates matching data by matching the point cloud information and the RGB color information; and
A detection unit that detects a three-dimensional object region corresponding to the object region by applying the matching data to a second learning model
3D object detection device comprising a. The method of claim 1,
The first learning model consists of at least one network structure, and is pre-trained to detect a two-dimensional object region in the image using a two-dimensional recognition algorithm based on a 2D image,
The second learning model has at least one network structure and is pre-trained to detect the 3D object region in the matching data using a 3D recognition algorithm based on matching data corresponding to the 2D image. 3D object detection device. The method of claim 1,
The data acquisition unit additionally collects RGB coordinate system information corresponding to the image from the camera,
The color information calculator collects the RGB color information of the object region based on the RGB coordinate system information. The method of claim 1,
The data matching unit,
3D object detection apparatus, characterized in that it checks field of view (FOV) angle information of the camera, and performs crop processing on the lidar data based on the field of view angle information. The method of claim 1,
The data matching unit,
Projecting the point cloud information on two-dimensional image coordinates, and matching the point cloud information and the RGB color information based on the image coordinates and the RGB coordinate system information collected from the camera . 6. The method of claim 5,
The data matching unit,
generating a UV map by projecting the point cloud information onto the image coordinates, and mapping the RGB color information to coordinates corresponding to the UV map based on the RGB coordinate system information to generate the matching data 3D object detection device. The method of claim 1,
The 3D object detection apparatus of claim 1, wherein the color information calculating unit and the detecting unit detect an object area defined by a 2D bounding box on a 2D image and a 3D bounding box on a 3D space, respectively. 8. The method of claim 7,
The detection unit,
3D object detection apparatus, characterized in that the size and position of the 2D bounding box and the 3D bounding box are compared, and an error value of the size and position calculated according to the comparison result is used as loss information. 9. The method of claim 8,
The detection unit,
Projecting the 2D bounding box and the 3D bounding box on an image plane, and comparing the size and position of each bounding box based on the corner points of each bounding box. 9. The method of claim 8,
The detection unit,
3D object detection apparatus, characterized in that the distance difference from the ground to the 3D bounding box is calculated based on a ground plane extracted from the point cloud information, and the distance difference is additionally utilized as loss information. The method of claim 1,
The detection unit,
3D object detection apparatus, characterized in that generating 3D labeling data based on the 3D object region, and providing the 3D labeling data as learning data for 3D object detection. a process of acquiring images and lidar data collected for a surrounding area of the vehicle from a camera and a lidar sensor;
detecting an object region in which an object in the image exists by applying the image to a first learning model, and calculating RGB color information for the object region;
calculating point cloud information corresponding to the object region among the lidar data, and generating matching data by matching the point cloud information and the RGB color information; and
A process of detecting a three-dimensional object region corresponding to the object region by applying the matching data to a second learning model
3D object detection method comprising a.

Images