KR101907883B1 - Object detection and classification method - Google Patents

Abstract

The present invention relates to a method for extracting objects in an image and classifying objects. An apparatus for detecting an object comprises: a two-dimensional image sensor capturing a two-dimensional image; a two-dimensional image processing unit extracting a specific object from the photographed two-dimensional image and classifying a class for the specific object; a three-dimensional image sensor capturing a three-dimensional image; a three-dimensional image processing unit extracting the specific object from the captured three-dimensional image and classifying the class for the specific object; and a convergence processing unit calculating a final classification result for the specific object using the class classification result of the specific object extracted by the two-dimensional image processing unit and the class classification result of the specific object extracted by the three-dimensional image processing unit. Therefore, the method improves accuracy of object detection of the apparatus.

Description

{OBJECT DETECTION AND CLASSIFICATION METHOD}

The present invention relates to a method for extracting and classifying objects in an image.

Advanced driver assistant system (ADAS) provides assistance for driver safety in unpredictable driving situations. ADAS can be divided into two parts, the cognitive system and the alert notification interface. Examples of the first part include collision prediction and operator's carelessness detection, and the second part includes providing information to the driver about the event. However, both parts have commonality in requiring reliable object and event detection such as continuous area detection, mapping, and moving object tracking and detection.

Object detection and recognition methods that perform identification and region detection for effective objects of interest are critically important in various fields. Object detection is performed on real - time maps while driving, and object classification is performed using a classification model learned in an off - line database. The object detection and classification system is divided into a candidate detector that detects an object candidate and a classification model that classifies the detected object candidate region. The region of interest is usually represented as a feature vector, and it is conventional to classify it through a machine learning model such as support vector machine (SVM) or Adaboost.

In intelligent vehicle systems, one way to improve object detection and classification performance is to combine measurements from several sensors. At this time, managing incomplete measurements of different sensors is very important in constructing such a system. There are two main methods for fusion of different sensors, which include feature fusion and crystal fusion. The feature fusion method is a method of selectively fusing non-processed data or data-specific characteristics. Although many feature fusion methods have been proposed, if a problem occurs in one of the sensors included in the multi-sensor modality, it may have a negative effect on the whole system. On the other hand, the decision fusion method performs object detection and classification processes independently for each sensor, and fuses results from each sensor to derive the final result.

In the present invention, a method of detecting an object candidate region for each sensor is proposed. For effective object candidate detection, we aim to find a small number of meaningful candidate object regions. It is possible to effectively detect an object candidate region by performing image segmentation using color smoothing and semi-segmented image grouping on the image data measured by the CCD sensor. For the 3D point cloud data measured by the LIDAR sensor, super-voxel segmentation and region growth method are applied to detect object candidate regions. We also propose an object classification method that fuses a multi-layer laser scanner (3D LIDAR) with a CCD sensor. To do this, object candidate regions detected by each sensor are classified using convolutional neural network (CNN) and final fusion classification is performed through another CNN.

The object of the present invention is to improve the accuracy of object detection of an object detection apparatus.

It is another object of the present invention to improve the accuracy of object detection by fusing sensor information of a plurality of sensors installed in the object detection apparatus.

The present invention relates to a two-dimensional image sensor for photographing a two-dimensional image, a two-dimensional image processing unit for extracting a specific object from the photographed two-dimensional image, classifying a class of the specific object, A three-dimensional image processing unit for extracting the specific object from the photographed three-dimensional image and classifying a class for the specific object, a classification result of the specific object extracted from the two-dimensional image processing unit, And a fusion processing unit for calculating a final class classification result for the specific object by using the class classification result of the extracted specific object.

In one embodiment, the two-dimensional image processing unit performs color smoothing processing on the two-dimensional image, and detects an object candidate region from the two-dimensional image subjected to the color smoothing processing.

In one embodiment, the two-dimensional image processing unit divides the color-smoothed two-dimensional image into a plurality of regions, and based on the non-skewness of colors and textures, converts at least a part of the plurality of regions into object candidates .

In one embodiment, the 3D image is composed of point cloud data, and the 3D image processor converts the point cloud data constituting the 3D image into a voxel space, and converts the converted voxel space into a unit size And super voxels are grouped based on height difference of point cloud data included in each space to detect an object candidate region.

In one embodiment, the voxel space has a predetermined size to minimize a noise effect.

In one embodiment, the fusion processing unit calculates a correlation between a classification result of a specific object extracted from the two-dimensional image processing unit and a classification result of a specific object extracted from the three-dimensional image processing unit, The final class classification result for the specific object is calculated.

In one embodiment, when it is determined that the class classification result of the specific object extracted by the two-dimensional image processing unit and the classification result of the specific object extracted by the three-dimensional image processing unit are related to each other, A classification result of a specific object extracted by the 2D image processing unit and a classification result of a specific object extracted by the 3D image processing unit are combined to calculate a final class classification result, And the class classification result of the specific object extracted by the 3D image processing unit is not related to each other, the final class classification result is not calculated.

The present invention detects and classifies specific objects in two-dimensional images and three-dimensional images, and determines whether two objects are fused or not according to whether the detected and classified result information is related to each other, Can be improved.

1 is a conceptual diagram showing a configuration of an object detecting apparatus.
2 is a conceptual diagram showing a state in which an object detection apparatus 1000 is installed in a vehicle.
3 is a flowchart illustrating a method of detecting an object according to an embodiment of the present invention.
4 is a flowchart illustrating a method of extracting a specific object from a three-dimensional image according to an embodiment of the present invention.
5 is a flowchart illustrating a method of combining extraction results extracted through a plurality of sensors by the object detection apparatus according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. However, the present invention is not limited to the following embodiments, but merely serves as a means for efficiently explaining to a person having ordinary skill in the art to which the present invention belongs.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. In the drawings of the present invention, the same constituent elements are denoted by the same reference numerals throughout the entire specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as "including " an element, it does not exclude other elements unless specifically stated otherwise.

Herein, the term " part " or " module " means a unit realized by hardware or software, a unit realized by using both, and a unit realized by using two or more hardware Or two or more units may be realized by one hardware.

The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification.

Autonomous vehicles require the detection and classification of objects in the vicinity of the vehicle for autonomous driving. The present invention proposes a method for performing detection and classification of objects existing in the vicinity of a vehicle by using a plurality of measurement results measured in a plurality of sensors in an autonomous vehicle.

The object detection apparatus 1000 according to the present invention can perform object detection and classification using a plurality of measurement results measured in a plurality of sensors.

The object detecting apparatus 1000 may be an internal component of the vehicle, and may be formed as an external component of the vehicle so as to be able to communicate with other components of the vehicle in a wired or wireless manner.

Hereinafter, the configuration of the object detecting apparatus 1000 will be described with reference to FIGS. 1 and 2. FIG.

1, the object detecting apparatus 1000 includes a two-dimensional image sensor 110, a two-dimensional image processing unit 120, a three-dimensional image sensor 210, a three-dimensional image processing unit 220, a fusion processing unit 310, . ≪ / RTI >

The two-dimensional image sensor 110 is an image sensor that captures a two-dimensional image. An example of such an image sensor is a charging coupled device (CCD) sensor. The two-dimensional image sensor 110 may be included in the binocular camera. Accordingly, the two-dimensional image sensor 110 can generate a two-dimensional image obtained by photographing the surroundings of the vehicle at 360 degrees.

The two-dimensional image sensor 110 may be installed on the outer circumferential surface of the vehicle so as to photograph the surrounding environment of the vehicle. In particular, referring to FIG. 2, the two-dimensional image sensor 110 may be installed on the ceiling of the vehicle.

The two-dimensional image processing unit 120 may detect a specific object from the two-dimensional image captured through the two-dimensional image sensor 110. The specific object may be an object located in the vicinity of the vehicle, for example, a pedestrian, a nearby vehicle, a bicycle, a streetlight, or the like.

The three-dimensional image sensor 210 is an image sensor for capturing a three-dimensional image. As an example of the three-dimensional image sensor 210, there is a three-dimensional leisure scanner sensor. The three-dimensional image sensor 210 can generate a three-dimensional image of the surroundings of the vehicle 360 degrees. 2, the three-dimensional image sensor 210 may be installed on the ceiling of the vehicle.

The three-dimensional image processing unit 220 may detect a specific object from the three-dimensional image captured through the three-dimensional image sensor 210.

The fusion processing unit 310 may combine a specific object detected by the two-dimensional image processing unit 120 and a specific object detected by the three-dimensional image processing unit 220 to detect and classify a specific object located in the vicinity of the vehicle have.

The object classification apparatus 1000 for detecting and classifying a specific object existing around the vehicle has been described above.

Hereinafter, a method for detecting an object from a two-dimensional image by the object detecting apparatus according to the present invention will be described. 3 is a flowchart illustrating a method of detecting an object according to an embodiment of the present invention.

First, the two-dimensional image processing unit 120 can perform color smoothing processing on the two-dimensional image data (S310).

The two-dimensional image processing unit 120 may perform color smoothing processing on a two-dimensional image in order to improve object detection efficiency when detecting an object from a two-dimensional image. The color smoothing is an algorithm for uniformly processing hues of a two-dimensional image. That is, the two-dimensional image processing unit 120 may monotonously change the color of the two-dimensional image to more efficiently detect a specific object included in the two-dimensional image.

), L1 영상 변환이 수행된 2차원 영상을 변환 영상(

)으로 명명한다. The color smoothing is based on the L1 image color conversion technique. The two-dimensional image processing unit 120 may generate a transformed image by performing color smoothing using a two-dimensional image using an L1 image color conversion technique. Hereinafter, a two-dimensional image before the L1 image color conversion is performed is referred to as an original image (

), A two-dimensional image on which L1 image conversion is performed is converted into a transformed image

).

The two-dimensional image processing unit 120 may convert the original image into a transformed image using the energy function defined by Equation (1).

[Equation 1]

: 에너지 함수,

: 원본 영상의 픽셀과 변환 영상의 픽셀 간의 내부적 유사도,

: 이웃하는 두 픽셀 사이의 유사도)(

: Energy function,

: The internal similarity between the pixels of the original image and the pixels of the transformed image,

: Similarity between two neighboring pixels)

)를 계산할 수 있다. 상기

는 하기의 수학식 2로 정의될 수 있다. More specifically, the two-dimensional image processing unit 120 converts the internal similarity between pixels of the original image and pixels of the converted image (

) Can be calculated. remind

Can be defined by the following equation (2).

&Quot; (2) "

: 원본 영상과 변환 영상 사이의 픽셀 간의 내부적 유사도,

: 변환 영상의 모든 픽셀 값을 이어 붙인 벡터,

: 원본 영상의 모든 픽셀 값을 이어 붙인 벡터)(

: Internal similarity between pixels between original image and converted image,

: A vector to which all pixel values of a transformed image are connected,

: Vector in which all pixel values of the original image are connected)

는 원본 영상의 픽셀과, 변환 영상의 픽셀 간의 차이 값이 클수록 더 큰 값을 가질 수 있다. 따라서,

의 값이 크다는 것은, 원본 영상의 픽셀과 변환 영상의 픽셀 사이의 유사도가 낮다는 의미이고,

값이 작다는 것은, 원본 영상의 픽셀과 변환 영상의 픽셀 사이의 유사도가 높다는 의미이다.In other words,

The greater the difference between the pixel of the original image and the pixel of the transformed image, the larger the value can be. therefore,

Means that the degree of similarity between the pixels of the original image and the pixels of the transformed image is low,

The smaller value means that the similarity between the pixels of the original image and the pixels of the converted image is high.

에 근거하여, 색상 평활화를 수행함으로써, 2차원 영상에 포함된 모든 픽셀이 동일한 밝기 로 변환되는 것을 방지할 수 있다. The two-dimensional image processing unit 120 generates a two-

By performing color smoothing, it is possible to prevent all the pixels included in the two-dimensional image from being converted to the same brightness.

를 최소화하기 위하여,

가 최소가 되는 값을 찾을 수 있다. 2차원 영상 처리부(120)는

를 기 설정된 횟수만큼 반복적으로 변경하면서,

가 최소화되는

을 찾을 수 있다. 상기

에 대한 방향성은 split bregman 방법을 통하여, 결정될 수 있다. More specifically, the two-dimensional image processing unit 120

In order to minimize,

Can be found. The two-dimensional image processing unit 120

Is repeatedly changed a predetermined number of times,

Is minimized

Can be found. remind

The directionality to the bregman method can be determined through the split bregman method.

)를 계산할 수 있다. 이렇게 서로 이웃하는 픽셀 사이의 유사도를 지역적 완만함이라는 용어로도 명명할 수 있다. 이하, 설명의 편의를 위하여, 서로 이웃하는 2개의 픽셀을 픽셀 쌍이라고 명명하여 설명한다. In addition, the two-dimensional image processing unit 120 may calculate a similarity degree between neighboring pixels

) Can be calculated. The similarity between neighboring pixels can also be termed as the term "local smoothness". Hereinafter, for convenience of explanation, two neighboring pixels are referred to as pixel pairs.

는 하기의 수학식 3에 의하여 계산될 수 있다. remind

Can be calculated by the following equation (3).

&Quot; (3) "

: 이웃하는 픽셀 사이의 유사도,

: 변환 영상의 픽셀(

)에서의 RGB 벡터,

: 픽셀 쌍(

,

)의 가중치,

: 변환 영상의 픽셀 수,

: 변환 영상의 픽셀(

)에 이웃하는 mXm 픽셀 범위)(

: Similarity between neighboring pixels,

: The pixel of the transformed image (

) Of the RGB vector,

: Pixel pair (

,

),

: The number of pixels of the converted image,

: The pixel of the transformed image (

) In the mXm pixel range)

The weight may be assigned a high value to pixel pairs having similar color values in the original image. Therefore, the present invention can perform color smoothing by minimizing a difference in color values between neighboring pixels of a converted image by giving a high weight to pixel pairs having similar color values.

The weight can be calculated by the following equation (4).

&Quot; (4) "

: 픽셀 쌍(

,

)의 가중치,

: CIELab 색상 공간에서의 픽셀(

) ,

: 조명 변화와 관련된 상수,

: 분산,

: i번째 픽셀의 CIELab 색상 공간에서의 l, a, b 값)(

: Pixel pair (

,

),

: Pixels in the CIELab color space (

),

: Constant related to illumination change,

: Dispersion,

: l, a, b values in the CIELab color space of the ith pixel)

는 조명 변화와 관련된 상수로,

값을 조정하여, 조명 변화를 최소화 시킬 수 있다. 보다 구체적으로,

<1 일 때, 픽셀 쌍은 조명 변화에 대해 둔감해 질 수 있다. From here,

Is a constant related to illumination change,

By adjusting the value, the illumination change can be minimized. More specifically,

&Lt; 1, the pixel pair may become insensitive to the illumination change.

와

는 반복적인 실험을 통하여, 최적의 성능을 도출하기 위한 상수 값이 결정될 수 있다. 바람직하게는, 실험 결과에 따라

와

는 각각 0.3과 1.0으로 설정할 수 있다. Such

Wow

A constant value for deriving the optimum performance can be determined through an iterative experiment. Preferably, depending on the results of the experiment

Wow

Can be set to 0.3 and 1.0, respectively.

을 찾기 위하여, 하기의 수학식 5를 통하여, 수학식 3 및 수학식 4를 최적화할 수 있다. On the other hand, the two-dimensional image processing unit 120 converts the two-

(3) and (4) can be optimized through the following equation (5).

&Quot; (5) &quot;

: 상수,

: 최소 제곱 형태의 L1 에너지 항의 가중치를 제어하는 상수,

,

: split bregman 방법의 중간 변수) (z: transform vector,

: a constant,

: A constant controlling the weight of the L1 energy term of the least square form,

,

: intermediate variable of split bregman method)

은 서로 이웃하는 픽셀에 속하는 상태를 나타내는 행렬일 수 있다. 보다 구체적으로,

은 mXn 행렬이며, pi가 pj의 mXm 이웃 픽셀에 속하는 경우,

, 그렇지 않은 경우,

을 가질 수 있다. remind

May be a matrix indicating states belonging to neighboring pixels. More specifically,

Is an mXn matrix, and if pi belongs to an mXm neighboring pixel of pj,

, Otherwise,

Lt; / RTI &gt;

The two-dimensional image processing unit 120 can determine an object candidate region using the color and texture similarity (S320).

The two-dimensional image processing unit 120 may divide the converted image subjected to the color smoothing processing into a plurality of regions. Here, one divided area can be named as a partition.

,

) 사이의 유사도에 따라, 적어도 하나의 파티션을 하나의 그룹으로 그룹화할 수 있다. 상기 파티션 사이의 유사도는 색상 및 텍스쳐에 대한 유사도이다.The two-dimensional image processing unit 120 performs a two-

,

, It is possible to group at least one partition into one group. The similarity between the partitions is a similarity to color and texture.

,

) 사이의 비유사도를 계산할 수 있다. More specifically, the two-dimensional image processing unit 120 divides the adjacent partitions (i. E., &Lt; RTI ID = 0.0 &

,

) Can be calculated.

&Quot; (7) &quot;

: 서로 인접한 파티션 간의 비유사도,

: 서로 인접한 파티션(i, j) 간의 색상 비유사도,

: 서로 인접한 파티션(i,j) 간의 텍스쳐 비유사도,

: 색상, 텍스쳐 각각에 대한 가중치 상수)(

: A non-contour between adjacent contiguous partitions,

: Color contrast between adjacent (i, j) partitions,

: A texture analogy between adjacent (i, j) partitions,

: Weight, constant for each color, texture)

)를 계산할 수 있다. 보다 구체적으로, 2차원 영상 처리부(120)는 각 파티션의 색상 공간에 대한 각 색상 채널을 25칸의 히스토그램으로 변환할 수 있다. 그리고, 2차원 영상 처리부(120)는 H, S, V 색상 공간의 각 25칸 히스토그램을 이어 붙여, 총 75칸의 히스토그램(

)을 계산할 수 있다. The two-dimensional image processing unit 120 uses the histogram of the HSV (Hue Saturation Value)

) Can be calculated. More specifically, the two-dimensional image processing unit 120 may convert each color channel for each partition color space into 25 histograms. Then, the two-dimensional image processing unit 120 adds 25 histograms of H, S, and V color spaces to the histogram of 75 squares

) Can be calculated.

) 사이의 거리를 계산하여, 색상 비유사도(

)를 계산할 수 있다. The two-dimensional image processing unit 120 may generate a histogram corresponding to each of the adjacent partitions

) To calculate the distance between the colors

) Can be calculated.

)을 이용하여, 텍스쳐 비유사도(

)를 계산할 수 있다. 여기에서, SIFT의 히스토그램은

에서 8 방향에 대한 가우시안 미분을 RGB 각 채널에 적용하는 히스토그램이다. 또한, 본 발명은 텍스처에 많은 압축이 가해진 변환 영상 대신 원본 영상을 이용하여, 텍스쳐 비유사도를 계산함으로써, 텍스처 비유사도 계산의 정확도를 향상시킬 수 있다. In addition, the two-dimensional image processing unit 120 generates a histogram of Scale Invariant Feature Transform (SIFT)

), The texture non-iso (

) Can be calculated. Here, the histogram of SIFT is

Is a histogram that applies the Gaussian differential for 8 directions to each of the RGB channels. In addition, the present invention can improve the accuracy of the texture algebra calculation by calculating the texture algebra using the original image instead of the transformed image with a lot of texture compression.

More specifically, the two-dimensional image processing unit 120 can generate a histogram composed of ten squares for each direction. Accordingly, the two-dimensional image processing unit 120 can calculate the SIFT histogram by 240 squares.

The two-dimensional image processing unit 120 can calculate the texture similarity using Equation (8) below.

&Quot; (8) &quot;

: 텍스쳐 비유사도,

: i 번째 파티션의 SIFT 히스토그램,

: j번째 파티션의 SIFT 히스토그램)(

: Texture,

: SIFT histogram of the i-th partition,

: SIFT histogram of the jth partition)

The two-dimensional image processing unit 120 can optimize the non-skewness of Equation (7) using the following Equation (9).

&Quot; (9) &quot;

: 비유사도 함수를 통해 계산된 영상 분할 결과,

: 기존 데이터 셋에서 결과를 알고있는 분할 결과(ground truth segmentation),

: 선형 SVM을 통해 미리 정의된 규칙화 변수,

: slack 변수)(

: Image segmentation result calculated through non -

: A ground truth segmentation that knows the result in an existing dataset,

: Predefined regularization variables via linear SVM,

: slack variable)

)가 기 설정된 값 미만인 경우, 적어도 두 개의 파티션을 그룹화하여, 그룹을 생성할 있다. 예를 들어, 2차원 영상 처리부(120)는 서로 인접한 제1파티션(

)과 제2파티션(

)의 비유사도(

)가 기 설정된 값(

) 미만인 경우, 제1파티션(

)과 제2파티션(

)을 하나의 그룹으로 설정할 수 있다. 한편, 여기에서, 기 설정된 값(

)은 반복적인 실험을 통하여 결정되는 상수이다. The two-dimensional image processing unit 120 converts the two-

) Is less than a predetermined value, at least two partitions may be grouped to create a group. For example, the two-dimensional image processing unit 120 may include a first partition

) And the second partition (

) Of the universe (

) Has a predetermined value (

), The first partition (

) And the second partition (

) Can be set as one group. On the other hand, here,

) Is a constant determined through repeated experiments.

Then, the two-dimensional image processing unit 120 can determine a group including partitions having a high degree of similarity as an object candidate region.

The two-dimensional image processing unit 120 may extract a specific object from the object candidate region (S330).

The two-dimensional image processing unit 120 can extract a specific object from the determined object candidate region using the CNN model structure.

The two-dimensional image processing unit 120 may construct a convolution cube (ConvCube) using output of a plurality of convolution layers for the determined object candidate region in order to improve extraction accuracy of a specific object. Here, the convolution layer is a layer that performs a function of applying a convolution operation to input data, and a convolution cube is a three-dimensional image of the output of two or more layers.

On the other hand, since the outputs of a plurality of convolutional layers used in the convolution cube have different sizes, different sizes can be standardized by applying different sampling methods according to the sizes of specific objects. Accordingly, since the two-dimensional image processing unit 120 minimizes the feature loss by varying the sampling method according to the size of the specific object, the specific object can be detected even if the size of the specific object is small.

More specifically, a Max pooling sampling scheme may be applied to a convolution layer having an output greater than that of the convolution cube, and in the opposite case, a deconvolution sampling scheme may be applied.

The two-dimensional image processing unit 120 may perform normalization of values through local response normalization (LRN) on the convolutional layers having been sampled by size.

On the other hand, the two-dimensional image processing unit 120 does not apply the CNN model structure to the entire object candidate region, but constructs a convolution cube for one frame constituting the object candidate region, applies ROI-pooing, A convolution cube for the entire region can be constructed.

The two-dimensional image processing unit 120 may sequentially apply two convolutional layers and two fully-connected layers to the convolution cube constructed for the entire object candidate region to obtain the final output.

After sequentially passing through two convolutional layers and two fully-connected layers , the two-dimensional image processing unit 120 can extract a specific object and classify a specific object class through a softmax classification layer . Here, the class indicates the kind of object, and can be defined as, for example, a person wearing a car, a person, or a two-wheeled vehicle.

In the foregoing, a method of extracting a specific object from a two-dimensional image by the object detection apparatus according to the present invention has been described.

Hereinafter, a method for extracting a specific object from a three-dimensional image by the object detecting apparatus according to the present invention will be described. 4 is a flowchart illustrating a method of extracting a specific object from a three-dimensional image according to an embodiment of the present invention.

Referring to FIG. 4, the 3D image processing unit 220 may convert the 3D point cloud data forming the 3D image into a voxel (S410).

The three-dimensional image sensor 210 can photograph the surroundings of the vehicle in a three-dimensional image. The three-dimensional image can be composed of three-dimensional point cloud data.

라고 정의했을 때, 3차원 점 구름 데이터 i에 해당하는 복셀 공간은,

으로 정의된다. 3차원 점 구름 데이터 i에 해당하는 복셀 공간은, 3차원 점 구름 데이터 i와 그 주변의 3차원 점 구름 데이터를 포함할 수 있다. The three-dimensional image processing unit 220 may convert the point cloud data constituting the three-dimensional image into a three-dimensional active voxel space. The three-dimensional activated voxel space means a space including three-dimensional point cloud data. For example, when the coordinates of the three-dimensional point cloud data i are

, The voxel space corresponding to the three-dimensional point cloud data &quot; i &quot;

. The voxel space corresponding to the three-dimensional point cloud data i may include three-dimensional point cloud data i and three-dimensional point cloud data therearound.

The voxel space may have a predetermined size. When the size of the voxel space is small, it is difficult to reduce the noise data, and when the size of the voxel space is large, the meaningful object shape is compressed. Therefore, the voxel space must have such a size that the shape of the meaningful object is not compressed while minimizing the noise data, and the size of the voxel space can be determined experimentally. Preferably, the size of the voxel space may be determined as 0.1X0.1X0.1. Accordingly, the three-dimensional image processing unit 220 can group the three-dimensional point cloud data into a plurality of voxel spaces having a specific size, thereby minimizing the influence of the noise data.

The three-dimensional image processing unit 220 can calculate the activation probability of each voxel space according to Equation (10) below.

&Quot; (10) &quot;

: 복셀(

)의 활성화 확률,

:복셀 개수,

: 복셀(

)을 이루는 3차원 점 구름 데이터의 수,

: 복셀(

)의 j번째 3차원 점 구름 데이터)(

: Voxel (

) Activation probability,

: Number of voxels,

: Voxel (

The number of three-dimensional point cloud data,

: Voxel (

) Th point 3D cloud data)

는 j번째 레이저가 장애물에 반사된 경우, 1, 반사되지 않은 경우, 0을 가질 수 있다. remind

May be 1 if the jth laser is reflected at the obstacle, and 0 if it is not.

The three-dimensional image processing unit 220 may convert the transformed plurality of voxel spaces into a plurality of super voxels, and determine an object candidate region using the height difference between the super voxels in operation S420.

The three-dimensional image processing unit 220 may divide the voxel space into super voxels having a unit size in order to accurately extract the boundary lines of the objects included in the three-dimensional image. More specifically, the three-dimensional image processing unit 220 may divide the voxel space into a plurality of super voxels through a voxel cloud connectivity segmentation (VCCS). The voxel cloud connectivity partitioning performs spatial partitioning using a seeding methodology on a three-dimensional data space.

The three-dimensional image processing unit 220 projects super voxels on a three-dimensional space into a 0.1 m X 0.1 m grid on a two-dimensional [X, Z] space so that the super voxels can include object- can do.

The three-dimensional image processing unit 220 can determine whether to connect the grid using the height difference of the super voxels in each grid. That is, when the height difference between the super voxels in the lattice is 0.1 m or less, the three-dimensional image processing unit 220 can group the super voxels in each lattice. Conversely, if the height difference between the super voxels in the lattice exceeds 0.1 m, the three-dimensional image processing unit 220 may not group the super voxels in each lattice.

The 3D image processing unit 220 may set an area corresponding to the grouped super voxel as an object candidate area.

The 3D image processing unit 220 may extract a specific object from the determined object candidate region (S430).

The three-dimensional image processing unit 220 converts the three-dimensional point cloud data into two-dimensional image depth information data, extracts a specific object and classifies the specific object class in the same manner as the two-dimensional image processing unit 120 . That is, the 3D image processing unit 220 can extract a specific object using the CNN model structure. Therefore, a detailed description thereof will be replaced with the description of S330.

In the above, a method of extracting a specific object from a three-dimensional image has been described.

Hereinafter, a method of improving the extraction accuracy of a specific object using a result of extracting a specific object from a two-dimensional image and a result of extracting a specific object from the three-dimensional image will be described. FIG. 5 illustrates a method of combining extraction results extracted by a plurality of sensors by the object detection apparatus according to the present invention.

The fusion processing unit 310 of the object detection apparatus 1000 according to the present invention can receive extraction results obtained by extracting specific objects from the two-dimensional image processing unit 120 and the three-dimensional image processing unit 220, respectively. Then, the fusion processing unit 310 can detect a specific object by merging the two extraction results. Hereinafter, the control of the fusion processing unit 310 will be described in more detail.

Referring to FIG. 5, the fusion processing unit 310 may calculate the degree of similarity between different object information extracted from image data measured by different sensors (S510).

The fusion processing unit 310 can calculate the degree of similarity between extraction results indicating specific objects extracted from the two-dimensional image sensor 120 and the three-dimensional image sensor 210. This similarity calculation method can use a basic belief assingment (BBA) method.

More specifically, the fusion processing unit 310 can detect a plurality of boundary regions including the contours of a specific object from the object candidate regions of the two-dimensional image and the three-dimensional image. That is, the fusion processing unit 310 detects the first boundary region including the contour of the specific object from the object candidate region of the two-dimensional image, and detects the second boundary region including the contour of the specific object from the object candidate region of the three- Can be detected.

) 및 클래스 비유사도(

)에 근거하여, 경계 영역 간의 관계 정보를 획득할 수 있다. 여기에서, Yager의 결합 이론이 사용될 수 있다. The fusion processing unit 310 calculates the distance

) And class analogy (

, It is possible to obtain the relationship information between the boundary regions. Here, Yager's coupling theory can be used.

)를 계산하기 위하여, 제1경계 영역과 제2경계 영역 간의 관계 행렬을 계산할 수 있다. First, the fusion processing unit 310 calculates the distance

), A relation matrix between the first boundary region and the second boundary region may be calculated.

)과, 제2경계 영역에 포함된 슈퍼 복셀(

) 사이의 관계 성분 (

)으로 구성될 수 있다. The relational matrix may be represented by an n X m matrix, where n is the number of partitions included in the first boundary region and m is the number of super voxels included in the second boundary region. Also, the relational matrix may include a partition (e.g.,

), A super voxel included in the second border area (

) &Lt; / RTI &gt;

).

The fusion processing unit 310 can express the hypothetical group between the first boundary region and the second boundary region through the following Equation (11) using the relational matrix.

&Quot; (11) &quot;

는 관계 확률

이고,

은 관계 확률

이다. From here,

The probability of relationship

ego,

Probability of relationship

to be.

)의 기본적 신뢰 할당을 하기의 수학식 12에 의하여 계산할 수 있다. The fusion processing unit 310 calculates the distance between the first boundary region and the classification result

) Can be calculated by Equation (12) below.

&Quot; (12) &quot;

: 증거 감소 팩터(evidence discounting factor),

:

와

사이의 Mahalanobis 거리) (

: Evidence discounting factor,

:

Wow

Mahalanobis street between)

On the other hand, the fusion processing unit 310 may use the following equation (13) to return a large value when the distances are close to each other.

&Quot; (13) &quot;

(

: 거리의 가까움을 나타내는 상수)

(

: A constant indicating the proximity of the distance)

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

) 사이의 클래스 관계를 나타내는 클래스 비유사도(

)를 계산할 수 있다. In addition, the fusion processing unit 310 may include a partition (not shown)

And a super voxel included in the second border area

) Representing the class relationship between classes (

) Can be calculated.

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)이 동일한 클래스인지 다른 클래스인지 여부를 의미한다. The class relationship is a relation between a partition included in the first border area

And a super voxel included in the second border area

) Is the same class or another class.

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)이 동일한 클래스인 경우, 상기 제1경계 영역에 포함된 파티션(

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)은 동일한 객체를 가질 수 있기도 하고, 서로 다른 객체를 포함할 수도 있다. 반면에, 상기 제1경계 영역에 포함된 파티션(

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)이 서로 다른 클래스인 경우, 상기 제1경계 영역에 포함된 파티션(

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)은 서로 다른 객체를 가질 수 있다. 따라서, 본 발명에서는, 클래스 유사도가 아닌, 클래스 비유사도를 측정하여, 상기 제1경계 영역에 포함된 파티션(

)과, 상기 제2경계 영역에 포함된 슈퍼 복셀(

)이 서로 다른 객체를 가지는 경우를 계산한다. The partitions included in the first border area (

And a super voxel included in the second border area

) Is the same class, the partition included in the first border area (

And a super voxel included in the second border area

) May have the same object, or they may contain different objects. On the other hand, the partitions included in the first border area

And a super voxel included in the second border area

) Are different classes, the partitions included in the first border area

And a super voxel included in the second border area

) Can have different objects. Therefore, in the present invention, the class non-class diagram is measured instead of the class class similarity, and the partition (s) included in the first boundary area

And a super voxel included in the second border area

) Have different objects.

으로 정의할 수 있다. 융합 처리부(310)는 각 클래스에 대한 확률 비율을 pignistic trasformation을 사용하여, BBA 밀도 함수로 변환할 수 있다. 상기 BBA 밀도 함수는,

으로 정의될 수 있다. 여기에서,

는 센서 S로부터 제공되는 k번째 경계 박스의 클래스 밀도를 의미한다. In order to calculate the class analogy, the fusion processing unit 310 sets a hypothetical group of class relations

. The fusion processing unit 310 may convert the probability ratios for each class into a BBA density function using pignistic trasformation. The BBA density function,

. &Lt; / RTI &gt; From here,

Denotes the class density of the kth bounding box provided from the sensor S.

The fusion processing unit 310 can calculate the class analogy according to Equation (14) below.

&Quot; (14) &quot;

)와 클래스 비유사도(

)를 이용하여, 최종 관계 성분(

)을 계산할 수 있다. The fusion processing unit 310 calculates a distance

) And the class analogy (

), The final relational component (

) Can be calculated.

)은 하기의 수학식 15에 의하여 계산될 수 있다. Final relationship components (

) Can be calculated by the following equation (15).

&Quot; (15) &quot;

(D, C: Space in the boundary area of each sensor)

The fusion processing unit 310 can determine whether the two boundary regions are related by the final relation component (S520). That is, the fusion processing unit 310 determines that there is a relation when the final relation component has a predetermined value or more, and determines that there is no correlation when the final relation component has a predetermined value or more.

More specifically, if it is determined that the two boundary regions are related to each other by the final relation component, the fusion processing unit 310 may concatenate the convolution cube corresponding to the two boundary regions and the class classification result (S530). That is, the fusion processing unit 310 passes through two convolution layers and two fully-connected layers, and finally outputs 2048-dimensional vectors.

In addition, the fusion processing unit 310 may reconfigure the 2048-dimensional vector into a 2054 vector that is a concatenated 6-dimensional vector by connecting the vector with the 3-dimensional vector at each sensor. Then, the fusion processing unit 310 can determine the final object candidate region by passing the reconstructed vector through the fully-connected layer and the final previous SVM, and can calculate the final specific object extraction result and class classification result of the final specific object.

Alternatively, the fusion processing unit 310 may not use the final detection and classification result if the two boundary regions are not related to each other due to the final relation component. Therefore, the present invention improves the accuracy of object extraction by not including two pieces of irrelevant boundary region information in the calculation of the classification result of the final specific object and the final specific object.

As described above, according to the present invention, a specific object is detected and classified in a two-dimensional image and a three-dimensional image, and a determination is made as to whether or not two results are fused, The accuracy of extraction and classification can be improved.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Claims (7)

A two-dimensional image sensor for capturing a two-dimensional image;
A two-dimensional image processing unit for extracting a specific object from the photographed two-dimensional image and classifying the class for the specific object;
A three-dimensional image sensor for capturing a three-dimensional image;
A three-dimensional image processing unit for extracting the specific object from the photographed three-dimensional image and classifying the class for the specific object;
And a fusion processing unit for calculating a final class classification result for the specific object using the class classification result of the specific object extracted by the two-dimensional image processing unit and the classification result of the specific object extracted by the three-dimensional image processing unit In addition,
The fusion processing unit
Calculating a correlation between a class classification result of the specific object extracted by the two-dimensional image processing unit and a classification result of the specific object extracted from the three-dimensional image processing unit,
And calculates a final class classification result for the specific object based on the association. The method according to claim 1,
The two-dimensional image processing unit
The two-dimensional image is subjected to color smoothing processing,
And detects an object candidate region from the color-smoothed two-dimensional image. 3. The method of claim 2,
The two-dimensional image processing unit
Wherein the color-smoothed two-dimensional image is divided into a plurality of regions,
And extracts at least a part of the plurality of regions as an object candidate region based on the non-derivation of the color and the texture. The method according to claim 1,
The three-dimensional image is composed of point cloud data,
The three-dimensional image processing unit
Transforming the point cloud data constituting the three-dimensional image into a voxel space,
Dividing the converted voxel space into super voxels having a unit size,
And grouping the super voxels based on a height difference of the point cloud data included in each space to detect an object candidate region. 5. The method of claim 4,
The voxel space
And has a predetermined size to minimize a noise effect. delete The method according to claim 1,
The fusion processing unit
When the classification result of the specific object extracted by the two-dimensional image processing unit and the classification result of the specific object extracted by the three-dimensional image processing unit are related to each other, And a class classification result of the specific object extracted by the three-dimensional image processing unit,
Wherein when the class classification result of the specific object extracted by the two-dimensional image processing unit and the class classification result of the specific object extracted by the three-dimensional image processing unit are not associated with each other, the final class classification result is not calculated Object detection device.

Images