KR101478709B1 - Method and apparatus for extracting and generating feature point and feature descriptor rgb-d image - Google Patents

Abstract

The method includes generating a depth gradient based on a pixel-by-pixel position through a depth-image function relationship of each pixel of a three-dimensional image including depth information obtained through a camera, Calculating a set of adjacent vertexes based on a predetermined vertex of the depth image, calculating a preset vector using the calculated vertexes, and performing normalization from the computed cross-product of the vector dimensional vector into a two-dimensional image by performing rendering using the depth normalized vector normal information and a surface normal vector obtained by normalizing the surface normal vector, , Calculating a feature descriptor from the 2D image by applying a Scale Invariant Feature Transform (SIFT) algorithm, and A it characterized in that it comprises.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for extracting feature points of RGB-D images and a method of generating feature descriptors,

The present invention relates to generation of image feature points and feature descriptors based on a depth image obtained from an RGB-D image sensor.

In the fields of computer vision, robotics, and augmented reality, three - dimensional space and three - dimensional object detection and recognition technology are becoming more important. In particular, various researches are being conducted to enhance the detection and recognition of objects even in a light-change condition of a low-detailed texture object or various lighting conditions.

Since the real time acquisition of RGB image and depth image can be achieved through image sensor using Microsoft Kinect method, many changes have been made in object detection, tracking and recognition [1, 2] .

If only existing RGB images are used, various feature point detection and descriptor generation methods such as SIFT [3] and SURF [4] can be used.

However, in case of lack of texture or extreme illumination change, extraction of feature points and matching of descriptors in RGB images are impossible, which is a major factor for lowering object recognition rate [5] [6].

Accordingly, in order to solve such a problem, the present invention uses an image sensor of a Kinect type to simultaneously use a depth image-based feature descriptor of an RGB feature descriptor to simultaneously learn an object, We propose a method to increase the object recognition rate in various environmental changes.

According to an aspect of the present invention, there is provided a method for generating a depth gradient based on a pixel-by-pixel position, the method comprising: generating a depth gradient based on a pixel- Calculating a set of vertexes adjacent to a predetermined vertex of the depth image based on a gradient, calculating a predetermined vector using the calculated vertexes, and calculating a cross-product of the calculated vector ) Extraction of the normalized surface normal vector from the computed depth normal vector, and rendering using the extracted depth image vector information composed of the extracted surface normal vector to obtain a three-dimensional vector as a two-dimensional image And applying a Scale Invariant Feature Transform (SIFT) algorithm to the feature descriptor tor) of the image.

According to another aspect of the present invention, there is provided a depth-gradient-based gradient-based pixel-by-pixel-based depth-image function relationship between a three-dimensional image and a depth image function under the control of a controller for controlling the overall operation of the RGB- And acquiring a 3D image and an RGB image including depth information from the photographing unit through mode switching, respectively, and acquiring a predetermined vertex of the depth image based on the depth gradient generated from the gradient generating unit, a controller for calculating the adjacent vertexes based on the vertex of the depth image and calculating a predetermined vector using the calculated vertexes and extracting the feature points and feature descriptors of the depth image from the image generated through the conversion into the surface normal vector From the cross-product calculation of the vector calculated from the controller, a surface normal vector extracting unit for extracting an ed surface normal vector and a rendering unit for performing rendering using the depth image vector information configured by the surface normal vector under the control of the control unit .

INDUSTRIAL APPLICABILITY The present invention has the effect of strengthening the detection and recognition of objects even in the case of changes in the rotation and movement of a camera, a change in a subject with insufficient texture or various lighting conditions.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an overall flowchart of an RGB-D image feature point extraction and feature descriptor generation method according to an embodiment of the present invention; FIG.
FIG. 2 is a diagram illustrating a screen for extracting RGB-D image feature points and generating feature descriptors according to an embodiment of the present invention; FIG.
FIG. 3 is a diagram illustrating a result of a surface normal vector mesh rendering and a gray-scale transformation using a feature point extraction and feature descriptor generation method according to an exemplary embodiment of the present invention.
FIG. 4 is a test environment to which an RGB-D image feature point extraction and feature descriptor generation method according to an exemplary embodiment of the present invention is applied.
FIG. 5 is a block diagram of an apparatus for generating feature points and extracting feature points of RGB-D images according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 6 is a graph showing recognition rates per object according to a detection method in the method of extracting feature points and generating characteristic descriptors of RGB-D images according to an exemplary embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It will be appreciated that those skilled in the art will readily observe that certain changes in form and detail may be made therein without departing from the spirit and scope of the present invention as defined by the appended claims. To those of ordinary skill in the art.

The present invention relates to generation of image feature points and feature descriptors based on a depth image obtained from an RGB-D image sensor based on Microsoft's Kinect method, Dimensional object recognition function that is robust to environmental changes such as texture, camera rotation, and movement change by extracting a surface normal vector representing the geometric information of the 3D object from the acquired depth image and imaging the result. To provide a technology that can improve the performance of the system.

Hereinafter, an RGB-D image feature point extraction and feature descriptor generation method according to an embodiment of the present invention will be described in detail with reference to FIG. 1 and FIG.

First, FIG. 1 is an overall flowchart of an RGB-D image feature point extraction and feature descriptor generation method according to an embodiment of the present invention.

Referring to FIG. 1, in step 110, an image for each setting mode is acquired under the control of the control unit. The setting mode is classified into a three-dimensional image including depth information and an RGB image acquisition mode.

In step 112, it is checked whether the acquired image is a three-dimensional image. If the acquired image is a three-dimensional image, the process moves to step 116 and a depth gradient based on the pixel- .

More specifically, the depth image function D (x) for the pixel position x of the three-dimensional image is in the form of the following equation.

로 나타낼 수 있으며, 상기 그래디언트

의 최소 제곱법(least-square)으로부터 추정 가능하다.Thus, an arbitrary offset dx based on the pixel position x of the depth image is

, And the gradient

(Least-squares) of < / RTI >

In step 120, neighboring vertexes are calculated based on a predetermined vertex of the depth image based on the depth gradient generated in step 116. In step 122, a predetermined vector is calculated using the calculated vertexes. .

In operation 124, a normalized surface normal vector is extracted from the computed cross-product operation of the vector.

According to the present invention, a surface normal vector transformation method that constantly expresses the three-dimensional surface information of a target object as a three-dimensional vector is used even in a state change of a depth camera, and this defines an image feature expressing three- This is because the depth image needs to be transformed to maintain a certain shape in the rotation, translation, and affine changes of the camera.

,

,

는 각각 하기의 수학식과 같이 표현될 수 있다.That is, three adjacent vertices existing in the three-dimensional space

,

,

Can be expressed by the following equations respectively.

At this time Represents a vector from the principal point of the depth sensor camera through the pixel position x to the three-dimensional point X, it can be calculated through the internal parameter K of the depth sensor camera using the following equation have.

,

,

를 사용하여, 정점

로부터 시작하는 두 개의 벡터

-

및

-

로 표현할 수 있고, 이 두 벡터의 외적(cross-product) 연산으로부터 정규화(normalized)된 표면 정규 벡터의 추출이 가능하다.The three points of the three-dimensional space thus calculated

,

,

, The vertex

Two vectors starting from

-

And

-

, And it is possible to extract the normalized surface normal vector from the cross-product operation of these two vectors.

Referring to FIG. 2, FIG. 2 is a view illustrating a screen for extracting feature points of RGB-D images and generating feature descriptors according to an exemplary embodiment of the present invention. The upper part of FIG. 2 is an example And the lower end of FIG. 2 is an example of a screen showing a depth image-based surface normal vector view and a result.

을 기준으로 반시계 방향(CCW)으로 인접한 세 정점

,

,

를 연결하는 하나의 폴리곤 메쉬(polygon mesh)를 랜더링한다.Subsequently, in step 126,

(CCW) relative to the three vertices

,

,

A polygon mesh is connected.

,

,

,

의 평균값으로 지정한다.The direction of the normal vector of the polygon mesh generated in this process is the surface regular vector of each of the four vertices

,

,

,

As the average value of "

Through the operation of these 126 processes, x. y, and z directions into a two-dimensional image.

3 is a screen illustrative book to which an RGB-D image feature point extraction and feature descriptor generation method according to an embodiment of the present invention is applied, and FIG. 3 shows a result of a surface normal vector mesh rendering on the left side of FIG. Is an example of a screen in which a result of gray scale conversion is shown.

In operation 128, a Scale Invariant Feature Transform (SIFT) algorithm is applied. In operation 130, a feature descriptor is calculated from the 2D image.

If it is determined that the 3D image is not a 3D image, the RGB image is converted to a gray scale image in step 118, and the RGB image is converted to a gray scale image in step 118. In step 128, The feature descriptor is calculated by applying the SIFT algorithm to the gray scale image.

As described above, the technology to which the RGB-D image feature point extraction and feature descriptor generation according to the present invention is applied is to acquire an RGB image and a depth image from a Kinect type camera, and then obtain SIFT feature points ), And generates a feature point.

The generated descriptors are then bagged with histograms through Rondom Forest (RF) based codebook learning. This method of bidding uses the learned codebooks so that each feature descriptor is best distinguished when the proposed feature descriptors having the same dimension reach the leaf node through the codebook [6, 7].

Any feature points detected in the image shown in FIG. 2 are accumulated at the index of the histogram corresponding to the corresponding leaf node when reaching the leaf node through the codebook. Through this process, all the feature descriptors generated in the image pass through the codebook and are regenerated as a new histogram representing the object itself.

Each of the learned reference histograms thus generated is matched by using a histogram generated at the time of matching and a nearest neighbor (k-NN) algorithm.

In the present invention, in order to emphasize the significance of the engineer itself, the matching is the most similarity is obtained by using the classifier of SVM (Support Vector Machine) or RF 1-NN, which selects high-appearing ones, was used.

Experiment and Conclusion

In order to verify the performance of the RGB-D image feature point extraction and feature descriptor generation of the present invention, a test environment was constructed using 10 different objects as shown in FIG.

Target objects were constructed based on a test set including texture presence, camera rotation and movement change, and a training set of 25 frames.

Experimental environment was tested in Core i7 3.40Ghz CPU and GTX580 graphics processor environment for GPUSift [8].

The basic recognition rate verification for the experiment is divided into Fast Corner detection method, SURF feature descriptor detection method, RGB image based SIFT feature detection method, and RGB-D image based SIFT feature detection.

As shown in FIG. 6, the improvement of the recognition rate can be confirmed by using the description scheme to which the present invention is applied. Numerical results show that the recognition rate is improved by 12.2% compared to the case where only RGB image features are used and the recognition rate of 74.4% is confirmed in a given test environment.

Hereinabove, the RGB-D image feature point extraction and the feature descriptor generation method according to an embodiment of the present invention have been described.

Hereinafter, an RGB-D image feature point extracting and feature descriptor generating apparatus according to an embodiment of the present invention will be described with reference to FIG.

FIG. 5 is a block diagram of an apparatus for extracting feature points of RGB-D images and generating feature descriptors according to an embodiment of the present invention. Referring to FIG.

5, an RGB-D image feature point extracting and feature descriptor generating apparatus 500 to which the present invention is applied includes an image capturing unit 510, a gradient generating unit 512, a control unit 514, a rendering unit 516, And a normal vector extracting unit 518.

The gradient generating unit 512 generates a gradient based on a pixel-by-pixel position based on a pixel-by-pixel depth image function relationship of the three-dimensional image under the control of the controller 514 for controlling the overall operation of the RGB- Create a gradient.

The control unit 514 acquires the 3D image and the RGB image including the depth information from the photographing unit through the mode switching and obtains a predetermined vertex of the depth image based on the depth gradient generated from the gradient generating unit 512. [ and calculates the predetermined vector using the computed cleavage points and extracts the feature points and feature descriptors of the depth image from the image generated through the transformation into the surface normal vector.

The controller 514 calculates a feature descriptor from the 2D image by applying a Scale Invariant Feature Transform (SIFT) algorithm.

Also, the controller 514 converts a gray scale image of the RGB image obtained in the mode different from the 3D image acquisition mode, and applies a SIFT algorithm to the converted gray scale image to calculate a feature descriptor do.

The surface normal vector extraction unit 518 extracts a normalized surface normal vector from the cross-product operation of the vector calculated by the control unit 514.

That is, a normalized surface normal vector is extracted from a cross-product operation of two vectors starting from a predetermined vertex using a triple vertex.

The rendering unit 516 performs rendering using the corresponding depth image vector information configured by the surface normal vector under the control of the controller 514.

That is, a polygon mesh connecting three vertices adjacent in a counterclockwise direction with respect to one vertex of the depth image is rendered, and the direction of the normal vector of the polygon mesh generated through the vertex is defined as the surface of each of the four vertices It is specified as an average value of normal vectors.

As described above, the operation of extracting the feature point of the RGB-D image and generating the feature descriptor according to the present invention can be performed. While the present invention has been described in detail with reference to the specific embodiments thereof, . Accordingly, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by equivalents of the claims and the claims.

510: photographing unit 512: gradient generating unit
514: User interface unit 516: Rendering unit
518: surface normal vector extracting unit
[references]

Claims (12)

Generating a depth gradient based on a pixel-by-pixel position through a pixel-by-pixel depth image function relationship of a three-dimensional image including depth information obtained through a camera;
Calculating three vertexes adjacent to each other based on a predetermined vertex of the depth image based on the generated depth gradient;
Calculating a predetermined vector using the computed cleavage point and extracting a normalized surface normal vector from the computed cross-product computation of the vector;
Rendering a three-dimensional vector into a two-dimensional image by performing rendering using vector information of a depth image extracted from the extracted surface normal vector,
And extracting a feature descriptor from the 2D image by applying a Scale Invariant Feature Transform (SIFT) algorithm,
The rendering,
A polygon mesh connecting three vertices adjacent to each other in a counterclockwise direction with respect to one vertex of the depth image is rendered and the direction of a normal vector of the polygon mesh generated through the vertex is defined as a surface normal of each of the vertices Wherein the feature points are designated by an average value of the vectors. The method according to claim 1,
Converting the RGB image obtained in the mode different from the 3D image acquisition mode into a gray scale image and applying a SIFT algorithm to the converted gray scale image to calculate a feature descriptor A method for extracting feature points of RGB-D images and generating feature descriptors. The method according to claim 1,
The depth image function D (x) is expressed by the following equation, and an arbitrary offset dx with respect to the pixel position x of the depth image with respect to a predetermined pixel position x

, And the gradient

And extracting feature points of the RGB-D image from the least-squares method.

The method according to claim 1,
The three vertexes adjacent to the predetermined vertex may be expressed by the following equations,

D image feature point generation method and feature descriptor generation method according to the present invention is characterized by representing a vector from a principat point of a depth sensor camera through a pixel position x to a three-dimensional point X.

5. The method of claim 4,

Quot;
Wherein the feature point is calculated using an internal parameter K of the depth sensor camera using the following equation.

2. The method of claim 1, wherein the extracting of the normal vector comprises:
Extracting a normalized normal vector from a cross-product operation of two vectors starting from a predetermined vertex using the cleavage point, and performing an RGB-D image feature point extraction and feature descriptor generation method . delete A gradient generating unit for generating a depth gradient based on a pixel-by-pixel position through a pixel-by-pixel depth image function relationship of the three-dimensional image under the control of a control unit for controlling the overall operation of the feature- ,
And acquiring three-dimensional images and RGB images including depth information from a photographing unit through mode switching, and acquiring three-dimensional images and RGB images that are adjacent to each other based on a predetermined vertex of the depth image based on the depth gradient generated from the gradient generating unit. A control unit for calculating a predetermined vector using the computed cleavage point and extracting feature points and feature descriptors of the depth image from the image generated through conversion to a surface normal vector,
A surface normal vector extracting unit for extracting a normalized surface normal vector from a cross-product calculation of the vector calculated by the controller,
And a rendering unit for rendering the three-dimensional vector into a two-dimensional image by rendering using the vector information of the depth image extracted from the surface normal vector formed by the surface normal vector under the control of the control unit,
The rendering unit may include:
A polygon mesh connecting three vertices adjacent in a counterclockwise direction with respect to a vertex of the depth image is rendered, and the direction of a normal vector of the polygon mesh generated through the rendering is a surface normal of each of the vertices Wherein the RGB-D image feature point extracting and feature descriptor generating unit is configured to generate an RGB-D image feature point. 9. The apparatus according to claim 8,
And extracting a feature descriptor from the two-dimensional image by applying a Scale Invariant Feature Transform (SIFT) algorithm. 9. The apparatus according to claim 8,
Wherein the feature descriptor is calculated by converting a gray scale image of an RGB image obtained in a mode different from the three-dimensional image acquisition mode and applying a SIFT algorithm to the converted gray scale image, Image feature point extraction and feature descriptor generation device. The apparatus of claim 8, wherein the surface normal vector extractor comprises:
Wherein the normalized normal normal vector is extracted from a cross-product operation of two vectors starting from a predetermined vertex using the cleavage point, and the RGB-D image feature point extraction and feature descriptor generation apparatus. delete

Images