KR102220099B1 - Registration method and apparatus for 3D visual data - Google Patents

Abstract

Selecting a key point from among points included in the 3D source data and 3D target data, based on a characteristic or geometric characteristic for color, and then the characteristic or distance of the key point of the 3D source data and the key point of the 3D target data Based on, the position of the key point of the 3D target data is adjusted, and then based on the characteristic of the key point of the 3D target data corresponding to the key point of the 3D source data, the key point of the 3D source data is A 3D matching method and apparatus are provided for calculating reliability and then performing 3D matching between 3D source data and 3D target data based on the reliability.

Description

Registration method and apparatus for 3D visual data

The following embodiments relate to a method and apparatus for processing a 3D image, and more particularly, to a method and apparatus for matching 3D image data.

3D (3D) image data is widely used in various areas. 3D matching is a technology for processing 3D image data, and a correspondence relationship or a transformation relationship between points of two or more 3D image data may be generated based on 3D matching. The generated correspondence relationship or transformation relationship may be used to align two or more 3D image data. Three-dimensional registration can be applied to object or scene reconstruction, human face identification and modeling, robot navigation, object identification, and the like.

There are two types of methods for performing 3D registration. The first is a method of performing 3D matching by dividing 3D image data based on a position of a key point of the 3D image data, and then transforming a point in the divided block based on a transformation of the key point. The second is a method of aligning 3D image data based on a non-rigid Non-Alignment Iterative Closest Point (ICP) algorithm, and 3D matching is performed based on smoothness and key points.

According to an embodiment, the step of selecting a key point from among points included in 3D source data and 3D target data, based on a color characteristic or geometric characteristic, a key point of the 3D source data and the 3D Adjusting the position of the key point of the 3D target data based on the characteristic or distance of the key point of the target data, the characteristic of the key point of the 3D target data corresponding to the key point of the 3D source data On the basis of, a three-dimensional matching method comprising calculating a reliability for a key point of the three-dimensional source data and performing three-dimensional matching between the three-dimensional source data and the three-dimensional target data based on the reliability Is provided.

According to an embodiment, in the step of selecting the key point, a three-dimensional matching method of generating a feature vector based on the color feature or the geometric feature of the key point is provided.

According to an embodiment, the adjusting of the position includes a point located in an area adjacent to the key point of the 3D target data and the 3D source based on the color characteristic or the geometric characteristic of the key point. There is provided a three-dimensional matching method including calculating a feature distance between key points of data and adjusting a location of the key point based on the location of the point having the smallest feature distance.

According to an embodiment, the calculating of the reliability may correspond to a key point of the 3D source data and a key point of the 3D source data based on the color characteristic or the geometric characteristic of the key point. And calculating a feature distance between key points of the three-dimensional target data and calculating the reliability based on the feature distance.

According to an embodiment, the performing of the 3D matching includes: aligning the 3D source data based on the key point and the 3D target data. The aligned 3D source data and the 3D target There is provided a 3D matching method comprising generating a transformation matrix between data and performing 3D matching between the 3D source data and the 3D target data based on the transformation matrix.

According to an embodiment, the aligning of the 3D source data includes performing a rigid alignment on the 3D source data based on smoothness or the key point, so that the first transformed data is 3D matching comprising generating and generating second transformed data by performing a non-rigid alignment on the first transformed data based on smoothness, the key point, or the reliability A method is provided.

According to an embodiment, the generating of the transformation matrix includes generating a candidate transformation matrix between the aligned 3D source data and the 3D target data, and based on the distance between the plurality of candidate transformation matrices, a plurality of A 3D matching method including determining a final transformation matrix from among candidate transformation matrices of is provided.

According to an embodiment, the generating of the candidate transformation matrix comprises: a distance between points of the 3D source data and points of the 3D target data transformed by the candidate transformation matrix, smoothness, or based on the key point. A three-dimensional matching method for generating the candidate transform matrix is provided.

According to an embodiment, the performing of the 3D matching includes: between a point of the 3D source data converted based on the transformation matrix and a point of the 3D target data corresponding to a point of the 3D source data. A 3D matching method for performing smoothing processing on the 3D source data based on distance or smoothness is provided.

According to an embodiment, in the performing of the 3D matching, a 3D matching method of performing matching based on the preprocessed 3D source data and the 3D target data is provided.

According to an embodiment, the step of obtaining first transform data by performing rigid alignment on 3D source data based on a key point and 3D target data, the key point and the 3D target data Based on, by performing a non-rigid alignment on the first transformed data, obtaining second transformed data, generating a transform matrix between the second transformed data and the three-dimensional target data And performing a 3D matching between the 3D source data and the 3D target data based on the transformation matrix.

According to an embodiment, a three-dimensional matching method is provided in which the key point is a point selected from among points included in the three-dimensional target data and the three-dimensional source data based on a color feature or a geometric feature.

According to an embodiment, the obtaining of the second transformed data comprises performing the non-rigid alignment based on a smoothness, a reliability for a key point of the 3D source data, or the key point, and the reliability is the A three-dimensional matching method calculated based on a feature of a key point of the three-dimensional target data corresponding to a key point of three-dimensional source data is provided.

According to an embodiment, the generating of the transformation matrix includes generating a candidate transformation matrix between the second transformation data and the 3D target data, and a plurality of candidate transformation matrices based on distances between the plurality of candidate transformation matrices. A three-dimensional matching method including determining a final transformation matrix among transformation matrices is provided.

According to an embodiment, the generating of the candidate transformation matrix comprises: a distance between points of the 3D source data and points of the 3D target data transformed by the candidate transformation matrix, smoothness, or based on the key point. A three-dimensional matching method for generating the candidate transform matrix is provided.

According to an embodiment, the performing of the 3D matching includes: between a point of the 3D source data converted based on the transformation matrix and a point of the 3D target data corresponding to a point of the 3D source data. A 3D matching method for performing smoothing processing on the 3D source data based on distance or smoothness is provided.

According to an embodiment, a key point selection unit for selecting a key point from among points included in 3D source data and 3D target data based on a color characteristic or geometric characteristic, a key point of the 3D source data, and Based on the characteristic or distance of the key point of the 3D target data, a position adjusting unit for adjusting the position of the key point of the 3D target data, the 3D target data corresponding to the key point of the 3D source data A reliability calculation unit that calculates the reliability of the key point of the 3D source data based on the characteristics of the key point, and a 3D match between the 3D source data and the 3D target data based on the reliability. A 3D matching apparatus including a 3D image data matching unit is provided.

According to an embodiment, there is provided a 3D matching apparatus further comprising a data preprocessor for performing preprocessing on the 3D source data and the 3D target data.

1A to 1C are diagrams illustrating examples of 3D source data and 3D target data used by a 3D matching device according to an exemplary embodiment.
2 is a flowchart illustrating an operation performed by a 3D matching device according to an exemplary embodiment.
3 is a diagram illustrating a key point selected by a 3D matching device according to an exemplary embodiment.
4 is a flowchart illustrating a part of an operation in which a 3D matching device performs 3D matching according to an exemplary embodiment.
5A to 5D are diagrams showing results of matching by a 3D matching device according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed in various forms and implemented. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof exists, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms, including technical or scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the relevant technical field. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this specification. Does not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals in each drawing indicate the same members.

1A to 1C are diagrams illustrating examples of 3D source data and 3D target data used by a 3D registration device according to an exemplary embodiment. The 3D matching device according to an embodiment may receive one or more 3D image data. The 3D image data may be in the form of a point cloud, a depth map, or a 3D grid. The 3D target data and 3D source data are 3D image data and may be used for 3D matching.

The 3D matching apparatus according to an embodiment may generate a correspondence relationship between 3D target data and 3D source data by performing 3D matching. The 3D matching apparatus according to an embodiment may generate a transformation relationship from 3D source data to 3D target data.

1A is a diagram illustrating a curved surface of 3D source data from which textures have been removed. 1B is a diagram showing 3D target data. 1C is a diagram illustrating a curved surface of 3D target data from which textures have been removed.

The 3D matching apparatus according to an embodiment may find a correspondence relationship between 3D source data and 3D target data. Also, the 3D matching device may arrange 3D source data based on the correspondence relationship. That is, the 3D matching device may make the 3D source data match the 3D target data by aligning the 3D source data.

2 is a flowchart illustrating an operation performed by a 3D matching device according to an exemplary embodiment.

In step 210, the 3D matching apparatus according to an embodiment may receive 3D source data and 3D target data that are targets for 3D matching. The 3D matching device may receive two or more 3D image data. The 3D matching apparatus may generate a correspondence relationship with respect to the 3D image data and a transformation relationship by performing 3D matching on two or more received 3D image data. Hereinafter, a specific operation for performing 3D matching will be described.

In step 220, the 3D matching apparatus according to an embodiment may pre-process the received 3D image data. For example, a 3D matching device may use cutting, hole filling, outlier removing, smoothing, or a point cloud library for 3D image data. Other preprocessing algorithms included in can be performed. The 3D matching apparatus according to an embodiment may not perform preprocessing. That is, the 3D matching device may perform step 210 and then proceed to step 230.

In step 230, the 3D matching apparatus according to an embodiment may select a key point from among points of 3D image data. The key point refers to a point having a special characteristic for color or a geometrical characteristic among a plurality of points included in the 3D image data.

According to an embodiment, the 3D matching device may automatically select a key point. In this case, the 3D matching apparatus may project the 3D image data into 2D image data and then measure a key point based on the 2D image data. The 3D matching apparatus may obtain a position on the 3D image data of a key point measured from the 2D image data by performing back projection.

According to an embodiment, the 3D matching device may select a key point based on a user's input. The 3D matching device may receive at least one of a location of a key point and an annotation on a key point from a user.

In step 240, the 3D matching device according to an embodiment may adjust the position of the selected key point. The 3D matching device can improve the accuracy of the key point by adjusting the position of the key point. Accordingly, the accuracy of the correspondence relationship or the transformation relationship generated by the 3D matching device can be increased.

According to an embodiment, the 3D matching apparatus may extract a key point of 3D source data or a feature of a key point of 3D target data. The extracted feature may include a color feature or a geometric feature of the key point. The characteristics of the color of the key point may include a local binary pattern (LBP), scale invariant feature transform (SIFT), or speed up robust features (SURF). The geometrical features of the key point may include Normal Aligned Radial Feature (NARF), Viewpoint Feature Histogram (VPH), Fast Point Feature Histograms (FPFH), or Surface Entropy (SURE).

The 3D matching device may adjust the position of the key point of the 3D target data based on the extracted feature. The 3D matching device may calculate a feature distance between a point of 3D source data and a point of 3D target data based on a color characteristic or a geometric characteristic of the key point. The 3D matching device may adjust the position of the key point based on the calculated feature distance.

를 포인트 u의 위치로 조절할 수 있다. 즉, 3차원 정합 장치는 수학식 1에 기초하여, 3차원 타겟 데이터의 키 포인트의 위치를 조절할 수 있다.More specifically, the three-dimensional aligning apparatus according to one embodiment is located in the third claim i of the l _i ^T second key point neighboring region of dimension target data Nei (l _i ^T), a three-dimensional source data corresponding to the l _i ^T A point u having a feature distance closest to the key point l _i ^{S of can be identified.} The 3D matching device is the position of the adjusted key point of the 3D target data.

Can be adjusted to the position of point u. That is, the 3D matching device may adjust the position of the key point of the 3D target data based on Equation 1.

는 포인트 u 및 키 포인트 l_i ^T에 대한 특징 벡터간의 거리이므로, 포인트 u 및 키 포인트 l_i ^T간의 특징 거리이다.Referring to Equation 1, f() is a function for extracting a feature, and may output a feature or a geometric feature for the color of a point. According to an embodiment, the 3D matching device may generate a feature vector representing a feature extracted from a point. In particular, the feature extraction function f() can output a feature vector. therefore,

Is characterized because the distance between the vectors, the feature distance between the points u and key point l _i ^T for the point u and the key point l _i ^T.

The 3D matching device obtains a more accurate correspondence between the key point of the 3D source data and the key point of the 3D target data by adjusting the position of the key point of the 3D target data corresponding to the key point of the 3D source data. I can.

Furthermore, the 3D matching apparatus according to an embodiment may estimate corresponding key points of 3D source data and 3D target data. Specifically, the 3D matching apparatus may estimate a key point by performing at least one of a back projection method, a normal shooting method, and a near-neighbor algorithm.

In addition, the 3D matching apparatus according to an exemplary embodiment may select key points of 3D target data corresponding to key points of 3D source data. More specifically, when the distance between the key point of the 3D source data and the key point of the 3D target data corresponding thereto is greater than or equal to a preset distance, or when the narrow angle is greater than the preset angle, the 3D matching device Key points of dimensional target data can be excluded.

Although a plurality of objects (including a background) may be included in the 3D image data, noise tends to increase at the boundary between the objects. The 3D matching apparatus may exclude the key point of the 3D target data when the key point is located at a boundary between different objects included in the 3D target data. Therefore, it is possible to reduce the influence of the unstable characteristic of the point located at the boundary on the key point. Therefore, the accuracy of 3D matching can be increased.

As described above, the key point of the 3D source data and the key point of the 3D target data can accurately correspond to each other by adjusting the position of the key point and estimating or selecting the corresponding point. Therefore, the three-dimensional matching device can more accurately perform three-dimensional matching.

In step 250, the 3D matching apparatus according to an embodiment may calculate the reliability of the key point. The reliability may be calculated based on a feature distance between a key point of the 3D source data and a key point of the 3D target data corresponding thereto. Reliability and feature distance may be in inverse proportion to each other. According to an embodiment, the 3D matching device may calculate the reliability of the i-th key point of the 3D source data based on Equation (2).

는 키 포인트 v_j와 l_i사이의 특징 거리로써, l_i는 3차원 소스 데이터의 제i번째 키 포인트, v_j는 l_i에 대응하는 3차원 타겟 데이터의 키 포인트이다.Referring to Equation 2, g() may be a monotonically decreasing function.

Is a feature distance between the key points v _j and l _{i , where l i} is the i-th key point of the 3D source data, and v _j is the key point of the 3D target data corresponding to _{l i.}

를 수학식 3을 이용하여 계산할 수 있다.According to an embodiment, the 3D matching apparatus is a weighted matrix for a plurality of key points located in the 3D source data.

Can be calculated using Equation 3.

Referring to Equation 3, L is the number of key points, θ is the weight of the key point constraint, and β is the reliability of the key points of the source 3D image data.

In step 260, the 3D matching apparatus according to an embodiment may perform 3D matching. According to an embodiment, the 3D matching device performs rigid alignment or non-rigid alignment on 3D source data or 3D target data, and then aligns 3D source data. And a transformation matrix between 3D target data may be generated. The 3D matching device may perform 3D matching based on the generated transform matrix. A detailed description of this is described in FIG. 4.

Before performing step 260, the 3D matching device adjusts the position of the key point of the 3D image data in step 240, and calculates the reliability of the key point of the 3D image data in step 250. do. Since the 3D matching apparatus performs 3D matching based on the adjusted position and reliability of the key point, it is possible to more accurately find a correspondence relationship or a transformation relationship between the 3D source data and the 3D target data.

3 is a diagram illustrating a key point selected by a 3D matching device according to an exemplary embodiment. Referring to FIG. 3, the 3D matching apparatus may select a key point 310 from among a plurality of points included in the 3D image data based on a color characteristic or a geometric characteristic of the 3D image data.

The operation of selecting the key point 310 by the 3D matching device is the same as described in step 230. The 3D matching device may select the key point 310 automatically or based on a user's input. The 3D matching device may generate a feature vector representing a color feature or a geometric feature of the key point 310. The generated feature vector can be used to calculate the feature distance or reliability.

4 is a flowchart illustrating a part of an operation in which a 3D matching device performs 3D matching according to an exemplary embodiment. The 3D matching apparatus may perform 3D matching by calculating the reliability of the key point and then performing the operation shown in FIG. 4.

In step 410, the 3D matching apparatus according to an embodiment may align 3D source data based on the key point and 3D target data.

More specifically, in step 411, the 3D matching apparatus according to an embodiment may perform rigid alignment on 3D source data based on the key point and 3D target data. The 3D matching device may generate first transformed data that is aligned 3D source data by performing rigid alignment.

The 3D matching device may perform a translation, rotation, or reflection of points of 3D source data by performing rigid alignment. The distance between any two points of the 3D source data may be the same as before the rigid alignment is performed. According to an embodiment, stiffness alignment may be performed based on an Iterative Closest Point (ICP) algorithm.

In step 412, the 3D matching apparatus according to an embodiment may generate second transformed data, which is the aligned first transformed data, by performing a non-rigid alignment on the first transformed data. have. That is, the second transformed data may be generated based on the 3D source data. Non-rigid alignment may include enlargement, reduction, or nonlinear transformation. Since the 3D matching device performs rigid alignment as well as non-rigid alignment, the accuracy of correspondence between 3D target data and 3D source data can be increased.

에 대하여, 수학식 4의 에너지 함수 E₁(X)를 최소로 만드는 X 에 기초하여 제2 변환 데이터를 생성할 수 있다.The 3D matching apparatus according to an embodiment may perform non-rigid alignment based on a smoothness or a key point. The 3D matching apparatus may perform non-rigid alignment based on the reliability of the key point of the 3D source data calculated in operation 250. More specifically, the 3D matching device is the transformation matrix X= from the first transformed data to the 3D target data.

For, it is possible to generate the second transform data based on X that minimizes the energy function E _{1 (X) of Equation 4.}

의 X_i는 3 x 4의 행렬이며, 3차원 소스 데이터의 제i번째 포인트의 변환행렬이다(n은 포인트의 개수).

는 수학식 3의 가중 행렬이다. E_s(X)는 평활도의 한도에 대한 함수이며, 수학식 5로 정의될 수 있다. E_l(X)는 키 포인트의 한도에 대한 함수이며, 수학식 6으로 정의될 수 있다.Referring to Equation 4, X=

X _{i of} is a 3 x 4 matrix, and is a transformation matrix of the i-th point of the 3D source data (n is the number of points).

Is the weighting matrix of Equation 3. E _s (X) is a function for the limit of smoothness, and may be defined by Equation 5. E _l (X) is a function for the limit of the key point, and may be defined by Equation 6.

Referring to Equation 5, E _{s (X) denotes the Proveni of the transformation matrix X i} of the ith point and the transformation matrix X _j of the j th point for the ith point and the jth point adjacent to each other It can be calculated based on the F norm (Frobenius norm). As described above, since the 3D matching device generates second transform data based on X that minimizes _{E 1 (x), the probenius norms of X i} and X _j should be minimized.

을 구성한다. 3차원 정합 장치는 E₁(x)를 최소로 만드는 X 에 기초하여 제2 변환 데이터를 생성하므로, 변환행렬 X_i에 의해 변환된 v_i는 l과 가까워져야 한다.Referring to Equation 6, l is a key point of 3D source data; v _i is a key point corresponding to l among the key points of the 3D target data. l and the corresponding v _i are ordered pairs

Configure. Since the 3D matching device generates the second transformed data based on X that minimizes _{E 1 (x), v i} transformed by the transform _{matrix X i} must be close to l.

In step 420, the 3D matching apparatus according to an embodiment may generate a transformation matrix between the second transformation data and the 3D target data. The 3D matching apparatus may generate a plurality of candidate transformation matrices between the second transformation data and the 3D target data, and then determine a final transformation matrix from among the plurality of candidate transformation matrices based on a distance between the candidate transformation matrices.

More specifically, in step 421, the 3D matching apparatus according to an embodiment may generate a first-order candidate transformation matrix that is a transformation matrix between the second transformation data and the 3D target data. A detailed operation of generating the candidate transformation matrix will be described in step 423.

In step 422, the 3D matching apparatus according to an embodiment may check whether the first-order candidate transformation matrix satisfies the convergence condition. The convergence condition will be described in detail in step 424. If the convergence condition is satisfied, the 3D matching device proceeds to step 425. Accordingly, the first-order candidate transformation matrix may be determined as the final transformation matrix. If the convergence condition is not satisfied, the 3D matching apparatus performs step 423.

In step 423, the 3D matching apparatus according to an embodiment may generate an nth-th transform matrix. The 3D matching apparatus may use the 3D source data converted to the n-1 th transform matrix calculated in the previous step.

Specifically, when generating the second order transformation matrix, the 3D matching apparatus may use transformation data for the first order candidate transformation matrix obtained by transforming the second transformation data according to the first order candidate transformation matrix. Similarly, when generating a third order transformation matrix, the 3D matching apparatus may use transformation data obtained by converting transformation data for the first order candidate transformation matrix according to the second order candidate transformation matrix.

The 3D matching device may generate an n-th candidate transformation matrix based on X that minimizes the energy function E(X) of Equation 7.

Referring to Equation 7, E _s (X) is a function for the limit of smoothness and may be defined by Equation 5, and E _l (X) is a function for the limit of key points, and is defined by Equation 6. Can be.

는 E_s (X)의 가중계수이고, B는 E_l(X)의 가중계수이다. 일실시예에 따른, 3차원 정합 장치는 국부적 최적 해(local optimal solution)가 계산되는 것을 방지하기 위하여, 수학식 7의

를 점진적으로 감소시킬 수 있다. 즉,

로써

은 미리 설정된 가중계수이며,

의 관계가 성립할 수 있다. 3차원 정합 장치는

을 사용하여 제n차 후보 변환행렬을 생성할 수 있다.

Is the weighting factor of E _s (X), and B is the weighting factor of _{E l (X).} According to an embodiment, in order to prevent a local optimal solution from being calculated, the 3D matching device

Can be gradually decreased. In other words,

As

Is a preset weighting factor,

The relationship of can be established. 3D matching device

The n-th candidate transformation matrix can be generated using.

E _d (X) is a function of the limit of the distance between the points of the 3D source data converted to the n-th order transformation matrix and the corresponding points of the 3D target data. E _d (X) may be defined by Equation 8.

는 3차원 소스 영상 데이터의 포인트의 집합이다.

는 제j차(j는 n-1과 동일) 후보 변환행렬에서 3차원 소스 데이터의 제 i번째 포인트의 변환행렬이다.

는 3차원 타겟 데이터이며, dist는 3차원 소스 데이터의 포인트 v_i의 위치가 변환행렬

로 변환된 위치와 이에 대응하는 3차원 타겟 데이터의 포인트간의 거리이다. w_i는 이진 파라미터로써,

에 대응하는 3차원 타겟 데이터의 포인트가 존재할 때, w_i는 1이고, 포인트가 존재하지 않는 경우, w_i는 0이다.Referring to Equation 8,

Is a set of points of the 3D source image data.

Is a transformation matrix of the i-th point of the 3D source data in the jth order (j is equal to n-1) candidate transformation matrix.

Is the 3D target data, and dist is the location _{of the point v i of the 3D source data.}

It is the distance between the converted position and the corresponding point of the 3D target data. w _i is a binary parameter,

When there is a point of the 3D target data corresponding to, w _i is 1, and when there is no point, w _i is 0.

When the difference between the 3D source data and the 3D target data is relatively large, and the transformation matrix is calculated by comparing only the key points of the 3D source data and the corresponding key points of the 3D target data, the key points of the 3D source data The difference between the conversion result of and the conversion result of points around the key point may be large.

The 3D matching apparatus according to an exemplary embodiment can solve this problem by additionally considering not only the key point but also other points. More specifically, the 3D matching apparatus may calculate the nth-th transform matrix by additionally considering the inequality condition of Equation 9.

는 제n차 변환행렬에 따라 변환된 3차원 소스 데이터의 제u번째 포인트에 대한 변환행렬의 제e번째 원소 값이다.

는 제n차 변환행렬에 따라 변환된 3차원 소스 데이터의 제u번째 포인트에 대한 변환행렬의 제m번째의 키 포인트, 즉 제i_m번째 포인트에 대한 변환행렬의 제e번째 원소 값이다. v_u는

의 이웃 영역에 포함되고,

은 3차원 소스 데이터 중의 제m번째 키 포인트, 즉 제i_m번째 포인트이다.

는 v_u와

사이의 거리에 비례할 수 있다.Referring to Equation 9,

Is the value of the e-th element of the transformation matrix for the u-th point of the 3D source data converted according to the n-th transformation matrix.

Is the m-th key point of the transformation matrix with respect to the u-th point of the 3D source data transformed according to the n-th transformation matrix, that is, the e-th element value of the transformation matrix for _{the i-m-th point.} v _u is

Is included in the neighboring area of,

Is the m-th key point in the 3D source data, that is, the i _m- th point.

Is v _u and

It can be proportional to the distance between them.

Since the 3D matching apparatus may calculate the transformation matrix in consideration of the neighboring regions of the key points, a difference between the transformation result of the key point of the 3D source data and the transformation result of the points surrounding the key point may be reduced. As a result, the accuracy of the generated three-dimensional registration can be improved.

In step 424, the 3D matching apparatus according to an embodiment may check whether the nth-th transform matrix satisfies the convergence condition. More specifically, when the 3D matching apparatus satisfies Equation 10, it can be confirmed that the nth-th transform matrix satisfies the convergence condition.

은 제n-1차 변환행렬, Xⁿ은 제n차 변환행렬, δ는 미리 설정한 반복 중지 한계치이다. 수렴 조건을 만족하지 않는 경우, 3차원 정합 장치는 단계(423)로 진행하여 후보 변환행렬을 다시 계산한다. 이로써 계산되는 제n+1차 후보 변환행렬은 제n차 변환행렬 및 제n차 변환행렬에 의해 변환된 변환 데이터를 참조할 수 있다.Referring to Equation 10,

Is an n-th transformation matrix, X ⁿ is an n-th transformation matrix, and δ is a preset iteration stop limit. If the convergence condition is not satisfied, the 3D matching apparatus proceeds to step 423 and recalculates the candidate transform matrix. The n-th order candidate transformation matrix calculated as a result may refer to an n-th order transformation matrix and transform data converted by the n-th order transformation matrix.

When the convergence condition is satisfied, the 3D matching apparatus proceeds to step 425 and may determine the nth-th transform matrix as the final transform matrix.

In step 430, the 3D matching apparatus according to an embodiment may perform a smoothing process on the transform matrix. The 3D matching apparatus may perform smoothing processing based on a smoothness, a key point of the 3D source data, and a distance between the key points of the 3D target data corresponding thereto. In particular, the 3D matching apparatus may use third transformed data obtained by transforming the 3D source data using the transform matrix calculated in step 420. More specifically, the 3D matching device may perform smoothing processing based on X making _{the energy function E 2 (X) of Equation 11 to a minimum.}

는 평활도의 한도에 대한 함수 E_s(X)의 가중계수이다. E_s는 평활도의 한도에 대한 함수로 수학식 5로 정의될 수 있다. E_d (X)는 제3 변환 데이터의 포인트 및 이에 대응하는 3차원 타겟 데이터의 포인트 간의 거리 제약 함수로써, 수학식 8로 정의될 수 있다.Referring to Equation 11,

Is the weighting factor of _{the function E s} (X) on the limit of smoothness. E _s may be defined as Equation 5 as a function of the smoothness limit. E _d (X) is a distance constraint function between the point of the third transform data and the point of the 3D target data corresponding thereto, and may be defined by Equation 8.

In step 440, the 3D matching apparatus according to an embodiment may perform 3D matching on 3D source data and 3D target data. The 3D matching device may perform 3D matching based on the smoothed transform matrix.

5A to 5D are diagrams showing results of matching by a 3D matching device according to an exemplary embodiment. The 3D matching apparatus may perform 3D matching between the 3D source data shown in FIG. 1A and the 3D target data shown in FIGS. 1B to 1C based on the 3D target data shown in FIGS. 1B to 1C. . 3D source data arranged according to a result of performing 3D matching are shown in FIGS. 5A to 5D.

5A to 5B are diagrams showing results of performing 3D matching using an Optimal Step non-rigid ICP (OSICP) method. That is, a diagram showing a result of sorting the 3D source data shown in FIG. 1A based on the result of 3D matching. 5A is a view showing only the curved surface by removing the texture, and FIG. 5B is a view showing the texture on the curved surface.

5C to 5D are diagrams illustrating results of arranging 3D source data according to 3D registration by a 3D matching device according to an exemplary embodiment. 5C is a view showing only the curved surface by removing the texture.

5A to 5D, when 3D source data is aligned according to 3D registration, it may be aligned in the shape of 3D target data. Comparing the key points around the eyes, nose, and mouth of FIGS. 5B and 5D, it can be seen that the 3D matching apparatus according to an exemplary embodiment aligned more accurately than the OSICP method.

A 3D matching device and an OSICP method may be compared based on a quantization value obtained by calculating a degree of matching between the 3D target data and the 3D source data. Specifically, quantization values for smoothness and quantization values for data matching are calculated and compared.

After calculating the displacement difference between each point and the points in the neighboring area, the average of the displacement differences between all points is defined as a quantized value of the smoothness. The quantization value of the smoothness may decrease as the 3D source data and the 3D target data accurately correspond.

After calculating the distance between the points of the 3D source data and the corresponding points of the 3D target data, the distances for all points are defined as quantization values for data matching. The quantization value for data matching may decrease as the 3D source data and the 3D target data accurately correspond.

Table 1 shows a comparison of quantization values for smoothness of 3D matching and quantization values for data matching performed according to the 3D matching method performed by the 3D matching apparatus and the OSICP method.

Way Quantization value of smoothness Quantization value for data match OSICP method 0.3361 0.6486 3D matching device 0.3152 0.6096

Referring to Table 1, it can be seen that the correspondence of the 3D registration performed by the 3D matching apparatus according to an embodiment is more accurate.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

210: 3D source data/3D target data input
220: data preprocessing
230: key point selection
240: Key point position adjustment
250: reliability calculation
260: Perform 3D registration

Claims (18)

Selecting a key point from among points included in the 3D source data and the 3D target data based on a color feature or a geometric feature;
Adjusting a position of a key point of the 3D target data based on a key point of the 3D source data and a characteristic or distance of the key point of the 3D target data;
Calculating a reliability for a key point of the 3D source data based on a characteristic of a key point of the 3D target data corresponding to the key point of the 3D source data; And
3D matching between the 3D source data and the 3D target data is performed based on the reliability,
The step of performing the three-dimensional matching,
Aligning the 3D source data based on the key point and the 3D target data;
Generating a transformation matrix between the aligned 3D source data and the 3D target data; And
Based on the transformation matrix, performing 3D matching between the 3D source data and the 3D target data,
The step of generating the transformation matrix,
Generating a candidate transformation matrix between the aligned 3D source data and the 3D target data; And
Determining a final transformation matrix from among a plurality of candidate transformation matrices based on the distance between the plurality of candidate transformation matrices,
Generating the candidate transformation matrix,
Generating the candidate transformation matrix based on the distance between the points of the 3D source data and the points of the 3D target data transformed by the candidate transformation matrix, smoothness, and the key points,
Three-dimensional registration method.
The method of claim 1,
The step of selecting the key point,
A three-dimensional matching method for generating a feature vector based on the color feature or the geometric feature of the key point. The method of claim 1,
The step of adjusting the position,
Calculating a feature distance between a point located in a region adjacent to a key point of the 3D target data and a key point of the 3D source data based on the color characteristic or the geometric characteristic of the key point; And
Adjusting the position of the key point based on the position of the point having the smallest feature distance
Three-dimensional matching method comprising a. The method of claim 1,
The step of calculating the reliability,
Calculate a feature distance between a key point of the 3D source data and a key point of the 3D target data corresponding to the key point of the 3D source data based on the color characteristic or the geometric characteristic of the key point Step to do; And
Calculating the reliability based on the feature distance
Three-dimensional matching method comprising a. delete The method of claim 1,
Aligning the three-dimensional source data,
Generating first transformed data by performing rigid alignment on the 3D source data based on smoothness or the key point; And
Generating second transformed data by performing a non-rigid alignment on the first transformed data based on smoothness, the key point, or the reliability;
Three-dimensional matching method comprising a.

delete delete The method of claim 1,
The step of performing the three-dimensional matching,
Smoothing processing for the 3D source data based on a distance or smoothness between points of the 3D source data converted based on the transformation matrix and points of the 3D target data corresponding to the points of the 3D source data A three-dimensional matching method to perform.
The method of claim 1,
The step of performing the three-dimensional matching,
A 3D matching method for performing matching based on the preprocessed 3D source data and the 3D target data. Obtaining first transform data by performing rigid alignment on the 3D source data based on the key point and the 3D target data;
Obtaining second transformed data by performing Non-Rigid Alignment on the first transformed data based on the key point and the 3D target data;
Generating a transformation matrix between the second transformed data and the 3D target data; And
Based on the transformation matrix, performing 3D matching between the 3D source data and the 3D target data,
The step of generating the transformation matrix,
Generating a candidate transform matrix between the second transform data and the 3D target data; And
Determining a final transformation matrix from among a plurality of candidate transformation matrices based on the distance between the plurality of candidate transformation matrices,
Generating the candidate transformation matrix,
Generating the candidate transformation matrix based on the distance between the points of the 3D source data and the points of the 3D target data transformed by the candidate transformation matrix, a smoothness, and the key point,
Three-dimensional registration method.
The method of claim 11,
The key point is,
A three-dimensional matching method in which a point is selected based on a color feature or a geometric feature from among the points included in the three-dimensional target data and the three-dimensional source data. The method of claim 11,
Obtaining the second transformed data,
Performing the non-rigid alignment based on smoothness, reliability for a key point of the 3D source data, or the key point,
The reliability is calculated based on a characteristic of a key point of the 3D target data corresponding to a key point of the 3D source data. delete delete The method of claim 11,
The step of performing the three-dimensional matching,
Smoothing processing for the 3D source data based on a distance or smoothness between points of the 3D source data converted based on the transformation matrix and points of the 3D target data corresponding to the points of the 3D source data A three-dimensional matching method to perform. A key point selection unit for selecting a key point from among points included in the 3D source data and the 3D target data, based on a color characteristic or a geometric characteristic;
A position adjusting unit configured to adjust a position of a key point of the 3D target data based on a key point of the 3D source data and a characteristic or distance of the key point of the 3D target data;
A reliability calculator configured to calculate a reliability for a key point of the 3D source data based on a characteristic of a key point of the 3D target data corresponding to the key point of the 3D source data; And
A 3D image data matching unit configured to perform 3D matching between the 3D source data and the 3D target data based on the reliability,
The 3D image data matching unit,
Based on the key point and the 3D target data, the 3D source data is aligned, a transformation matrix between the aligned 3D source data and the 3D target data is generated, and based on the transformation matrix, the 3D matching between the 3D source data and the 3D target data,
The 3D image data matching unit,
In order to generate the transformation matrix, a candidate transformation matrix between the aligned 3D source data and the 3D target data is generated, and a final transformation matrix among a plurality of candidate transformation matrices based on the distance between the plurality of candidate transformation matrices To decide,
The 3D image data matching unit,
In order to generate the candidate transformation matrix, generating the candidate transformation matrix based on the distance, smoothness, and key points between points of the 3D source data and points of the 3D target data transformed by the candidate transformation matrix. ,
Three-dimensional registration device.
The method of claim 17,
A data preprocessing unit that performs preprocessing on the 3D source data and the 3D target data
A three-dimensional matching device further comprising a.

Images