KR102455546B1 - Iterative Closest Point Algorithms for Reconstructing of Colored Point Cloud - Google Patents

Abstract

The present invention relates to an iterative nearest point search method for reconstructing a color-reflective point cloud, and the iterative nearest point search method for matching a source point cloud with a reference point cloud in the image processing system of the present invention includes According to an iterative K-nearest point algorithm that assigns a matching probability to a predetermined number of neighbors using a pose vector comprising a rotation vector and a translation vector between the cloud and the reference point cloud, associated with the pose vector and the source Calculating a cost based on the Mahalanobis distance between a point and a reference point, and repeating the process of updating the pose vector using the rotation increment of the rotation vector and the translation increment of the translation vector are repeated a plurality of times, and generating a point cloud in which the source point cloud is updated by the pose vector.

Description

Iterative Closest Point Algorithms for Reconstructing of Colored Point Cloud

The present invention relates to an Iterative Closest Point (ICP) search method, and in particular, to reconstruct a 3D image such as a person, using an iterative K-Closest Point Algorithms, A point in a source point cloud is naturally matched to a reference point cloud by K-nearest points, and the iterative nearest point search method is capable of minimizing the cost.

RGB-D (RGB-Depth) cameras have recently been used in many applications in robotics and 3D modeling due to their cost savings and ease of use. A dense 3D map can be created by taking an RGB-D video of the environment and registering point clouds obtained from the RGB-D image. A 3D model of an object can be obtained using a multiview system consisting of several calibrated RGB-D cameras.

The Iterative Closest Point (ICP) algorithm plays an important role in the compensation of registration and calibration errors. The ICP algorithm matches the two point clouds by alternately searching for correspondences and estimating poses. Over the past 30 years, researchers have been developing algorithms to overcome the problems of algorithms. In order to reduce the ambiguity of finding correspondences that depend only on 3D positions, the correspondence search stage has been developed to find the closest points in high-dimensional space using both 3D positions and colors. Due to the limited resolution of the point cloud, it may not be possible to find an exact point corresponding to a point. This can be addressed by allowing a point to match any point in another point cloud and assigning a probability of matching to every match. Alternatively, the cost function is improved by replacing the point-to-point distance with a point-to-plane or plane-to-plane distance. By relaxing the assumption of point-to-point correspondence, the algorithm becomes more accurate and robust.

However, since the conventional ICP algorithm lacks natural image reconstruction and is expensive, there is a need for improvement.

Accordingly, the present invention has been devised to solve the above problems, and an object of the present invention is to naturally match the points of the point cloud to the K-nearest point group and to minimize the cost in reconstructing a 3D image such as a person. In order to be able to do this, we use an iterative K-nearest-nearest-points algorithm, but in order to strike a balance between computational speed and accuracy, we allow a point to match a K-nearest point and limit the probability of matching between source/reference points to only K pairs. The purpose of the pose search is to provide an iterative nearest point search method in 6D vector space by combining 3D coordinates and the color of the point.

First, to summarize the features of the present invention, the iterative nearest point search method for matching a source point cloud with a reference point cloud in an image processing system according to an aspect of the present invention for achieving the above object is, According to an iterative K-nearest point algorithm that assigns a matching probability to a predetermined number of neighbors using a pose vector comprising a rotation vector and a translation vector between a reference point cloud, the pose vector is associated with the source point and calculating a cost based on the Mahalanobis distance between reference points; and repeating the process of updating the pose vector using the rotation increment of the rotation vector and the translation increment of the translation vector a plurality of times to generate a point cloud in which the source point cloud is updated by the pose vector. includes steps.

The cost corresponds to a value added after weighting the square of the point-to-point or point-to-plane distance between the source point and the reference point.

The search for the pose vector according to the iterative K-nearest point algorithm is performed in a 6D vector space in which a 3D coordinate vector and a 3D color vector of the source and reference points are combined.

,

에 따라 상기 포즈 벡터가 업데이트된다.In the iteration process, an increment in which the gradient of the cost becomes zero is calculated, and the iteration process is performed within a predetermined maximum number of iterations until the magnitude of the rotational increment and the translational increment is less than or equal to a threshold, and the rotation is performed. According to the rotation increment ΔR of the vector R and the translation increment ΔT of the translation vector T, in the iteration process

,

The pose vector is updated accordingly.

In the iterative nearest point search method, before calculating the cost, the iterative K-nearest point algorithm with respect to the depth value of the source point obtained from the 3D camera with the pose vector as a constant Accordingly, a depth vector search step of performing the process of updating the depth value a plurality of times so that the cost based on the Mahalanobis distance is minimized; and applying the updated depth value to the source point cloud in the step of calculating the cost.

The iterative nearest point search method further comprises the step of merging multi-view point clouds, wherein in the merging, according to the updated pose vector, a point farthest from a nearest point cloud among the multi-view point clouds. You can merge them in point cloud order.

In the merging, according to the updated depth value, merging from the furthest point cloud to the nearest point cloud order among the multi-view point clouds, and then alternately inverting the processing order of the furthest and the closest ones in the round can

And, in the image processing system according to another aspect of the present invention, the recording medium in which the computer-readable code for processing the iterative nearest point search for matching the source point cloud with the reference point cloud is recorded, the source point cloud and the reference point According to an iterative K-nearest point algorithm that assigns matching probabilities to a predetermined number of neighbors using a pose vector comprising a rotation vector and a translation vector between clouds, the pose vector is associated with the source point and the reference point the ability to calculate a cost based on the Mahalanobis distance between them; and repeating the process of updating the pose vector using the rotation increment of the rotation vector and the translation increment of the translation vector a plurality of times to generate a point cloud in which the source point cloud is updated by the pose vector. function can be implemented.

According to the iterative nearest point search method according to the present invention, it uses an iterative K-nearest point algorithm, but allows a point to match the K-nearest point and source/reference point in order to strike a balance between computation speed and accuracy. Allocating the inter-match probability to only K pairs, but the pose search is performed in 6D vector space by combining 3D coordinates and the color of the point. It is naturally matched to the reference point cloud and the cost can be minimized.

In the present invention, the cost function integrating all important results is defined as the weighted sum of squares for point-to-point and point-to-plane distances to improve stability and accuracy. In the present invention, 1) it is possible to minimize the cost by finding a pose parameter that minimizes the cost through the ICP algorithm, and 2) it is assumed that a point cloud is obtained from the RGB-D image, and there is an error in the depth value measured with the RGB-D camera. Considering the existence of

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical spirit of the present invention.
1A is a flowchart illustrating a method of reconstructing a point cloud in an image processing system according to an embodiment of the present invention.
1B is a flowchart illustrating an iterative K-nearest point algorithm for depth vector search in an image processing system according to an embodiment of the present invention.
2 illustrates (a) an RGB-D camera system comprising an RGB color camera, two infrared (IR) stereo cameras and one IR projector, for acquiring a depth image, (b) according to a depth value Z; This is an example of the standard deviation σ _z .
3 is an example of positioning of an RGB-D camera for multi-view image acquisition.
4 shows a sample RGB-D image acquired with an RGB-D camera.
5 is an example of rendered images for comparing the conventional method and the merged point clouds of the present invention.
6 is a diagram for explaining an example of an implementation method of an image processing system for processing reconstruction of a point cloud according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In this case, the same components in each drawing are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of already known functions and/or configurations will be omitted. The content disclosed below will focus on parts necessary to understand the operation according to various embodiments, and descriptions of elements that may obscure the gist of the description will be omitted. Also, some components in the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not fully reflect the actual size, so the contents described herein are not limited by the relative size or spacing of the components drawn in each drawing.

In describing the embodiments of the present invention, if it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. And, the terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification. The terminology used in the detailed description is for the purpose of describing embodiments of the present invention only, and should not be limiting in any way. Unless explicitly used otherwise, expressions in the singular include the meaning of the plural. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, acts, elements, some or a combination thereof, one or more other than those described. It should not be construed to exclude the presence or possibility of other features, numbers, steps, acts, elements, or any part or combination thereof.

In addition, terms such as first and second may be used to describe various components, but the components are not limited by the terms, and the terms are for the purpose of distinguishing one component from other components. used only as

1A is a flowchart illustrating a method of reconstructing a point cloud in an image processing system according to an embodiment of the present invention.

를 데이터베이스의 참조 포인트 클라우드 Sr=

와 비교하고 유사도 연관성을 탐색하여 그와 정합하기 위한 포인트 클라우드로 변환(재구성)한다. 이외에도 본 발명에서 반복적인 K-최근접 포인트 알고리즘에 따라 업데이트된 포즈 벡터나 깊이 벡터는 멀티-뷰의 병합이나 정합에도 이용될 수 있다.1A, the image processing system of the present invention for performing iterative nearest point search through the ICP of the present invention, that is, iterative K-Closest Point Algorithms, is a camera ( A three-dimensional captured image including a depth value in addition to a two-dimensional coordinate value based on an RGB-D camera or a stereo camera, that is, a point cloud is received (S110). For example, the point cloud may be formed of point data including color and grayscale values at points (X, Y, Z) of a three-dimensional coordinate system. In the image processing system of the present invention, the received source point cloud S _s =

is the reference point cloud in the database Sr=

, and converts (reconstructs) into a point cloud for matching with similarity associations. In addition, the pose vector or depth vector updated according to the iterative K-nearest point algorithm in the present invention may be used for merging or matching multi-views.

<Iterative K-Nearest Point Algorithm for Pose Vector Search>

와 같이 포즈 벡터 R, T에 의해 변환될 수 있다. The image processing system of the present invention searches for pose parameters (pose vectors) including a rotation matrix R (eg, 3Х3 matrix) and a translation vector T (eg, three-dimensional) between a source point cloud and a reference point cloud. (S120). The relationship between the pair of points X _i ^(s) and X _j ^(r) corresponding to the source point cloud S _s and the reference point cloud S _r can be expressed using a pose parameter as in [Equation 1]. The point X _i ^(s) of the source point cloud S _s is according to [Equation 1]

It can be transformed by the pose vectors R and T as

[Equation 1]

와 3D 색상 벡터 C_i ^(s)을 결합하여 6D 벡터 공간에서 이루어진다. 참조 포인트 클라우드의 6D 벡터 포인트들은 3D 좌표 X_i ^(r)와 색상(예, RGB, Red Green Blue) 벡터 C_i ^(r)을 결합하여

와 같이 나타낸다(o은 하다마드곱(Hadamard product)). 소스 포인트 클라우드의 6D 벡터 포인트 {

, βoC_i ^(s)} (i=1~N_s)에 대한 ICP 알고리즘에 따른 K-최근접 네이버의 세트 N_i에서의 K-최근접 포인트 X_j ^(r) (j∈N_i)의 탐색이 이루어진다. 여기서, β는 서로 다른 색상 모델 간(예, RGB-D / YIQ)의 밸런싱(balancing)을 위해 곱해진다. 이때,

와 X_j ^(r) 사이의 공간 상의 거리 차이 벡터 d_ij는 [수학식2]와 같이 정의되고,

와 X_j ^(r) 사이의 6D 차이 벡터 c_ij는 [수학식3]과 같이 정의된다. The image processing system of the present invention for such a transformation calculates the matching probability between source/reference points in each iteration process only to neighbors of K pairs having the closest similarity by K-nearest point (K is a positive integer). Allocate and explore Navigation is the 3D coordinates of the source point

and the 3D color vector C _i ^(s) are combined in 6D vector space. The 6D vector points of the reference point cloud are obtained by combining 3D coordinates X _i ^(r) and color (eg RGB, Red Green Blue) vector C _i ^(r) .

It is expressed as (o is the Hadamard product). 6D vector points in the source point cloud {

, βoC _i ^(s) } (i=1~N _s ) Search for the K-nearest point X _j ^(r) (j∈N _i ) in the set N _i of K-nearest neighbors according to the ICP algorithm this is done Here, β is multiplied for balancing between different color models (eg, RGB-D / YIQ). At this time,

A distance difference vector d _ij in space between and X _j ^(r) is defined as in [Equation 2],

A 6D difference vector c _ij between and X _j ^(r) is defined as [Equation 3].

[Equation 2]

[Equation 3]

The set N _i of K-nearest neighbors is sorted in ascending order of the magnitude of c _ij (||c _ij ||). The first element at the nearest index of N _i is denoted by a _i .

In the pose vector search step, the image processing system of the present invention calculates the cost function E associated with the pose vector as shown in Equation 4, and minimizes the pose vector, ie, R (eg, 3Х3 matrix), translation A vector T (eg, three-dimensional) is searched for (S120).

[Equation 4]

이다. M_i,j는 임의의 형태에서 소스 포인트와 참조 포인트 간의 마할라노비스(Mahalanobis) 거리로서, 비용함수를 정의하기 위한 행렬(예, 3Х3 행렬)이다. 또한, 1/2은 E의 계산 상에서 2차 함수가 구분되어 2번 곱해진 팩터를 제거하기 위함이다. In [Equation 4], p _ij is the matching probability between X _i ^(s) and X _j ^(r) . where p _ij =0 at (j∈N _i ) or

to be. M _i,j is a Mahalanobis distance between a source point and a reference point in an arbitrary form, and is a matrix for defining a cost function (eg, a 3Х3 matrix). In addition, 1/2 is to remove a factor that is multiplied twice by dividing a quadratic function in the calculation of E.

와 X_j ^(r)가 3D 위치와 색상에서 유사도가 가깝다고(네이버) 가정할 때, 매칭 확률 p_ij는 [수학식5]와 같이, 나타낼 수 있다. 여기서, γ_i는 모든 p_ij가 0이 아니라면 p_ij의 합이 1이 되게 하는 정규화 계수이고, τ_c는 ICP 알고리즘 상의 거리 임계치이다.corresponding point

Assuming that and X _j ^(r) have close similarity in 3D position and color (Naver), the matching probability p _ij can be expressed as in [Equation 5]. Here, γ _i is a normalization coefficient that makes the sum of p _ij equal to 1 if all p _ij are non-zero, and τ _c is the distance threshold on the ICP algorithm.

[Equation 5]

의 평균이 τ_c 보다 크면, τ_c 는 해당 평균으로 대체된다. 이는 충분히 많은 대응 포인트들이 계산되도록 한다.

의 표준편차 σ_c는 τ_c 로 설정될 수 있다. 초기 포즈 벡터 R⁰, T⁰는 소정의 알고리즘을 통해 추정되는 것도 가능하고, 미리 주어진 것으로 가정할 수도 있다. In the present invention, for the initial pose vectors R ⁰ , T ⁰ ,

If the average of τ _c is greater than τ c , τ _c is replaced with the corresponding average. This allows a sufficiently large number of corresponding points to be calculated.

The standard deviation σ _c of can be set to τ _c . The initial pose vectors R ⁰ , T ⁰ may be estimated through a predetermined algorithm, and may be assumed to be given in advance.

Unlike the conventional plane-to-plane distance, for a point-to-plane (or point-to-point) spatial distance between a source point and a reference point, M _i,j depends only on the reference points, and is recalculated after searching for all correspondences. Without it, it is calculated only once as in [Equation 6]. Here, n _j is the surface normal vector on the three-dimensional surface of X _j ^(r) . The rank of n _j n _j ^T is 1, so it is noninvertible.

[Equation 6]

In order to increase the mathematical stabilization in the present invention, M _i,j is defined as in [Equation 7] using the identity matrix I and the coefficient ε (eg, 0.001). Here, M _i,j allows the cost function corresponding to the summed value to be calculated after applying a weight to the square of the point-to-point and point-to-plane distances as in [Equation 4].

[Equation 7]

In order to minimize the cost function E as in [Equation 4], a Gauss-Newton algorithm is applied to the ICP algorithm in the present invention. The matching probability p _ij can vary depending on the pose parameters, but is considered a fixed variable.

The image processing system of the present invention calculates a transform increment of a pose vector in an iterative process ( S130 ). Incremental rotation ΔR is the same as [Equation 8], and Δθ _X , Δθ _Y , and Δθ _Z are rotation angles for each axis in the XYZ rectangular coordinate system. The translational increment can be expressed as ΔT=(ΔT _X , ΔT _Y , ΔT _Z ), and the transformation increment parameters are the vector δ=(Δθ _X , Δθ _Y , Δθ _Z ,ΔT _X , ΔT _Y , ΔT _Z ) ^T can be represented.

[Equation 8]

Accordingly, d _i,j in the increment δ calculated by the pose vectors can be expressed as [Equation 9] and [Equation 10] (J _i is the Jacobian matrix), and the rate of change of the cost function E ( gradient) ▽E can be expressed as [Equation 11].

[Equation 9]

[Equation 10]

[Equation 11]

At this time, δ that satisfies ▽E=0 so that the cost is minimized can be calculated as in [Equation 12], and the rotation and translation vector to which the increment δ is applied is [Equation 13] (by 'ΔR*R' updated to a new R), [Equation 14] (updated to a new T by 'ΔR*T+ΔT') may be updated and applied.

[Equation 12]

[Equation 13]

[Equation 14]

In the image processing system of the present invention, an updated point cloud in which X _i ^(s) is rotated and translated by the changed rotation and translation vector is generated by using the changed rotation and translation vector (S140), and a Gauss-Newtonian Until the algorithm application converges, the changed rotation and translation vectors are used for cost calculation and the above process is repeated (S150). This iteration process for registration from the source point to the reference point is repeated until the magnitude of the rotation and translation increments is below a threshold (eg rotation 0.001 ^o , translation 0.001 mm), which is the maximum number of iterations (eg 80 times). ) can be repeated within that time.

In the ICP algorithm for matching the above source point cloud S _s to the reference point cloud S _r , the process of searching for a pose vector is summarized in Algorithm1 below.

*복셀크기)로 설정될 수 있다.The initial pose vectors R ⁰ , T ⁰ are set to converge the above calculations to obtain an accurate pose vector. At first, while performing the above iterative process to find the pose vectors for registration from the source point to the reference point for some coarse voxels (eg, size 4cm*2cm*1cm), the more accurate solution is obtained and the cost function is minimized. The distance threshold τ _c is (

*Voxel size).

<Iterative K-Nearest Point Algorithm for Depth Vector Search>

1B is a flowchart illustrating an iterative K-nearest point algorithm for depth vector search in an image processing system according to an embodiment of the present invention.

In the iterative process according to the iterative K-nearest point algorithm, matching between the source point and the reference point may be achieved by searching for a depth value obtained from a 3D camera. In order to minimize the cost function of [Equation 4], a solution to the measured depth value Z _i ^(s) of the point X _i ^(s) can be obtained. The pose vectors R and T are regarded as constants, and the cost function E can be viewed as a sum of independent cost functions E _i as shown in [Equation 15] and [Equation 16] (S210).

[Equation 15]

[Equation 16]

If the normalized coordinate vector corresponding to X _i ^(s) is expressed as x _i , then X _i ^(s) = Z _i ^(s) x _i becomes, and the gradient of d _ij ▽ d _ij is the same as [Equation 17].

[Equation 17]

을 만족하는

은 [수학식19]와 같이 계산된다. When the chain rule is applied, the derivative of E _i with respect to Z _i ^(s) is the same as [Equation 18], so that the cost is minimized.

to satisfy

is calculated as in [Equation 19].

[Equation 18]

[Equation 19]

In the ICP algorithm for matching the above source point cloud S _s to the reference point cloud S _r , the process of searching the depth vector is summarized in Algorithm2 below. The spatial distance threshold τ _d is set on the order of 4 cm to prevent a spatially distant point from becoming a neighbor to X _i ^(s) .

대신에

으로 대체된다. Point pairs having similar colors according to the weight function of [Equation 5] have a high matching probability p _ij . For 3D positions excluding colors in [Equation 5], ||c _i,j || is replaced with ||d _i,j ||, the condition

Instead of

is replaced by

는 τ_d 보다 클 수 있고, 이때

는 Z_i ^(s) +τ_d 또는 Z_i ^(s) - τ_d 로 설정될 수 있다. In Algorithm2, it is set to a positive constant for mathematical stabilization, such as η=0.1. at what point

may be greater than τ _d , where

can be set as Z _i ^(s) +τ _d or Z _i ^(s) - τ _d .

로 대체하지 않고 비용 함수 F를 최소화할 수 있다. 여기서, p_i는 Z_i ^(s)에 대한

의 식별자로서, 0 또는 1의 값을 갖는다. △Z_i,m은 F를 최소화하기 전의 초기 차이 Z_i ^(s) - Z_m ^(s), R_i는 이미지에서 X_i ^(s) 의 복셀(또는 픽셀) 위치 주위의 네이버들의 인덱스(예, 8개)이다.For all source points, the nearest points within the threshold may not be found, and in this case, the corresponding source points may be changed by Algorithm2. Points that are not changed become points searched for improvement, and as in [Equation 20], Z _i ^(s) is

We can minimize the cost function F without substituting where p _i is for Z _i ^(s)

As an identifier of , it has a value of 0 or 1. ΔZ _i,m is the initial difference before minimizing F Z _i ^(s) - Z _m ^(s) , R _i is the index of neighbors around the voxel (or pixel) position of X _i ^(s) in the image (e.g., 8).

[Equation 20]

If the Jacobian method for the minimization of F is applied, an updated expression such as [Equation 21] can be obtained. Here, λ is a coefficient (eg, 0.2) for balancing between two different terms, and |R _i | is the number of elements of R _i . Here, for Z _i ^(s) with pi equal to 0, the neighbor updated Z _m ^(s ) pulls Z _i ^(s) to Z _m ^(s) by ΔZ _i,m holding the initial relative difference.

[Equation 21]

The depth vector search step may be repeated a predetermined number of times (eg, 6 times) to obtain an updated depth value as in [Equation 21] (S220). Reducing this maximum iteration count is reasonable if the source fragment corresponds to an appropriate subset of the referenced portion. Like Io, the updated depth value in the depth vector search step is applied to the source point cloud of the pose vector search step (S230). In addition, such a method is preferably applied to a merged point cloud (merged point cloud) as described later.

<Multi-view point cloud registration>

Given a point cloud of multiple viewpoints (multi-view), using the method as described above for updating the pose vector and the depth value, iteratively merges adjacent point clouds to eventually merge all multi-view point clouds. merge Although there are advanced algorithms based on pose-graph optimization, we propose a simpler method suitable for evaluating the robustness of pairwise matching algorithms. Since errors in a single pair propagate to errors in subsequent pairs, robustness of pairwise matching is important in this method of the present invention.

로 변환되고, S는 S_r과

의 합성으로 계산된다. S를 새로운 참조 포인트 클라우드로 간주하면 새로운 인접 포인트 클라우드가 S에 병합된다. 각 뷰의 포인트 클라우드에 대하여, 업데이트된 상기 포즈 벡터를 이용하여 거리(수학식2,3 참조)가 가장 가까운 포인트 클라우드 S_r = S₀ 로부터 가장 먼 포인트 클라우드 S_r =S_f가 병합될 때까지 이와 같은 각 대응 쌍의 병합이 반복된다. Given a reference point cloud S _r = S ₀ and an adjacent point cloud S _s , S _s can be matched to S _r by the pose vector search method (Section 3.1) using the ICP algorithm of the present invention as described above. Using the estimated pose parameters to obtain the merged point cloud S, S _s is

, and S is S _r and

is calculated by the synthesis of Considering S as a new reference point cloud, a new adjacent point cloud is merged into S. For the point cloud of each view, using the updated pose vector, the point cloud S _r = S ₀ closest to the distance (refer to Equations 2 and 3) until the point cloud S r = S f furthest from the point cloud S _r = S _f is merged This merging of each corresponding pair is repeated.

(S_f가 S₀ 좌표계로 변환된 것)의 합성으로 설정되어있는 S와 함께 역순으로 절차를 시작한다. Although the sequential merging procedure cannot prevent error accumulation, the following can be handled to reduce errors. After one round of merging, S is the first S ₀ and

Start the procedure in the reverse order with S set to the synthesis of (S _f converted to S ₀ coordinate system).

Since the pairwise merging algorithm is not symmetrical, if the source and reference points are inverted, different solution results may be obtained. Reversing the order gives the algorithm a chance to explore to update the current pose parameters for the inverted input pair. If the current pose parameter is not correct, an inverted input can help deviate from the local minimum. If the single forward path from S ₀ to S _f is correct, then the pause parameter of S _f can be correct. In this case, merging the erroneous point cloud into S _f with the correct position and orientation can reduce the errors of other backward paths. However, if there is only one path to S _f , it is difficult to expect such error correction.

에 병합한다. 상기 깊이 벡터 탐색 프로세스를 기초로, 업데이트된 상기 깊이값을 이용함으로써, 거리(수학식2,3 참조)가 가장 먼 포인트 클라우드 S_f에서 시작하여 첫 번째 라운드에서 참조 포인트 클라우드 S₀으로 끝난다. 이 순서는 가장 먼 포인트 클라우드가 가장 오류가 많다는 가정을 기반으로 한다. 멀티-뷰 포인트 클라우드들 중 포인트 클라우드를 병합하는 두 번째 이후 라운드에서는 처리 순서가 교대로 반전된다. 오류의 크기에 따라 이 순서는 바뀔 수 있다. Algorithm 3 below summarizes the proposed multi-view point cloud merging algorithm. Without loss of generality, the 0th view is assumed as the reference view. ITER _max is set to 5 in the present invention. After merging the point clouds, the merging error can be further reduced by applying the depth vector search method (Section 3.2) as described above. In this step, we adjust the depth value of S _s to make S _s

merge into Based on the depth vector search process, by using the updated depth value, the distance (refer to Equations 2 and 3) starts from the farthest point cloud S _f and ends with the reference point cloud S ₀ in the first round. This order is based on the assumption that the furthest point cloud is the most error-prone. In the second and subsequent rounds of merging the point clouds among the multi-view point clouds, the processing order is alternately reversed. Depending on the size of the error, this order may change.

Algorithm 4 summarizes such a proposed algorithm. Here, ITER _max is set to 2 in the present invention.

<Synthetic Multi-View RGB-D Data Set>

First, a graphic model of a human model data set that varies according to poses was rendered, and a synthetic multi-view RGB-D data set was constructed and stored in a database such as memory. That is, male (model 0-199) and female (model 6800-6999) appearances were used and 20 mesh models in different poses from each appearance were regularly sampled, and the height of the mesh model was set to 1.7 m. It is assumed that RGB-D image data is collected by a system including two infrared (IR) stereo cameras and one IR projector in addition to the RGB color camera as shown in FIG. 2 and depth values are acquired. In this case, it is assumed that the mismatch error obtained by stereo matching to obtain the depth value follows the standard deviation σ _d =0.5 pixel 0-mean Gaussian distribution. The depth error approximately follows a zero-mean Gaussian distribution in which σ _z of [Equation 22], which is the standard deviation, varies according to the depth value Z.

[Equation 22]

where f _d and B _d are the focal length and baseline length of the stereo camera, and were set to 780 and 26 cm reflecting the actual parameters below. The width and height of the depth image were 1468 and 1228 pixels, and the image center coordinate vector was (734, 614).

2 illustrates (a) an RGB-D camera system comprising an RGB color camera, two infrared (IR) stereo cameras and one IR projector, for acquiring a depth image, (b) according to a depth value Z; This is an example of the standard deviation σ _z .

The noisy depth image is forward-warped in the color camera coordinate system by using the built-in and external camera parameters to form the RGB-D image. The focal length of the color camera is 900 pixels and the distance from the reference IR camera is 6.5 cm. A color image has the same width, height, and image center as a depth image. The actual depth image is obtained by rendering the mesh model directly into the color image coordinate system.

3 is an example of positioning of an RGB-D camera for multi-view image acquisition.

Referring to FIG. 3 , the RGB-D cameras may be arranged at intervals of 30 degrees on an ellipse to capture images at 12 positions, and may be arranged at a distance of 3 m on the semi-major axis and 1.5 m on the semi-short axis.

4 shows a sample RGB-D image acquired with an RGB-D camera. In FIG. 4 , the first and second rows represent the color image and depth image of the male model, and the third and fourth rows represent the color image and the depth image of the female model. Here, the first, second, third, and fourth columns are images of positions 0, 3, 6, and 9 of View of FIG. 3 .

4 , the depth image has missing data due to self-occlusions as observed in the actual depth image. In addition, it can be seen that the point resolution is uneven in the views due to different distances from the model.

은 본 발명의 알고리즘에 입력으로 사용되는 초기 부정확한 포즈 파라미터를 생성하기 위해 조정된다. 회전 벡터(행렬) R_i,gt를 조정하기 위해 [수학식23]과 같이 R_i,gt에 임의의 회전 축에 대한 회전 행렬 βQ를 미리 곱한 회전 벡터 R_i를 이용한다. 역시 [수학식24]와 같이 실질적인 병진 벡터(행렬) T_i,gt에 임의의 3D 벡터 βU를 더한 병진 벡터 T_i를 이용한다.RGB-D camera's gt, ground truth pose parameters

is adjusted to produce an initial incorrect pose parameter that is used as input to the algorithm of the present invention. In order to adjust the rotation vector (matrix) R _i,gt , a rotation vector R _i obtained by multiplying R _i,gt by a rotation matrix βQ for an arbitrary rotation axis in advance is used as in [Equation 23]. Also, as in [Equation 24], a translation vector T _i obtained by adding an arbitrary 3D vector βU to an actual translation vector (matrix) T _i,gt is used.

[Equation 23]

[Equation 24]

These adjustments of five different rotation and translation vectors were applied to each set of RGB-D images obtained from the mesh model, respectively, with a rotation angle of 2 to 10 degrees and a translational length of 5 cm to 25 cm. Therefore, each set is adjusted in 10 different ways in the experiment.

5 is an example of rendered images for comparing the conventional method and the merged point clouds of the present invention. In FIG. 5 , rows 1 and 3 represent merged point clouds, and rows 2 and 4 are enlarged images of a hand part.

As shown in Figure 5, the ICP algorithm using the conventional point-to-plane cost function is completely different from the ground truth pose, and the visual quality is poor. The conventional color 6D GICP and the present pose search algorithm (Ours (pose)) achieve image registration visually similar to the Ground Truth pose. Furthermore, it was confirmed that the depth search algorithm (Ours (pose+depth)) of the present invention can improve visual quality and reduce registration errors by moving noisy points toward the surface and visually closer matching.

6 is a view for explaining an example of an implementation method of the image processing system for processing the reconstruction of a point cloud according to an embodiment of the present invention.

The image processing system for processing the reconstruction of the point cloud according to an embodiment of the present invention may be formed of hardware, software, or a combination thereof. For example, the image processing system of the present invention may be implemented in the form of a computing system 1000 as shown in FIG. 6 having at least one processor for performing the above functions/steps/processes or a server on the Internet.

The computing system 1000 includes at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage 1600 connected through a bus 1200 , and a network. An interface 1700 may be included. The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or the storage 1600 . The memory 1300 and the storage 1600 may include various types of volatile or nonvolatile storage media. For example, the memory 1300 may include a read only memory (ROM) 1310 and a random access memory (RAM) 1320 .

In addition, the network interface 1700 is a communication module such as a modem that supports wired Internet communication in user terminals such as smartphones, notebook PCs, and desktop PCs, wireless Internet communication such as WiFi and WiBro, and mobile communication such as WCDMA and LTE, or short-distance communication. It may include a communication module such as a modem supporting communication of a wireless communication method (eg, Bluetooth, Zigbee, Wi-Fi, etc.).

Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by the processor 1100 , or a combination of the two. A software module may be a storage/recording medium (i.e., memory 1300 and/or memory 1300) readable by a device such as a computer, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM. Alternatively, it may reside in storage 1600 . An exemplary storage medium is coupled to the processor 1100 , the processor 1100 capable of reading information (code) from, and writing information (code) to, the storage medium. Alternatively, the storage medium may be integrated with the processor 1100 . The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

As described above, the image processing system for processing the reconstruction of a point cloud according to the present invention uses an iterative K-nearest point algorithm, but in order to strike a balance between calculation speed and accuracy, the point is K-nearest. In reconstructing a 3D image such as a person, by allowing the point to match and assigning the matching probability between the source/reference point only to K pairs, the pose search is performed in the 6D vector space by combining the 3D coordinates and the color of the point. Points in the point cloud are naturally matched to the reference point cloud by K-nearest points, and the cost can be minimized. In the present invention, the cost function integrating all important results is defined as the weighted sum of squares for point-to-point and point-to-plane distances to improve stability and accuracy. In the present invention, 1) it is possible to minimize the cost by finding a pose parameter that minimizes the cost through the ICP algorithm, and 2) it is assumed that a point cloud is obtained from the RGB-D image, and there is an error in the depth value measured with the RGB-D camera. Considering the existence of

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and all technical ideas with equivalent or equivalent modifications to the claims as well as the claims to be described later are included in the scope of the present invention. should be interpreted as

Claims (8)

In the iterative nearest point search method for matching a source point cloud with a reference point cloud in an image processing system,
A process of updating the depth value so that the cost based on the Mahalanobis distance is minimized according to the iterative K-nearest point algorithm with respect to the depth value of the source point obtained from the 3D camera with the pose vector as a constant. a depth vector search step performed a plurality of times;
applying the updated depth value to a source point cloud;
According to the iterative K-nearest point algorithm, which assigns a matching probability to a predetermined number of neighbors by using a pose vector including a rotation vector and a translation vector between the source point cloud and the reference point cloud, the pose vector calculating a cost associated with and based on the Mahalanobis distance between the source point and the reference point; and
generating a point cloud in which the source point cloud is updated by the pose vector by repeating the process of updating the pose vector using the rotation increment of the rotation vector and the translation increment of the translation vector a plurality of times;
Iterative nearest point search method including According to claim 1,
The cost is an iterative nearest point search method corresponding to a value added after applying a weight to the square of the point-to-point or point-to-plane distance between the source point and the reference point. According to claim 1,
The search for the pose vector according to the iterative K-nearest point algorithm is performed in a six-dimensional vector space in which a three-dimensional coordinate vector and a three-dimensional color vector of each of the source and reference points are combined. According to claim 1,
In the iteration process, calculating the increment at which the gradient of the cost becomes zero, and performing the iteration process within a predetermined maximum number of iterations until the magnitude of the rotational increment and the translational increment becomes less than or equal to a threshold,
According to the rotation increment βR of the rotation vector R and the translation increment βT of the translation vector T, in the iteration process

,

An iterative nearest point search method in which the pose vector is updated according to delete According to claim 1,
further comprising merging multi-view point clouds;
In the merging, according to the updated pose vector, iterative nearest point search method for merging from the nearest point cloud among the multi-view point clouds in the order of the furthest point cloud. According to claim 1,
further comprising merging multi-view point clouds;
In the merging, according to the updated depth value, merging from the furthest point cloud to the nearest point cloud order among the multi-view point clouds, and then alternately reversing the processing order of the furthest and the closest ones in the round Iterative nearest point search method. In an image processing system, a computer-readable code is recorded for processing iterative nearest point search for matching a source point cloud with a reference point cloud, the recording medium comprising:
A process of updating the depth value so that the cost based on the Mahalanobis distance is minimized according to the iterative K-nearest point algorithm with respect to the depth value of the source point obtained from the 3D camera with the pose vector as a constant. a depth vector search function performed multiple times;
a function of applying the updated depth value to a source point cloud;
According to the iterative K-nearest-nearest-point algorithm, which assigns a matching probability to a predetermined number of neighbors by using a pose vector including a rotation vector and a translation vector between the source point cloud and the reference point cloud, the pose vector a function associated with and calculating a cost based on the Mahalanobis distance between the source point and the reference point; and
A function of generating a point cloud in which the source point cloud is updated by the pose vector by repeating the process of updating the pose vector using the rotation increment of the rotation vector and the translation increment of the translation vector a plurality of times
recording medium for implementing

Images