WO2019219012A1

WO2019219012A1 - Three-dimensional reconstruction method and device uniting rigid motion and non-rigid deformation

Info

Publication number: WO2019219012A1
Application number: PCT/CN2019/086889
Authority: WO
Inventors: 刘烨斌; 戴琼海; 徐枫; 方璐
Original assignee: 清华大学; 清华大学深圳研究生院
Priority date: 2018-05-15
Filing date: 2019-05-14
Publication date: 2019-11-21
Also published as: CN108711185A; CN108711185B

Abstract

A three-dimensional reconstruction method and device uniting a rigid motion and a non-rigid deformation. The method comprises: performing a depth camera-based photography on a target object to obtain a single depth image (S101); performing three-dimensional framework extraction on a depth point cloud by means of a three-dimensional framework extraction algorithm (S102); obtaining a matching point pair between a three-dimensional point cloud and vertexes of a reconstruction model; establishing an energy function according to the matching point pair and three-dimensional framework information, solving a non-rigid motion position conversion parameter of each vertex on the reconstruction model, and optimizing an object framework parameter (S104); performing GPU optimal solution on the energy function to obtain the non-rigid deformation of each surface vertex, and deforming the previous frame of reconstruction three-dimensional model according to the solving result, so that the deformed model is aligned with the current frame of three-dimensional point cloud (S105); and obtaining the current frame of updated model to enter iteration of the next frame. The method can effectively improve the real-time performance, robustness, and accuracy of reconstruction, is high in expandability, and simple and easy to be implemented.

Description

Three-dimensional reconstruction method and device for joint rigid motion and non-rigid deformation

Cross-reference to related applications

The present application claims the priority of the Chinese Patent Application No. 201810460091.5, filed on May 15, 2018, the entire disclosure of which is incorporated herein by reference.

Technical field

The invention relates to the technical field of computer vision and computer graphics, in particular to a three-dimensional reconstruction method and device for joint rigid motion and non-rigid deformation.

Background technique

Dynamic object 3D reconstruction is a key issue in the field of computer graphics and computer vision. High-quality dynamic object 3D models, such as human body, animal, human face, human hand, etc., have broad application prospects and important application value in the fields of film and television entertainment, sports games, virtual reality and so on. However, the acquisition of high-quality 3D models usually relies on expensive laser scanners or multi-camera array systems. Although the accuracy is high, there are also some shortcomings: First, the object is required to remain absolutely still during the scanning process. Movement will lead to obvious errors in the scanning results. Second, the fraud is expensive and difficult to spread to the daily lives of ordinary people, often applied to large companies or national statistical departments. Third, the speed is slow, and it often takes at least 10 minutes to several hours to reconstruct a 3D model. The cost of reconstructing a dynamic model sequence is greater.

From a technical point of view, the existing reconstruction method concentrates on solving the rigid motion information of the object first, obtaining the approximation of the object, and reconstructing the non-rigid surface motion information. However, this reconstruction method requires obtaining a three-dimensional model of the key frame of the object in advance. On the other hand, although the existing frame-by-frame dynamic fusion surface reconstruction method can realize dynamic three-dimensional reconstruction without template, the robustness of tracking reconstruction is low only by using the non-rigid surface deformation method.

Summary of the invention

The present invention aims to solve at least one of the technical problems in the related art to some extent.

Therefore, an object of the present invention is to propose a three-dimensional reconstruction method combining joint rigid motion and non-rigid deformation, which can effectively improve the real-time, robustness and accuracy of reconstruction, and has strong scalability and is easy to implement.

Another object of the present invention is to provide a three-dimensional reconstruction apparatus that combines rigid motion and non-rigid deformation.

In order to achieve the above object, an embodiment of the present invention provides a three-dimensional reconstruction method combining joint rigid motion and non-rigid deformation, including the following steps: performing depth camera-based photographing on a target object to obtain a single depth image; The skeleton extraction algorithm performs three-dimensional skeleton extraction on the depth point cloud; transforms the single depth image into a three-dimensional point cloud, and acquires a matching point pair between the three-dimensional point cloud and the reconstructed model vertex; and according to the matching point pair The three-dimensional skeleton information establishes an energy function, and solves a non-rigid motion position transformation parameter of each vertex on the reconstruction model and optimizes an object skeleton parameter; performing a GPU (Graphics Processing Unit) optimization solution on the energy function to Obtaining a non-rigid deformation of each surface vertex, deforming the reconstructed three-dimensional model of the previous frame according to the solution result, so that the deformation model is aligned with the current frame three-dimensional point cloud; and the current frame three-dimensional point cloud and the deformation model are merged to obtain The updated model of the current frame to enter the iteration of the next frame.

The three-dimensional reconstruction method of the joint rigid motion and the non-rigid deformation according to the embodiment of the present invention combines the three-dimensional information of the dynamic object surface frame by frame by real-time non-rigid alignment method, in order to achieve robust tracking, realizing the three-dimensional frame without the first frame Robust real-time dynamic 3D reconstruction under template conditions can effectively improve the real-time, robustness and accuracy of reconstruction, and it is scalable and easy to implement.

In addition, the three-dimensional reconstruction method of the joint rigid motion and the non-rigid deformation according to the above-described embodiments of the present invention may further have the following additional technical features:

Further, in an embodiment of the present invention, the converting the single depth image into a three-dimensional point cloud further comprises: projecting the single depth image into the three-dimensional space by using an internal parameter matrix of the depth camera, Generating the three-dimensional point cloud, wherein the depth map projection formula is:

Where u, v are pixel coordinates, and d(u, v) is a depth value at a pixel (u, v) position on the depth image,

Is the internal reference matrix of the depth camera.

Further, in an embodiment of the invention, the energy function is:

E _t =λ _n E _n +λ _s E _s +λ _j E _j +λ _g E _g +λ _b E _b ,

Where E _t is the total energy term, E _{n is the} non-rigid surface deformation constraint term, E _s is the rigid skeleton motion constraint term, E _j is the rigid skeleton recognition constraint term, E _g is the local rigid motion constraint term, λ _n , λ _s , λ _j and λ _g are weight coefficients corresponding to respective constraint terms, respectively.

Further, in an embodiment of the present invention, wherein

Where u _i represents the position coordinate of the three-dimensional point cloud in the same matching point pair, c _i represents the i-th element in the set of matching point pairs, and the non-rigid surface deformation constraint item

with

Representing the vertex coordinates of the model and its normal direction driven by the non-rigid deformation, respectively, in the rigid skeleton motion constraint

with

Representing the coordinates of the model vertex and its normal direction driven by the motion of the object skeleton,

with

Representing the model vertex coordinates driven by the target rigid motion and the motion-driven model vertex coordinates obtained by the three-dimensional skeleton estimation, where i represents the i-th vertex on the model.

Represents a collection of adjacent vertices around the ith vertex on the model,

with

Representing the driving effects of known non-rigid motion on the surface vertices v _i and v _j of the model, respectively.

with

Represents the positional transformation effect of the non-rigid motion acting on v _i and v _j simultaneously on v _j .

Further, in one embodiment of the invention, the model vertices are driven according to the surface non-rigid deformation and the object rigid skeleton motion, wherein the calculation formula is:

among them,

a deformation matrix acting on the vertex v _i , including two parts of rotation and translation;

Is the rotating portion of the deformation matrix;

a set of bones that have a driving effect on the vertex v _i ; α _{i, j} is the weight of the driving action of the jth bone on the i-th model vertex, indicating the strength of the bone driving the vertex; T _bj is the first The motion deformation matrix of j bones themselves,

Is the rotating part of the deformation matrix.

In order to achieve the above object, another embodiment of the present invention provides a three-dimensional reconstruction apparatus combining rigid motion and non-rigid deformation, comprising: a photographing module for performing depth camera-based photographing on a target object to obtain a single depth An image extraction module is configured to perform three-dimensional skeleton extraction on the depth point cloud by using a three-dimensional skeleton extraction algorithm; and the matching module converts the single depth image into a three-dimensional point cloud, and acquires between the three-dimensional point cloud and the reconstructed model vertex a matching point pair; a solution module, configured to establish an energy function according to the pair of matching points and the three-dimensional skeleton information, and solve a non-rigid motion position transformation parameter of each vertex on the reconstruction model and optimize an object skeleton parameter; And performing GPU optimization on the energy function to obtain a non-rigid deformation of each surface vertex, and deforming the reconstructed three-dimensional model of the previous frame according to the solution result, so that the deformation model is aligned with the current frame three-dimensional point cloud. a model update module for fusing a current frame three-dimensional point cloud and the deformation model to obtain Updated model frame to the next iteration into the frame.

The joint rigid motion and non-rigid deformation three-dimensional reconstruction device of the embodiment of the invention combines the three-dimensional information of the dynamic object surface frame by frame by real-time non-rigid alignment method, in order to achieve robust tracking, realizing the three-dimensional frame without the first frame Robust real-time dynamic 3D reconstruction under template conditions can effectively improve the real-time, robustness and accuracy of reconstruction, and it is scalable and easy to implement.

In addition, the combined rigid motion and non-rigid deformation three-dimensional reconstruction apparatus according to the above-described embodiments of the present invention may further have the following additional technical features:

Further, in an embodiment of the present invention, the matching module is further configured to project the single depth image into a three-dimensional space by using an internal parameter matrix of a depth camera to generate the three-dimensional point cloud, wherein the depth map The projection formula is:

Is the internal reference matrix of the depth camera.

Further, in an embodiment of the invention, the energy function is:

E _t =λ _n E _n +λ _s E _s +λ _j E _j +λ _g E _g +λ _b E _b ,

Further, in an embodiment of the present invention, wherein

with

among them,

Is the rotating portion of the deformation matrix;

a set of bones that have a driving effect on the vertex v _i ; α _{i, j} is the weight of the driving action of the jth bone on the i-th model vertex, indicating the strength of the bone driving the vertex; T _bj is the first The motion deformation matrix of j skeletons themselves, rot(T _bj ) is the rotation part of the deformation matrix.

The additional aspects and advantages of the invention will be set forth in part in the description which follows.

DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from

1 is a flow chart of a three-dimensional reconstruction method for joint rigid motion and non-rigid deformation according to an embodiment of the present invention;

2 is a flow chart of a three-dimensional reconstruction method combining joint rigid motion and non-rigid deformation according to an embodiment of the present invention;

3 is a schematic structural view of a three-dimensional reconstruction apparatus combining rigid motion and non-rigid deformation according to an embodiment of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are intended to be illustrative of the invention and are not to be construed as limiting.

A three-dimensional reconstruction method and apparatus for joint rigid motion and non-rigid deformation according to an embodiment of the present invention will be described below with reference to the accompanying drawings. First, a three-dimensional reconstruction method for joint rigid motion and non-rigid deformation according to an embodiment of the present invention will be described with reference to the accompanying drawings. .

1 is a flow chart of a three-dimensional reconstruction method for joint rigid motion and non-rigid deformation according to an embodiment of the present invention.

As shown in FIG. 1, the three-dimensional reconstruction method of the joint rigid motion and the non-rigid deformation includes the following steps:

In step S101, depth camera-based photographing is performed on the target object to obtain a single depth image.

It can be understood that, as shown in FIG. 2, the real-time video frame rate depth point cloud acquisition is performed, and the dynamic object is subjected to depth map shooting to obtain a frame-by-frame depth point cloud. Specifically, a dynamic object is photographed using a depth camera to obtain a continuous single depth image sequence. Transform a single depth image into a set of 3D point clouds.

In step S102, the depth point cloud is subjected to three-dimensional skeleton extraction by a three-dimensional skeleton extraction algorithm.

It can be understood that, as shown in FIG. 2, the 3D skeleton extraction is performed by the skeleton recognition algorithm, and the three-dimensional rigid skeleton information of the current frame of the object is extracted by the existing skeleton recognition algorithm. For example, the object 3D skeleton stealing is implemented by KinectSDK.

In step S103, the single depth image is transformed into a three-dimensional point cloud, and a matching point pair between the three-dimensional point cloud and the reconstructed model vertex is acquired.

It can be understood that, as shown in FIG. 2, a three-dimensional model and a cloud matching point pair are established, and a matching point pair between the current frame three-dimensional point cloud and the reconstructed model vertex is calculated.

Further, in an embodiment of the present invention, converting the single depth image into the three-dimensional point cloud further includes: projecting the single depth image into the three-dimensional space by using an internal parameter matrix of the depth camera to generate a three-dimensional point cloud, wherein The depth map projection formula is:

The internal reference matrix for the depth camera.

It can be understood that the object is photographed by the depth camera to obtain a depth image, and the depth map is transformed into a set of three-dimensional point clouds. Based on the internal camera matrix of the depth camera calibration, the depth map is projected into the three-dimensional space to generate a set of three-dimensional point clouds. The depth map projection formula is:

For the depth camera internal reference matrix.

Specifically, the internal parameter matrix of the depth camera is acquired, and the depth map is projected into the three-dimensional space according to the internal reference matrix and transformed into a set of three-dimensional point clouds. Among them, the formula of the transformation is:

For the depth camera internal reference matrix. In terms of obtaining matching point pairs, the vertices of the three-dimensional model are projected onto the depth image using a camera projection formula to obtain matching point pairs.

among them,

Is the rotating portion of the deformation matrix;

In step S104, an energy function is established according to the matching point pair and the three-dimensional skeleton information, and the non-rigid motion position transformation parameters of each vertex on the reconstruction model are solved and the object skeleton parameters are optimized.

It can be understood that the energy function is established, and an energy function is established according to the current frame matching point pair information and the extracted three-dimensional rigid skeleton information of the current frame.

For example, a single depth camera, such as a Microsoft Kinect depth camera, an IphoneX depth camera, an Obi immersion depth camera, etc., captures a dynamic scene, and obtains real-time depth image data (video frame rate, 20 frames/second or more) transmitted to the computer. On the computer, the three-dimensional geometric information of the dynamic object is calculated by the computer in real time, the three-dimensional model of the object at the same frame rate is reconstructed, and the three-dimensional skeleton information of the object is output.

Further, in one embodiment of the invention, the energy function is:

E _t =λ _n E _n +λ _s E _s +λ _j E _j +λ _g E _g +λ _b E _b ,

Further, in an embodiment of the present invention, wherein

Where u _i represents the position coordinate of the three-dimensional point cloud in the same matching point pair, c _i represents the i-th element in the matching point pair set, and the non-rigid surface deformation constraint item

with

Representing the vertex coordinates of the model and its normal direction driven by non-rigid deformation, respectively, in the rigid skeleton motion constraint

with

Representing the vertex coordinates of the model driven by the target rigid motion and the motion-driven model vertex coordinates obtained by the three-dimensional skeleton estimation. In the local rigid motion constraint, i represents the i-th vertex on the model.

with

Specifically, the rigid motion constraint term E _s and the non-rigid motion constraint term E _n are simultaneously used to perform an optimal solution of the object motion, and a single depth image is used to perform the object rigid skeleton constraint term E _j to constrain the solved rigid motion.

(1) The surface non-rigid constraint E _n ensures that the model after the non-rigid deformation is aligned with the three-dimensional point cloud obtained from the depth map as much as possible;

with

Representing the vertex coordinates of the model and its normal direction after being driven by non-rigid deformation,

with

Represents the vertex coordinates of the model and its normal direction driven by the object skeleton motion.

(2) The rigid skeleton motion constraint E _s ensures that the rigid deformation model driven by the skeleton motion is aligned with the three-dimensional point cloud obtained from the depth map as much as possible.

(3) In the rigid skeleton motion and the non-rigid deformation consistency constraint E _b ,

with

The vertex coordinates of the model driven by the target rigid motion are respectively consistent with the motion-driven model vertex coordinates obtained by the three-dimensional skeleton estimation. The constraint is used to ensure that the calculated rigid skeleton and the identified skeleton are as much as possible. Consistently, through single-frame skeleton recognition, the error accumulated in the dynamic tracking process can be prevented from being accumulated and cannot be recovered, so that the non-rigid motion calculated by the final solution can be guaranteed to conform to the object skeleton dynamics model, and fully and from the depth map. Get the 3D point cloud alignment.

(4) In the local rigid motion constraint term E _g , i represents the i-th vertex on the model,

with

It represents the positional transformation effect of the non-rigid motion acting on v _i and v _j simultaneously on v _j , that is, to ensure that the non-rigid driving effects of adjacent vertices on the model are as uniform as possible.

Is a robust penalty function,

with

Representing the driving effect of the rigid skeleton motion on the surface vertices v _i and v _j of the model respectively. When the two adjacent vertices of the model surface are driven by the rigid skeleton motion, the value of the robust penalty function is small, when the two phases are When the neighboring vertex is less affected by the skeleton motion driving effect, the robust penalty function value is larger. Through the robust penalty function, the model can be subjected to local rigid constrained motion while ensuring a large amplitude of reasonable non-rigid motion. Can be well solved, so that the model is more accurately aligned with the 3D point cloud.

In step S105, GPU optimization is performed on the energy function to obtain a non-rigid deformation of each surface vertex, and the reconstructed three-dimensional model of the previous frame is deformed according to the solution result, so that the deformation model is aligned with the current frame three-dimensional point cloud. .

It can be understood that, as shown in FIG. 2, the GPU optimization of the energy function is performed to solve the non-rigid motion position transformation parameters of each vertex on the reconstructed model, and the three-dimensional rigid motion information of the object is optimized, and the previous frame is reconstructed according to the solution result. The model is deformed to align with the current frame 3D point cloud.

Specifically, the energy function is solved, and the reconstructed model is aligned with the three-dimensional point cloud according to the solution result. The non-rigid motion position transformation parameters and the object skeleton motion parameters of each vertex on the reconstruction model are solved. The information obtained by the final solution is the transformation matrix of each 3D model vertex and the object skeleton motion parameters, that is, the individual transformation matrix of each bone. In order to achieve the requirement of fast linear solution, the method of the embodiment of the present invention approximates the deformation equation by using an exponential mapping method:

among them,

For the cumulative transformation matrix of the model vertex v _i up to the previous frame, for the known amount,

a non-rigid deformation for each surface apex; I is a four-dimensional unit matrix;

among them,

make

That is, the model vertices after the previous frame transformation are transformed:

For each vertex, the unknown parameter that requires the solution is the six-dimensional transformation parameter x = (v ₁ , v ₂ , v ₃ , w _x , w _y , w _z ) ^T . The linearization of bone movement is the same as that of non-rigid motion.

In step S106, the current frame three-dimensional point cloud and the deformation model are merged to obtain an updated model of the current frame to enter an iteration of the next frame.

It can be understood that, as shown in FIG. 2, Poisson fusion is performed on the aligned model and the point cloud to obtain a relatively complete three-dimensional model of the new frame.

Specifically, the point cloud and the three-dimensional model are merged to obtain an updated model of the current frame. Update and complete the 3D model aligned with the depth point cloud, fuse the newly obtained depth information into the 3D model, update the surface vertex position of the 3D model or add new vertices to the 3D model to make it more consistent with the current depth image expression.

In summary, the core function of the embodiment of the present invention is to receive a depth image code stream in real time and calculate a three-dimensional model of each frame in real time. At the same time, the time-varying three-dimensional model of the dynamic object is calculated by using the large-scale rigid skeleton motion of the object and the small-scale surface non-rigid deformation information. The method of the embodiment of the invention is accurate and can realize high-precision reconstruction of the dynamic object in real time. Since the method is a real-time reconstruction method and only needs to provide a single depth camera input, the system has the advantages of simple equipment, convenient deployment and scalability, and the like. The required input information is very easy to acquire and a dynamic 3D model can be obtained in real time. The method is accurate, robust, simple and easy to operate, and runs at a real-time speed. It has broad application prospects and can be quickly implemented on hardware systems such as PC (personal computer) or workstations.

According to the three-dimensional reconstruction method of joint rigid motion and non-rigid deformation according to the embodiment of the present invention, the three-dimensional information of the dynamic object surface is merged frame by frame by real-time non-rigid alignment method, in order to achieve robust tracking, realizing the key without the first frame Robust real-time dynamic 3D reconstruction under frame 3D template conditions can effectively improve the real-time, robustness and accuracy of reconstruction, and it is scalable and easy to implement.

Next, a three-dimensional reconstruction apparatus combining joint rigid motion and non-rigid deformation according to an embodiment of the present invention will be described with reference to the accompanying drawings.

3 is a schematic view showing the structure of a three-dimensional reconstruction apparatus combining rigid motion and non-rigid deformation according to an embodiment of the present invention.

As shown in FIG. 3, the joint rigid motion and non-rigid deformation 3D reconstruction apparatus 10 includes a photographing module 100, an extraction module 200, a matching module 300, a solution module 400, a solution module 500, and a model update module 600.

The shooting module 100 is configured to perform depth camera-based shooting on the target object to obtain a single depth image. The extraction module 200 is configured to perform three-dimensional skeleton extraction on the depth point cloud by using a three-dimensional skeleton extraction algorithm. The matching module 300 transforms the single depth image into a three-dimensional point cloud, and acquires a matching point pair between the three-dimensional point cloud and the reconstructed model vertex. The solving module 400 is configured to establish an energy function according to the matching point pair and the three-dimensional skeleton information, and solve the non-rigid motion position transformation parameter of each vertex on the reconstruction model and optimize the object skeleton parameter. The solving module 500 is configured to perform GPU optimization on the energy function to obtain a non-rigid deformation of each surface vertex, and deform the reconstructed three-dimensional model of the previous frame according to the solution result, so that the deformation model is aligned with the current frame three-dimensional point cloud. . The model update module 600 is configured to fuse the current frame three-dimensional point cloud and the deformation model to obtain an updated model of the current frame to enter an iteration of the next frame. The device 10 of the embodiment of the invention can effectively improve the real-time, robustness and accuracy of the reconstruction, has strong scalability, and is simple and easy to implement.

Further, in an embodiment of the present invention, the matching module 300 is further configured to project a single depth image into the three-dimensional space by using an internal parameter matrix of the depth camera to generate a three-dimensional point cloud, wherein the depth map projection formula is:

The internal reference matrix for the depth camera.

Further, in one embodiment of the invention, the energy function is:

E _t =λ _n E _n +λ _s E _s +λ _j E _j +λ _g E _g +λ _b E _b ,

Further, in an embodiment of the present invention, wherein

with

among them,

Is the rotating portion of the deformation matrix;

It should be noted that the foregoing explanation of the embodiment of the three-dimensional reconstruction method for joint rigid motion and non-rigid deformation is also applicable to the joint rigid motion and non-rigid deformation three-dimensional reconstruction apparatus of the embodiment, and details are not described herein again.

According to the three-dimensional reconstruction device of the joint rigid motion and the non-rigid deformation according to the embodiment of the present invention, the three-dimensional information of the dynamic object surface is merged frame by frame by real-time non-rigid alignment method, in order to achieve robust tracking, the key in the first frame is realized. Robust real-time dynamic 3D reconstruction under frame 3D template conditions can effectively improve the real-time, robustness and accuracy of reconstruction, and it is scalable and easy to implement.

In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " After, "Left", "Right", "Vertical", "Horizontal", "Top", "Bottom", "Inside", "Outside", "Clockwise", "Counterclockwise", "Axial", The orientation or positional relationship of the "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is merely for convenience of description of the present invention and simplified description, and does not indicate or imply the indicated device or component. It must be constructed and operated in a particular orientation, and is not to be construed as limiting the invention.

Moreover, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

In the present invention, the terms "installation", "connected", "connected", "fixed" and the like shall be understood broadly, and may be either a fixed connection or a detachable connection, unless explicitly stated and defined otherwise. , or integrated; can be mechanical or electrical connection; can be directly connected, or indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction of two elements, unless otherwise specified Limited. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood on a case-by-case basis.

In the present invention, the first feature "on" or "under" the second feature may be a direct contact of the first and second features, or the first and second features may be indirectly through an intermediate medium, unless otherwise explicitly stated and defined. contact. Moreover, the first feature "above", "above" and "above" the second feature may be that the first feature is directly above or above the second feature, or merely that the first feature level is higher than the second feature. The first feature "below", "below" and "below" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely that the first feature level is less than the second feature.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the invention. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Although the embodiments of the present invention have been shown and described, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the invention. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

A three-dimensional reconstruction method combining joint rigid motion and non-rigid deformation, characterized in that it comprises the following steps:

Performing depth camera based shooting on the target object to obtain a single depth image;

Three-dimensional skeleton extraction of deep point clouds by three-dimensional skeleton extraction algorithm;

Converting the single depth image into a three-dimensional point cloud, and acquiring a matching point pair between the three-dimensional point cloud and the reconstructed model vertex;

Generating an energy function according to the pair of matching points and the three-dimensional skeleton information, and solving non-rigid motion position transformation parameters of each vertex on the reconstruction model and optimizing the object skeleton parameters;

Performing GPU optimization on the energy function to obtain a non-rigid deformation of each surface vertex, and deforming the reconstructed three-dimensional model of the previous frame according to the solution result, so that the deformation model is aligned with the current frame three-dimensional point cloud;

The current frame 3D point cloud is merged with the deformation model to obtain an updated model of the current frame to enter an iteration of the next frame.
The three-dimensional reconstruction method of the joint rigid motion and the non-rigid deformation according to claim 1, wherein the converting the single depth image into a three-dimensional point cloud further comprises:

The single depth image is projected into the three-dimensional space by an internal parameter matrix of the depth camera to generate the three-dimensional point cloud, wherein the depth map projection formula is:

Where u, v are pixel coordinates, and d(u, v) is a depth value at a pixel (u, v) position on the depth image,
Is the internal reference matrix of the depth camera.
The method of claim 3, wherein the energy function is:

E t =λ n E n +λ s E s +λ j E j +λ g E g +λ b E b ,

Where E t is the total energy term, E n is the non-rigid surface deformation constraint term, E s is the rigid skeleton motion constraint term, E j is the rigid skeleton recognition constraint term, E g is the local rigid motion constraint term, λ n , λ s , λ j and λ g are weight coefficients corresponding to respective constraint terms, respectively.
The method of three-dimensional reconstruction of joint rigid motion and non-rigid deformation according to claim 3, wherein

Where u i represents the position coordinate of the three-dimensional point cloud in the same matching point pair, c i represents the i-th element in the set of matching point pairs, and the non-rigid surface deformation constraint item
with
Representing the vertex coordinates of the model and its normal direction driven by the non-rigid deformation, respectively, in the rigid skeleton motion constraint
with
Representing the coordinates of the model vertex and its normal direction driven by the motion of the object skeleton,
with
Representing the model vertex coordinates driven by the target rigid motion and the motion-driven model vertex coordinates obtained by the three-dimensional skeleton estimation, where i represents the i-th vertex on the model.
Represents a collection of adjacent vertices around the ith vertex on the model,
with
Representing the driving effects of known non-rigid motion on the surface vertices v i and v j of the model, respectively.
with
Represents the positional transformation effect of the non-rigid motion acting on v i and v j simultaneously on v j .
A three-dimensional reconstruction method for joint rigid motion and non-rigid deformation according to any one of claims 1 to 4, characterized in that the model vertices are driven according to the surface non-rigid deformation and the object rigid skeleton motion, wherein the calculation formula is:

among them,
a deformation matrix acting on the vertex v i , including two parts of rotation and translation;
Is the rotating portion of the deformation matrix;
a set of bones that have a driving effect on the vertex v i ; α i, j is the weight of the driving action of the jth bone on the i-th model vertex, indicating the strength of the bone driving the vertex; T bj is the first The motion deformation matrix of j skeletons themselves, rot(T bj ) is the rotation part of the deformation matrix.
A three-dimensional reconstruction device combining rigid motion and non-rigid deformation, comprising:

a shooting module for performing depth camera based shooting on the target object to obtain a single depth image;

An extraction module for performing three-dimensional skeleton extraction on the depth point cloud by using a three-dimensional skeleton extraction algorithm;

a matching module, transforming the single depth image into a three-dimensional point cloud, and acquiring a matching point pair between the three-dimensional point cloud and the reconstructed model vertex;

a solution module, configured to establish an energy function according to the pair of matching points and the three-dimensional skeleton information, and solve a non-rigid motion position transformation parameter of each vertex on the reconstruction model and optimize an object skeleton parameter;

a solution module for performing GPU optimization on the energy function to obtain a non-rigid deformation of each surface vertex, and deforming the reconstructed three-dimensional model of the previous frame according to the solution result, so that the deformation model and the current frame three-dimensional point cloud Align; and

And a model updating module, configured to fuse the current frame three-dimensional point cloud and the deformation model to obtain an updated model of the current frame to enter an iteration of the next frame.
The combined rigid motion and non-rigid deformation three-dimensional reconstruction apparatus according to claim 6, wherein the matching module is further configured to project the single depth image into the three-dimensional space by using an internal parameter matrix of the depth camera, Generating the three-dimensional point cloud, wherein the depth map projection formula is:

Where u, v are pixel coordinates, and d(u, v) is a depth value at a pixel (u, v) position on the depth image,
Is the internal reference matrix of the depth camera.
The combined rigid motion and non-rigid deformation three-dimensional reconstruction apparatus according to claim 6, wherein the energy function is:

E t =λ n E n +λ s E s +λ j E j +λ g E g +λ b E b ,

Where E t is the total energy term, E n is the non-rigid surface deformation constraint term, E s is the rigid skeleton motion constraint term, E j is the rigid skeleton recognition constraint term, E g is the local rigid motion constraint term, λ n , λ s , λ j and λ g are weight coefficients corresponding to respective constraint terms, respectively.
The combined rigid motion and non-rigid deformation three-dimensional reconstruction apparatus according to claim 8, wherein

Where u i represents the position coordinate of the three-dimensional point cloud in the same matching point pair, c i represents the i-th element in the set of matching point pairs, and the non-rigid surface deformation constraint item
with
Representing the vertex coordinates of the model and its normal direction driven by the non-rigid deformation, respectively, in the rigid skeleton motion constraint
with
Representing the coordinates of the model vertex and its normal direction driven by the motion of the object skeleton,
with
Representing the model vertex coordinates driven by the target rigid motion and the motion-driven model vertex coordinates obtained by the three-dimensional skeleton estimation, where i represents the i-th vertex on the model.
Represents a collection of adjacent vertices around the ith vertex on the model,
with
Representing the driving effects of known non-rigid motion on the surface vertices v i and v j of the model, respectively.
with
Represents the positional transformation effect of the non-rigid motion acting on v i and v j simultaneously on v j .
The combined rigid motion and non-rigid deformation three-dimensional reconstruction apparatus according to any one of claims 6-9, wherein the model vertices are driven according to the surface non-rigid deformation and the object rigid skeleton motion, wherein the calculation formula is:

among them,
a deformation matrix acting on the vertex v i , including two parts of rotation and translation;
Is the rotating portion of the deformation matrix;
a set of bones that have a driving effect on the vertex v i ; α i, j is the weight of the driving action of the jth bone on the i-th model vertex, indicating the strength of the bone driving the vertex; T bj is the first The motion deformation matrix of j skeletons themselves, rot(T bj ) is the rotation part of the deformation matrix.