CN101833786B - Method and system for capturing and rebuilding three-dimensional model - Google Patents

Abstract

The present invention proposes a method for capturing and reconstructing a static three-dimensional model, which includes the following steps: collecting images of moving objects in an annular field; obtaining a visible shell model; Depth point clouds of various viewing angles are obtained under constraint conditions; the obtained depth point clouds of various viewing angles are fused to obtain a static three-dimensional model. The invention can ensure the accuracy and completeness of the reconstructed shape of the static three-dimensional model. In addition, the invention also proposes a dynamic three-dimensional model capture and reconstruction method.

Description

Method and system for capturing and reconstructing three-dimensional models

technical field

The invention relates to the technical field of computer video processing, in particular to a method and system for capturing and reconstructing a three-dimensional model.

Background technique

For the 3D reconstruction of dynamic scenes, many works regard it as a simple accumulation of static scene reconstruction problems in the time dimension, that is, do not use time information to assist scene reconstruction, and perform static 3D modeling for each frame alone. However, this method has high complexity and large storage capacity, cannot guarantee the topological consistency of the model between frames, and is prone to jitter. In addition, using the above method for 3D modeling cannot effectively analyze the motion of the non-rigid body model, nor can it obtain the model at any time through interpolation in the time domain. Through research on this type of problem, the prior art proposes a joint solution to the 3D scene flow and the reconstruction method of the geometric model. It is further proposed to use the variational method to unify the reconstruction of dynamic scene geometry and motion. However, since geometric reconstruction and motion reconstruction are performed iteratively, that is, the model reconstruction at the next moment is deduced by using the geometric reconstruction at a certain moment as the initial value of motion reconstruction. Therefore, the joint spatial-temporal reconstruction efficiency of this method is still not high, and the actual effect is not satisfactory.

Therefore, in order to avoid the difficulty and general quality of spatio-temporal joint reconstruction, another video-based dynamic 3D reconstruction method uses the static 3D reconstruction result of the initial frame as the scene representation, and then applies the 3D motion tracking algorithm to solve the motion of the 3D object. , and apply a suitable deformation algorithm to drive the motion of the static model, so as to obtain dynamic 3D reconstruction results. Currently, video-based 3D motion tracking can be divided into two categories: 3D motion tracking with markers and 3D motion tracking without markers. Among them, the 3D motion tracking method with markers is accurate, but requires the person to be captured to wear tight clothing with markers, which limits the capture of shape and texture. The markerless 3D motion tracking method overcomes the above defects. A marker-free approach to 3D motion tracking captures the motion of a human body wearing more general clothing by combining a kinematic model and a clothing model, but this approach fails to capture the precise geometry of the moving object. Another marker-free 3D motion tracking method can capture both the motion of the object's skeleton and shape, but this method is still not effective for 3D motion tracking because some local surfaces do not change as they should over time. Furthermore, since the method only relies on contour information, it is very sensitive to contour errors. Although this label-free method has increased flexibility, it is difficult to achieve the same accuracy as the labeled method. In addition, most 3D motion tracking methods need to help capture motion by extracting a kinematic skeleton, which can only track rigid body motion, so this method often requires other scanning techniques to assist in capturing time-varying shapes. Finally, none of the above methods can track the motion of a person wearing arbitrary clothing.

In computer graphics, new methods of animation capture and design, animation editing, and deformation transfer have emerged in recent years. These methods no longer rely on kinematic skeletons and motion parameters, but are based on surface models and general shape deformation methods, so that deformations of rigid as well as non-rigid bodies can be captured. However, in all such motion capture and restoration methods based on multi-view video, the static 3D reconstruction of the initial frame needs to be performed by a laser scanner. Although laser scanners can obtain high-precision 3D reconstruction results, laser scanners are expensive, time-consuming and laborious, and people must remain completely still during scanning. Moreover, for the convenience of follow-up work, people usually stand with fists in both hands, and the multi-view video shot also makes movements with fists in both hands. In addition, using the reconstruction result of the laser scanner as the initial scene representation method, some surface features on the model during scanning, such as the folds of clothes, etc., are always preserved in the restored dynamic 3D sequence.

Contents of the invention

The purpose of the present invention is to at least solve the above-mentioned technical defects. The present invention proposes a method and system for capturing and reconstructing static and dynamic three-dimensional models.

In order to achieve the above object, the present invention proposes a method for capturing and reconstructing a static three-dimensional model, which includes the following steps: collecting images of moving objects in an annular field; obtaining a visual shell model; The depth point cloud of each viewing angle is obtained by visual shell model and preset constraint conditions; the obtained depth point cloud of each viewing angle is fused to obtain a static three-dimensional model.

Another aspect of the present invention also proposes a static three-dimensional model capture and reconstruction system, including: a plurality of cameras around the ring field, used to collect images of moving objects in the ring field; a static three-dimensional model reconstruction device for Obtain the visible shell model, and obtain the depth point cloud of each viewing angle according to the image of each viewing angle, the visible shell model and the preset constraints, and fuse the obtained depth point cloud of each viewing angle to obtain a static three-dimensional model .

Another aspect of the present invention also proposes a dynamic three-dimensional model capture and reconstruction method, including the following steps: obtaining a static three-dimensional model; converting the surface model of the static three-dimensional model into a volume model, and using it as a default scene representation for motion tracking ; Obtain the initial three-dimensional movement of the model vertices at the next moment; select the precise vertices from the obtained vertices according to the predetermined space-time constraints as the position constraints of the body deformation; drive the Laplacian body deformation framework to update the dynamics according to the position constraints 3D model.

Another aspect of the present invention also proposes a dynamic three-dimensional model capture and reconstruction system, including: a plurality of cameras around the ring field, used to collect images of moving objects in the ring field; a static three-dimensional model acquisition device for Obtain a static 3D model; a dynamic 3D model reconstruction device is used to convert the surface model of the static 3D model into a volume model, and use it as the default scene representation for motion tracking, and obtain the initial 3D motion of the model vertices at the next moment, and according to Predetermined space-time constraints select accurate vertices from the obtained vertices as position constraints for body deformation, and drive the Laplacian body deformation framework to update the dynamic 3D model according to the position constraints.

Through the present invention, the accuracy and integrity of the reconstructed shape of the static 3D model can be guaranteed. In addition, the present invention designs a new 3D motion estimation method based on sparse representation theory, and a deformation optimization framework based on the volume model, so high-quality dynamic reconstruction results can be obtained . In addition, the invention can be independent of 3D scanners and optical markers, so the cost is not high, and it can track the movement of people wearing any clothing.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

1 is a flowchart of a method for capturing and reconstructing a static three-dimensional model according to an embodiment of the present invention;

Fig. 2 is the scene to be collected surrounded by 20 cameras distributed in a ring according to the embodiment of the present invention;

3 is a flowchart of a method for capturing and reconstructing a dynamic three-dimensional model according to an embodiment of the present invention;

FIG. 4 is a schematic block diagram of the entire dynamic three-dimensional reconstruction method according to an embodiment of the present invention; and

Fig. 5 is the result of the dynamic three-dimensional model obtained by using the method of the present invention for two long time series.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

The embodiment of the present invention respectively proposes the capture and reconstruction methods of the static 3D model and the dynamic 3D model, but it should be noted that the capture and reconstruction of the dynamic 3D model can be based on the static 3D model obtained by the present invention, or can be obtained through other methods. Based on the static three-dimensional model obtained by other means, such as the existing three-dimensional scanner, etc., these should be included in the protection scope of the present invention.

As shown in Figure 1, it is a flowchart of a method for capturing and reconstructing a static three-dimensional model according to an embodiment of the present invention, including the following steps:

Step S101 , image acquisition of moving objects in the ring field. For example, 20 cameras are set in the ring field, and the frame rate of each camera is 30 frames per second, and each group of cameras is controlled to collect moving objects in the ring field. Of course, those skilled in the art can also select more cameras to obtain more viewing angle images, and of course reduce the number of cameras, all of which should be included in the protection scope of the present invention. As an example of the present invention, as shown in FIG. 2 , 20 cameras according to the embodiment of the present invention are distributed in a ring around the scene to be captured. Among them, Ci represents the i-th camera. The resolution of the image captured by the camera is 1024×768. The collected characters stand in the center of the circle.

Step S102, acquiring a visual hull model (visual hull) at the initial moment.

Step S103, obtaining the depth point cloud of each angle of view according to the image of each angle of view, the visible shell model and the preset constraints. Specifically, it may include:

Step S201 , intersecting the obtained visible shell model with the images of each viewing angle to obtain the visible point cloud of each viewing angle.

Step S202, project the visible point cloud of each viewing angle to the viewing angle image to obtain the initial depth point cloud estimation, that is, d=(a, b, 1) is the offset along the epipolar line.

Step S203, obtaining an accurate depth point cloud according to the initial depth point cloud estimation and the preset constraints, wherein the preset constraints include one of epipolar geometry constraints, brightness constraints, gradient constraints, and smoothness constraints one or more species. In a preferred embodiment of the present invention, the above four constraints can be included at the same time, and the accurate depth point cloud can be obtained by the following formula:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; \&Integral;</span>}_{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; \&Psi;</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; dx</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{<span\; class="notranslate">\; \&Integral;</span>}_{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}\mathrm{<span\; class="notranslate">\; \&Psi;</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; dx</span>}<span\; class="notranslate">\; ,</span>$

为空间梯度算子；β(x)为遮挡图，对于非遮挡区域的像素为1，否则为0。考虑到模型假设中野值的影响，我们采用鲁棒的惩罚函数

来产生一个全变差正则化，其中ε为一个很小的值(在实验中设为0.001)该公式包括了四个约束：对极几何约束(x_b+d＝x+w)，亮度约束(I(x_b+d)＝I(x))，梯度约束

和平滑性约束

Among them, x:=(x, y, c) defines a pixel position (x, y) in the image of the reference view angle c, and its brightness is defined as I(x); x _b :=(x _b , y _b , c ) is the antipole on the angle of view c+1, w is the offset of the corresponding point of x on the angle of view c+1;

is the spatial gradient operator; β(x) is the occlusion map, which is 1 for the pixels in the non-occlusion area, and 0 otherwise. Considering the influence of the outliers in the model assumptions, we use a robust penalty function

To produce a total variation regularization, where ε is a very small value (set to 0.001 in the experiment). This formula includes four constraints: epipolar geometric constraints (x _b +d = x + w), brightness constraints (I(x _b +d)=I(x)), gradient constraints

and smoothness constraints

Step S104, merging the obtained depth point clouds of each viewing angle to obtain a static three-dimensional model. Specifically, the following steps may be included:

Step S301, merging the depth point clouds of each viewing angle and removing some outliers through contour constraints.

In step S302, the complete surface model is reconstructed by the moving cube method to obtain a static three-dimensional model.

The invention can ensure the accuracy and integrity of the reconstructed shape of the static three-dimensional model, and the accuracy and integrity of the static three-dimensional model are the basis of the reconstruction of the dynamic three-dimensional model.

As shown in Figure 3, it is a flowchart of a method for capturing and reconstructing a dynamic three-dimensional model according to an embodiment of the present invention, including the following steps:

Step S401, converting the surface model of the static 3D model into a volume model, and using it as a default scene representation for motion tracking.

Step S402, acquiring the initial three-dimensional motion of the vertices of the model at the next moment. Specifically, the following steps may be included:

Step S501, calculating the optical flow of images of each view angle at the next moment.

Step S502 , calculating the scene flow of the visible point from the optical flow of each viewing angle and the optical flow of the adjacent viewing angle, and assigning a relatively large value, such as 10000, to the scene flow of the invisible point.

Step S503, taking the obtained scene flow of each view angle as a column, constructing a matrix M∈i ^m×n , where m is the number of surface vertices.

Step S504, based on the sparse representation theory, a new matrix X is obtained by solving the following low-rank matrix restoration problem.

minimize||X|| _*

为采样操作子，定义为Among them, X is an unknown variable, Ω is a subset of [m]×[n] complete element set ([n] is defined as the sequence {1, K, n}),

is a sampling operator defined as

从而得到下一时刻的顶点位置

Step S505, take the average value of each row in the matrix X as the motion of the vertex corresponding to the row

So as to get the vertex position at the next moment

Step S403, selecting accurate vertices from the obtained vertices according to predetermined space-time constraints as position constraints for volume deformation. In the embodiment of the present invention, the predetermined space-time constraints include:

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; sp</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; sil</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; tmp</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; smth</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; .</span>$

Among them, P _sil ⁿ (v′ _i ) is the contour error of the estimated value, if the pixel projected by v′ _i onto the image of camera n at the next moment is within the contour, then the function value is 1, otherwise it is 0; v( i) is the visible camera set of v _i ; N _v is the number of visible cameras; P _z ⁿ (p(v _i ), p(v′ _i ) calculates the distance between the projected positions of v _i and v′ _i on the image of camera n ZNCC correlation; N _s is the number of direct neighbors of vertex v _i .

In step S404, the dynamic three-dimensional model is updated by driving the Laplacian volume deformation framework according to the position constraints. Specifically, including:

Step S601, establish the following Laplacian volume deformation linear system, for each v′ _i , there is

$\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, R _i and R _j are rotation matrices and are initialized as identity matrix.

Step S602, define covariance matrix

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; ,</span>$

于是

如果det(R_i)≤0，则改变U_i中对应于最小奇异值的列的符号；The singular value decomposition of C _i has

then

If det(R _i )≤0, change the sign of the column in U _i corresponding to the smallest singular value;

Step S603, if the contour error is smaller than a given threshold, update the model, otherwise return to step S601.

As a preferred embodiment of the present invention, the above static 3D model and dynamic 3D model capture and reconstruction methods of the present invention can be used simultaneously, as shown in Figure 4, which is a schematic block diagram of the entire dynamic 3D reconstruction method of the embodiment of the present invention.

As shown in Fig. 5, it is the result of the dynamic three-dimensional model obtained by using the proposed inventive method on two long time series. Among them, the first picture of each sequence result is the general picture that puts the models of each time together, and the subsequent pictures are the modeling results of each time.

The embodiment of the present invention also proposes a system for capturing and reconstructing a static three-dimensional model. The system includes: multiple cameras surrounding an annular field and a static three-dimensional model reconstruction device. Among them, a plurality of cameras surrounding the ring field are used to collect images of moving objects in the ring field; the static 3D model reconstruction device is used to obtain a visible shell model, and according to each viewing angle image, the visible shell model and preset Obtain the depth point cloud of each angle of view according to the constraint conditions, and fuse the acquired depth point cloud of each angle of view to obtain a static three-dimensional model. For the specific working process of the static 3D model reconstruction device, reference may be made to the above embodiment of the static 3D model capture and reconstruction method, which will not be repeated here.

In addition, the embodiment of the present invention also proposes a dynamic three-dimensional model capture and reconstruction system,

It includes: a plurality of cameras surrounding the ring field, a static three-dimensional model acquisition device and a dynamic three-dimensional model reconstruction device. Among them, multiple cameras around the ring field are used to collect images of moving objects in the ring field; the static 3D model acquisition device is used to obtain a static 3D model; the dynamic 3D model reconstruction device is used to convert the surface model of the static 3D model It is a body model, and it is used as the default scene representation of motion tracking, and the initial three-dimensional motion of the model vertices at the next moment is obtained, and the precise vertices are selected from the obtained vertices according to the predetermined space-time constraints as the position constraints of the body deformation , to update the dynamic 3D model by driving the Laplacian body deformation framework according to the position constraints. For the specific working process of the static and dynamic 3D model reconstruction device, reference may be made to the above embodiment of the static and dynamic 3D model capture and reconstruction method, which will not be repeated here.

Through the present invention, the accuracy and integrity of the reconstructed shape of the static 3D model can be guaranteed. In addition, the present invention designs a new 3D motion estimation method based on sparse representation theory, and a deformation optimization framework based on the volume model, so high-quality dynamic reconstruction results can be obtained . In addition, the invention can be independent of 3D scanners and optical markers, so the cost is not high, and it can track the movement of people wearing any clothing.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (6)

1. A capture and reconstruction method of a static three-dimensional model, characterized in that, comprising the following steps: Image acquisition of moving objects in the ring field; Get the visual shell model; Obtain the depth point cloud of each angle of view according to the image of each angle of view, the visible shell model and the preset constraint conditions; Fusing the obtained depth point clouds of each viewing angle to obtain a static three-dimensional model, in, The obtaining the depth point cloud of each angle of view according to each angle of view image, the visible shell model and preset constraints includes: Intersect each viewing angle image with the obtained visible shell model to obtain a visible point cloud of each viewing angle; Project the visible point cloud of each viewing angle to the viewing angle image to obtain the initial depth point cloud estimation; Obtaining an accurate depth point cloud according to the initial depth point cloud estimation and the preset constraints; The preset constraints include one or more of epipolar geometric constraints, brightness constraints, gradient constraints, and smoothness constraints; Accurate depth point cloud is obtained by the following formula:

Among them, d = (a, b, 1) is the initial depth point cloud estimation, x : = (x, y, c) is a pixel position (x, y) in the reference view c image, and I(x) is the The brightness of the pixel position; x _b :=(x _b , y _b , c) is the antipole on the viewing angle c+1;

is the spatial gradient operator; β(x) is the occlusion map,

is a robust penalty function. 2. the capture of static three-dimensional model as claimed in claim 1 and reconstruction method, it is characterized in that, described to obtain the depth point cloud of described each angle of view that fusion obtains static three-dimensional model comprising: Fusion of the depth point cloud of each viewing angle and remove outliers through contour constraints; and The complete surface model is reconstructed by the moving cube method to obtain a static 3D model. 3. The capture and reconstruction method of a static three-dimensional model as claimed in claim 1 or 2, further comprising: A dynamic three-dimensional model is constructed according to the static three-dimensional model. 4. The capture and reconstruction method of a static three-dimensional model as claimed in claim 3, wherein said building a dynamic three-dimensional model according to said static three-dimensional model comprises: Convert the surface model of a static 3D model to a volume model and use it as the default scene representation for motion tracking; Obtain the initial three-dimensional motion of the model vertices at the next moment; Select accurate vertices from the obtained vertices according to predetermined space-time constraints as positional constraints for volume deformation; Drive the Laplacian body deformation framework to update the dynamic 3D model according to the position constraints, in, The acquisition of the initial three-dimensional movement of the model vertices at the next moment includes: Calculate the optical flow of images of each viewing angle at the next moment; Calculate the scene flow of visible points from the optical flow of each viewing angle and the optical flow of adjacent viewing angles, and assign a relatively large value to the scene flow of invisible points; Taking the obtained scene flow of each viewing angle as a column, construct a matrix M∈i ^m×n , where m is the number of surface vertices; Obtain matrix X based on sparse representation theory; Take the average value of each row in the matrix X as the motion of the vertex corresponding to the row

So as to get the vertex position at the next moment

in, The said obtaining matrix X based on sparse representation theory includes: The new matrix X is obtained by solving the following low-rank matrix recovery problem, minimize||X|| _*

Among them, X is an unknown variable, Ω is a subset of [m]×[n] complete element set, [n] is defined as the sequence {1,...,n},

is a sampling operator defined as

The predetermined space-time constraints include:

in,

is the contour error of the estimated value, if v′ _i is projected onto the pixel of the camera n image at the next moment is within the contour, then the function value is 1, otherwise it is 0;

is the visible camera set of v _i ; N _v is the number of visible cameras; Calculate the ZNCC correlation between the projected positions of v _i and v′ _i on the image of camera n; N _s is the number of direct neighbors of vertex v _i . 5. The capturing and rebuilding method of static three-dimensional model as claimed in claim 4, is characterized in that, described according to position constraints, drives the Laplace body deformation framework to update dynamic three-dimensional model and comprises: Initialize the rotation matrix as the identity matrix, R _i =R _j =I; Optimization using Laplacian volume deformation; Obtain new rotation matrices R _i and R _j ; It is judged whether the contour error is smaller than a predetermined value, if it is smaller than the predetermined value, the dynamic three-dimensional model is updated, and if it is not smaller than the predetermined value, the Laplacian body deformation is continued to be used for optimization. the 6. A capture and reconstruction system for a static three-dimensional model, characterized in that it comprises: a plurality of cameras around the ring field for image acquisition of moving objects in the ring field; and A static three-dimensional model reconstruction device, configured to obtain a visible shell model, and obtain depth point clouds of each viewing angle according to images of each viewing angle, the visible shell model, and preset constraints, and to obtain the depth of each viewing angle The point cloud is fused to obtain a static 3D model, in, The static 3D model reconstruction device intersects the obtained visible shell model with each viewing angle image to obtain the visible point cloud of each viewing angle, and then projects the visible point cloud of each viewing angle to the viewing angle image to obtain the initial depth point cloud Estimating, and obtaining an accurate depth point cloud according to the initial depth point cloud estimation and the preset constraints; The preset constraints include one or more of epipolar geometric constraints, brightness constraints, gradient constraints, and smoothness constraints; The static 3D model reconstruction device obtains an accurate depth point cloud through the following formula:

Among them, d = (a, b, 1) is the initial depth point cloud estimation, x : = (x, y, c) is a pixel position (x, y) in the reference view c image, and I(x) is the The brightness of the pixel position; x _b :=(x _b , y _b , c) is the antipole on the viewing angle c+1;

is the spatial gradient operator; β(x) is the occlusion map,

is a robust penalty function.

Images