JP5561786B2 - Three-dimensional shape model high accuracy method and program - Google Patents

Description

  The present invention relates to a method and program for restoring a subject three-dimensional model with high accuracy from an image obtained by photographing a subject and an image obtained by photographing only a background.

  The image quality of the free viewpoint video greatly depends on the accuracy of the three-dimensional shape model generated from the multi-view video when considering a model-based synthesis method. There is a visual volume intersection method as a typical method for constructing a three-dimensional shape model (three-dimensional voxel data) of a subject based on a multi-viewpoint image (Non-Patent Document 1). However, this method has a fundamental problem that the concave area of the object cannot be restored.

  Various methods such as space carving proposed in Non-Patent Document 2 have been proposed with respect to the above-described principle problem. Japanese Patent Application No. 2009-195334 proposes a technique for improving the accuracy of each camera image based on stereo matching.

  As a study on high accuracy of Visual Hull, not only Photo Consistency but also a method of defining an energy function considering the stability of the Visual Hull surface shape and correcting the shape by an optimization framework has been proposed. Non-Patent Document 3 proposes a shaping method based on stereo matching. The method defines an evaluation function by a linear combination of an external force for pushing the distance value from the camera to the subject into the Visual Hull inside by Photo Consistency and an internal force for maintaining the local shape feature of the Visual Hull surface. The surface shape of the subject is determined by obtaining the distance value to be minimized. Compared to a method that focuses on a single voxel such as Space carving, the result is that unnatural irregularities are less likely to occur on the surface, and a smooth shape can be restored.

  On the other hand, a method for shaping Visual Hull by graph cut in consideration of the adjacency relationship in the voxel space has been proposed. In Non-Patent Document 4, for voxels near the Visual Hull surface, energy based on Photo Consistency of voxels alone and energy function based on adjacency relationship between voxels is defined, and smooth shape is restored by applying graph cut It shows what you can do. Furthermore, by providing a Core area that assumes that the subject is surely present inside the Visual Hull, and limiting the application range of the graph cut between the Visual Hull surface and the Core area, the subject area may be accidentally shaved. It has succeeded in reducing.

Masahiro Toyoura, Masatsugu Iiyama, Takuya Tsuji, Kakudo Kou, Tetsuhiko Mino, “Acquisition of three-dimensional shape from time series silhouette including defect and over-extraction”, IEICE Technical Report, PRMU2007-168, Vol .107, No.427, pp.69-74, 2008-1 Kutulakos et al. "ATheory of Shape by Space Carving" International Journal of Computer Vision 2000 Ninohiro Hiyama et al. “Reconstruction method of 3D shape constrained by local shape features and its real-time video display” Journal of the Institute of Image Information and Television Engineers, 61, 4, pp. 471-481 (2007) Hisatomi et al. `` Methodof 3D reconstruction using graph cuts, and its application to preserving intangible cultural heritage '' IEEE conference on ICCV, pp. 923-930 (2009).

  However, since most of the methods such as the proposed method described in Non-Patent Document 2 determine whether or not to remove by focusing only on Photo Consistency of a single voxel, the restoration accuracy of Visual Hall finally obtained is There is a problem that it largely depends on the texture state of the subject. In Japanese Patent Application No. 2009-195334, the problem that the restoration accuracy is lowered due to the influence of the matching error is significant.

  In the method of Non-Patent Document 3, since the final restoration result largely depends on the search accuracy of stereo matching, there remains a problem that sufficient restoration accuracy cannot be obtained in a region where the texture change of the subject is small. .

  In the method of Non-Patent Document 4, since the energy function based on the adjacency relationship between voxels is simply defined by the average of the energy values of a single voxel, the adjacency relationship in the voxel space can be sufficiently considered. It can not be said. In addition to the fact that the application of the graph cut is limited to one time, there is no process for evaluating the application result, so there is a high possibility that the background region is included in the finally restored three-dimensional shape model it is conceivable that.

  Based on the above problems, the present invention fully considers the continuity in the voxel space and, based on the three-dimensional shape model of the shaping process, determines the image quality of the free viewpoint image generated from the real camera viewpoint. It is an object of the present invention to provide a Visual Hull high-accuracy method and program in consideration of the texture state of a three-dimensional shape model by introducing a process for evaluation.

3-dimensional shape model accuracy enhancement method according to the present invention for achieving the above object, a method for restoring a three-dimensional shape model of the object from the multi-view camera image, the volume intersection viewed from the object silhouette image of each imaging camera as input Visual Hull or the shaped Visual Hull restored by law, a calculation step of calculating the likelihood for an object ness of voxels on the surface of the Visual Hull, the based on the likelihood for said object ness Based on the comparison between the first shaping step for shaping the Visual Hull, the Visual Hull shaped in the first shaping step, and the last Visual Hull input in the calculation step, the shaping of the Visual Hull is converged. and whether or not the decision was, Vis it is determined that shaping al Hull has converged until a Visual Hull shaped by the first shaping step as the input of the calculation step, a first applying repeatedly said first shaping step and the calculation step The three-dimensional shape model acquired from the Visual Hull that is determined to have converged in the convergence step and the first convergence step or the shaped three-dimensional shape model is input, and the texture state of the three-dimensional shape model is input. An evaluation step for evaluation, a second shaping step for shaping the three-dimensional shape model based on the evaluation of the texture state of the three-dimensional shape model, and the three-dimensional shape model shaped in the second shaping step And 3D shape model shaping based on the comparison with the previous 3D shape model input in the evaluation step. It is determined whether or not the bundle, it is determined that the shaping of a three-dimensional shape model has converged until a three-dimensional shape model that is shaped by the second shaping step as the input of said evaluation step, and the evaluation step A second convergence step that repeatedly applies the second shaping step.

  Further, the calculating step projects each voxel existing on the surface of the Visual Hull onto each photographing camera viewpoint, calculates a variance of projection pixel values between the photographing cameras, and normalizes the variance to thereby make the object likeness. It is also preferable to calculate the likelihood for.

  In the first shaping step, an energy function is defined that considers the adjacent relationship between the voxels in the three-dimensional voxel space, and each voxel is defined as a subject area or a background area in a framework that minimizes the energy function. It is also preferable to perform shaping by deciding the subject area by assigning it and removing the background area as an unnecessary part.

In the first convergence step, the Visual Hull shaped in the first shaping step is compared with the Visual Hull just input as input in the calculation step, and the voxel inside the voxel designated as the surface is cut. It is also preferable to determine that the shaping has converged when it is not.

In addition, the evaluation step, in the imaging camera viewpoint, by extracting a difference image between the free viewpoint image generated from the 3-dimensional shape model and the camera image, evaluating the texture state of the three-dimensional shape model It is also preferable to do .

  In the second shaping step, an energy function that considers the adjacent relationship between the pixels of the difference image is defined at each photographing camera viewpoint, and each pixel is applied to a subject area in a framework that minimizes the energy function. Alternatively, it is determined which one of the background areas is to be assigned, the light rays of the pixels existing in the shaping candidates determined to be the background area are searched, and the intersection with the three-dimensional shape model is removed as an unnecessary part, thereby It is also preferable to shape the dimensional shape model.

The second convergence step is to compare the three-dimensional shape model immediately before was input in the second of said evaluation step and the shaped three-dimensional shape model by shaping step, the number of voxels is than one constant to be scraped It is also preferable to determine that the shaping has converged .

In order to achieve the above object, a program according to the present invention uses a computer for restoring a three-dimensional shape model of a subject from multi-viewpoint camera images, and a visual hull restored from the subject silhouette images of each imaging camera by a visual volume intersection method. or as input the shaped Visual Hull, first performs a calculation means for calculating the likelihood for an object ness of voxels on the surface of the Visual Hull, the original to shaping of the Visual Hull a likelihood for said object ness It is determined whether or not the Visual Hull shaping has converged based on a comparison between the first shaping means and the Visual Hull shaped by the first shaping means and the last Visual Hull input by the calculation means. , convergence is shaping of the Visual Hull Is determined that until the Visual Hull shaped by the first shaping means as an input of said calculation means, a first converging means for repeatedly applying the said calculating means first shaping means, said first The evaluation means for evaluating the texture state of the three-dimensional shape model, using as input the three-dimensional shape model acquired from the Visual Hull that has been determined that the shaping has converged by the convergence means, or the shaped three-dimensional shape model ; a second shaping means based on the evaluation of the texture state of the three-dimensional shape model performs shaping of the three-dimensional shape model, and input in the second three-dimensional geometric model and the evaluation means being shaped by the shaping means Based on the comparison with the immediately preceding three-dimensional shape model, it is determined whether or not the shaping of the three-dimensional shape model has converged, and it is determined that the shaping of the three-dimensional shape model has converged. Re until the three-dimensional shape model that is shaped by the second shaping means as an input of the evaluation unit, to serve as a second converging means for repeatedly applying said second shaping means and the evaluation means, 3 Restore the dimensional shape model.

  According to the present invention, it is possible to achieve high accuracy of Visual Hull, and it is possible to improve the quality of a free viewpoint video generated based on the finally restored three-dimensional shape.

2 shows a flowchart according to the invention. It shows that the vicinity of the surface of the Visual Hull in the voxel space is projected to each viewpoint. An example of a multi-viewpoint camera image is shown. The original three-dimensional shape model is shown. Indicates the input Visual Hull. The Visual Hull that has been shaped when Step 4 is completed is shown. The final shaped three-dimensional shape model is shown.

  The best mode for carrying out the present invention will be described in detail below with reference to the drawings. The proposed method is characterized by shaping in consideration of continuity in the voxel space and shaping in consideration of the image quality of the free viewpoint image at each viewpoint. A flowchart according to the present invention is shown in FIG. A three-dimensional shape model that has been finally shaped by inputting a multi-view camera image for one frame of time, a camera parameter of each viewpoint, and a Visual Hull restored by the visual volume intersection method from the subject silhouette image of each camera. Is output and the process ends. Hereinafter, description will be given based on this flowchart.

Step 1: The object-likeness of the voxel near the Visual Hull surface is calculated. As shown in FIG. 2, the vicinity of the Visual Hull surface in the voxel space is projected to each camera viewpoint. Viewpoint i (i = 1, ..., N) projected on the coordinates in the camera image and v _i, the projection pixel value at this point, the table as a multidimensional vector x (v _i) in a particular color space Is done. An example of the color space is an RGB space. In order to calculate the likelihood related to the object likeness, the average and variance of the projected pixel values at the camera viewpoints are calculated.
The mean vector u (v _i ) is
Is calculated by Here, # n _{vis (v i)} represents the number of occlusion without viewpoint (viewpoint voxels in the vicinity of the surface is visible), sigma is performed in perspective without occlusion. In the case of the RGB space, the average vector u (v _i ) has three components, u _r (v _i ), u _g (v _i ), and u _b (v _i ).
The variance vector σ (v _i ) is
Is calculated by Similarly, in the case of RGB space, the dispersion vector σ ² (v _i ) has three components, σ ² _r (v _i ), σ ² _g (v _i ), and σ ² _b (v _i ).

By normalizing this variance, the likelihood related to object-likeness is obtained. Note that normalization is obtained by dividing σ ² (v _i ) by the maximum value of σ ² (v _i ) (i = 1,..., N) and setting the maximum value to 1.0. [Sigma] ² appearing in the following expression represents the variance after normalization.

Step 2: Define an energy function in the voxel space.
The energy function is
Defined by Where _{v = (v 1, ...,} v i, ..., v N) , the coordinates in the camera image represents _{v i,} lambda _v is a positive constant, _{a v,} each voxel in the voxel space Is assigned to either the background area or the subject area by 0 or 1, respectively.

U (v; a _v ) is a data term that depends only on the likelihood value of each voxel alone, and the energy value when allocating to the region specified by a _v is given by the following equation.
Here, Σ takes the sum for the viewpoint i (i = 1,..., N), and th _{v (c)} is a constant threshold value. The above formula shows the formula in the case of RGB space. In the above formula, for example, when th _{v (c)} = 1.2, when σ _c ² = 0.5, th _{v (c)} −σ _c ² = 0.7, σ _c ² = 0. Since it is 5, the energy value is lower when a = 1, that is, when assigned to the subject area. When σ _c ² = 0.7, th _{v (c)} −σ _c ² = 0.5 and σ _c ² = 0.7, so that a = 0, that is, the energy allocated to the background region The value becomes lower. In other words, when the variance is small, the energy value is lower when assigned to the subject area.

On the other hand, V (v; a _v ) is a smoothing term, and is calculated by the following equation based on the difference in likelihood values between adjacent voxels.
Here, dis (i, j) represents the Euclidean distance between the voxels, the kappa _v is a positive constant. The constant κ _v is calculated as follows using the operator <·> of expected values for all combinations N _v of adjacent voxels.

Step 3: Perform Visual Hull shaping based on energy minimization. 0 or 1 is assigned to the pixel value so that the energy value E (v; a _v ) is minimized. As a result, Visual Hull is shaped. For example, Graph-cut algorithm is used to minimize the energy value. Here, a _v = 0, that is, voxels assigned to the background area are removed as unnecessary portions.

  Step 4: Determine convergence. If the Visual Hull obtained in Step 3 is compared with the Visual Hull input in Step 1, the voxels inside the voxels near the surface are not shaved, or the number of shaved voxels falls within a certain threshold. Convergence judgment is done by somehow. When the convergence is not sufficient, Step 1 to Step 3 are repeated with Visual Hull obtained in Step 3 as input of Step 1.

  Step 5: At each camera viewpoint, a free viewpoint image is generated based on the three-dimensional shape model acquired from the Visual Hull obtained in Step 3, and a difference image between the free viewpoint image and the captured image is calculated. In the following steps, the three-dimensional shape model is shaped by evaluating the texture state of the three-dimensional shape model.

Step 6: Define an energy function in the difference image at each camera viewpoint. At each photographing camera viewpoint, an energy function that considers the adjacent relationship between pixels of the difference image is defined as follows.
Defined by Here, p represents each pixel of the difference image, λ _p is a positive constant, and _ap represents 0 or 1 to assign each pixel of the difference image to either the background region or the subject region. ing.

U (v; a _p ) is a data term that depends only on the likelihood value of each pixel of the difference image, and the energy value when allocating to the region specified by a _p is given by the following equation.
Here, Σ is the sum for each pixel of the difference image, x _c (p _i ) is the pixel value (x _r , x _g , or x _b ) at pixel p _i , and th _{p (c)} is , A certain threshold. The above formula shows the formula in the case of RGB space.

On the other hand, V (v; a _p ) is a smoothing term, and is calculated by the following equation based on the difference in pixel values between adjacent pixels.
Here, dis _p (i, j) represents the Euclidean distance between the pixels, κ _p is a positive constant, and the vector x (p _i ) is the pixel value vector (x _r , x _g ) at the pixel p _i. , X _b ). Is a constant kappa _p using operator of an expected distance for all combinations N _v between adjacent pixels <-> is calculated as follows.

Step 7: Identify Visual Hull shaping candidates based on energy minimization. 0 or 1 is assigned to the pixel value so that the energy value E (v; a _p ) is minimized. Thereby, the Visual Hull shaping candidate is specified. For example, Graph-cut algorithm is used to minimize the energy value.

Step 8: Search the ray of each pixel included in the Visual Hull shaping candidate and remove the intersection with the three-dimensional shape model. The above steps 5 to 7 are performed on the camera image of each viewpoint i (i = 1,..., N). a _p = 0, that is, search for the ray of the pixel assigned to the background area, and remove the intersection with the three-dimensional shape model as an unnecessary part. This is performed for all camera images, and unnecessary voxels are removed.

  Step 9: Perform convergence determination. The three-dimensional shape model obtained in step 8 is compared with the input three-dimensional shape model in step 5. If the inner voxel in the vicinity of the surface is not shaved, or the number of shaved voxels is within a certain threshold. Convergence is determined by whether or not When the convergence is not sufficient, Steps 5 to 8 are repeated using the three-dimensional shape model obtained in Step 8 as the input of Step 5.

  Next, the processing results of the present invention are shown by experimental results. In the experiment, the accuracy of the three-dimensional shape obtained as a result of applying the method of the present invention to the Visual Hull restored by the visual volume intersection method is evaluated for a multi-viewpoint image including a subject including a concave region. As experimental data, an image obtained by projecting the CG model onto 23 viewpoints and camera parameters (center projection matrix) of each viewpoint were used.

  FIG. 3 shows an example of a multi-viewpoint camera image. FIG. 3 a shows a camera image from the viewpoint 02, and FIG. 3 b shows a camera image from the viewpoint 06. FIG. 4 shows the original three-dimensional shape model. FIG. 5 shows the input Visual Hull. FIG. 6 shows the Visual Hull that has been shaped when Step 4 is completed. FIG. 7 shows the finally shaped three-dimensional shape model.

  Comparing FIG. 4 and FIG. 5, it can be seen that Visual Hull created only by the visual volume intersection method cannot restore the concave area of the object. Comparing FIG. 4 and FIG. 6, it can be seen that the concave region of the object is restored by the processing of steps 1 to 4 of the present invention. Comparing FIG. 6 and FIG. 7, it can be seen that the concave region of the object can be reproduced more accurately by the processing of steps 5 to 9 of the present invention.

The above experimental results are shown quantitatively. Table 1 shows the Precision / Recall / F values for the original 3D shape model. Here, “Recall” indicates the ratio of voxels that are accidentally cut, and “Precision” indicates the ratio that the voxels that should be originally cut are not cut. The F value is an index representing the accuracy of the three-dimensional shape model calculated based on the Recall and the Precision.

  According to Table 1, the three-dimensional shape model finally shaped shows the best F value, and the concave region of the object is restored by executing steps 1 to 9 of the present invention. I understand.

  As described above, according to the present invention, the concave region of the object can be restored, the Visual Hull can be improved in accuracy, and the free viewpoint video generated based on the finally restored three-dimensional shape. Can be improved in image quality.

  Moreover, all the embodiments described above are illustrative of the present invention and are not intended to limit the present invention, and the present invention can be implemented in other various modifications and changes. Therefore, the scope of the present invention is defined only by the claims and their equivalents.

Claims (8)

A method for restoring a three-dimensional shape model of a subject from a multi-viewpoint camera image,
A calculation step for calculating a likelihood related to the object likeness of a voxel existing on the surface of the Visual Hull, using the Visual Hull restored by the visual volume intersection method or the shaped Visual Hull from the subject silhouette image of each imaging camera as an input ;
A first shaping step of based on shaping of the Visual Hull a likelihood for said object ness,
Based on the comparison between the Visual Hull shaped in the first shaping step and the previous Visual Hull input in the calculation step, it is determined whether or not the Visual Hull shaping has converged, and the Visual Hull shaping is completed. until it is determined converged with the Visual Hull shaped by the first shaping step as the input of the calculation step, a first converging step of repeatedly applying the said calculating step first shaping step,
An evaluation step for evaluating a texture state of the three-dimensional shape model by using the three-dimensional shape model acquired from the Visual Hull determined to have converged in the first convergence step or the shaped three-dimensional shape model as input. When,
A second shaping step of shaping of the three-dimensional shape model based on the evaluation of the texture state of the three-dimensional shape model,
It is determined whether or not the shaping of the 3D shape model has converged based on a comparison between the 3D shape model shaped in the second shaping step and the immediately preceding 3D shape model input in the evaluation step. , it is determined that the shaping of a three-dimensional shape model has converged until a three-dimensional shape model that is shaped by the second shaping step as the input of said evaluation step, repeat the second shaping step and the evaluation step A second convergence step to apply;
A method for restoring a three-dimensional shape model characterized by comprising: The calculating step includes:
Projecting each voxel present on the surface of the Visual Hull onto each photographing camera viewpoint,
Calculate the variance of the projected pixel values between the shooting cameras,
The method according to claim 1, wherein the likelihood related to the object-likeness is calculated by normalizing the variance. The first shaping step includes
Define an energy function that considers the adjacency relationship between each voxel in the 3D voxel space,
In the framework of minimizing the energy function, the subject area is determined by assigning each voxel to either the subject area or the background area, and shaping is performed by removing the background area as an unnecessary part. The method according to claim 1 or 2. The first convergence step includes:
The Visual Hull shaped in the first shaping step is compared with the Visual Hull just before input in the calculation step ,
The method according to claim 3, wherein the shaping is determined to have converged when a voxel inside the voxel designated as the surface is not cut. The evaluation step includes
In each photographic camera viewpoint, by extracting a difference image between the free viewpoint image generated from the 3-dimensional shape model and the camera image, claims and evaluating the texture state of the three-dimensional shape model Item 5. The method according to any one of Items 1 to 4. The second shaping step includes
At each shooting camera viewpoint, define an energy function that considers the adjacent relationship between the pixels of the difference image,
In a framework that minimizes the energy function, determine whether to assign each pixel to a subject area or a background area,
Searching for a ray of a pixel existing in a shaping candidate determined to be a background region, and removing the intersection with the three-dimensional shape model as an unnecessary part, thereby shaping the three-dimensional shape model The method of claim 5. The second convergence step includes
Comparing the three-dimensional shape model shaped in the second shaping step with the previous three-dimensional shape model input in the evaluation step ;
The method of claim 6, shaping the case number of voxels to be scraped or less one constant and judging to have converged. A computer for restoring a 3D shape model of a subject from multi-viewpoint camera images,
A calculation means for receiving a Visual Hull restored by a visual volume intersection method from a subject silhouette image of each imaging camera or a shaped Visual Hull, and calculating a likelihood related to the object likeness of a voxel existing on the surface of the Visual Hull;
A first shaping means for performing based on shaping of the Visual Hull a likelihood for said object ness,
Based on the comparison between the Visual Hull shaped by the first shaping means and the Visual Hull just before input by the calculation means, it is determined whether or not the Visual Hull shaping has converged, and the Visual Hull shaping is completed. until it is determined converged with, as an input of said calculating means the shaped Visual Hull by the first shaping means, a first converging means for repeatedly applying said first shaping means and the calculating means,
Evaluation means for evaluating the texture state of the three-dimensional shape model by using as input the three-dimensional shape model acquired from the Visual Hull that has been determined that the shaping has converged by the first convergence means or the shaped three-dimensional shape model When,
A second shaping means for performing a shaping of the three-dimensional shape model based on the evaluation of the texture state of the three-dimensional shape model,
It is determined whether or not the shaping of the 3D shape model has converged based on a comparison between the 3D shape model shaped by the second shaping unit and the immediately preceding 3D shape model input by the evaluation unit. , until it is determined that the shaping of the three-dimensional shape model has converged, the three-dimensional shape model that is shaped by the second shaping means as an input of the evaluation unit, the said evaluation means second shaping means A second convergence means to apply repeatedly;
And a program that restores a three-dimensional shape model.