CN102446366B - Time-space jointed multi-view video interpolation and three-dimensional modeling method - Google Patents

Abstract

The invention belongs to the technical field of computer multimedia. In order to provide a simple and practical method for multi-view video interpolation and three-dimensional modeling, the technical solution adopted by the present invention is to combine space-time multi-view video interpolation and three-dimensional modeling method to group multi-camera arrays at intervals; The three-dimensional model of the scene specifically includes the following steps: 1) Interpolating uncollected frames between two frames; 2) Using a model-assisted weighting method to obtain an image of a perspective that has not been collected at this moment; 3) Calculating and extracting key points; 4 ) Use the shape context to describe the extracted key points, and solve it through the Hungarian method; 5) Obtain the final interpolation frame by solving the Poisson editing optimization problem; 6) Reconstruct the 3D model of the scene and render it. The invention is mainly applied to antenna design and manufacture.

Description

Spatio-temporal joint multi-view video interpolation and 3D modeling method

technical field

The invention belongs to the technical field of computer multimedia, and specifically relates to a time-space joint multi-view video interpolation and three-dimensional modeling method.

Background technique

For a long time, the acquisition, processing and communication of single-channel video have made important breakthroughs in key technologies, and have become mature and widely used in broadcast television, Internet video, intelligent transportation and other fields. However, the traditional single-camera acquisition method cannot bring a sense of depth, a sense of three-dimensionality, and a comprehensive understanding of objects (variable viewing angles). Multi-channel video acquisition and reconstruction of scene objects based on a multi-camera system can achieve a full range of visual experience, and related research has become a research hotspot since the mid-1990s. Real-time acquisition and reconstruction of 3D scenes based on multi-camera systems has been widely used in the fields of free viewpoint video, virtual reality, immersive video conferencing, movie entertainment, stereoscopic video, and motion analysis. Many famous universities and research institutions in the world such as: Stanford, MIT, Carnegie Mellon, Columbia University, Mitsubishi Electronics, Microsoft Research Institute, Max Planck Institute for Information Research have built various multi-camera acquisition systems , for scene geometry capture, motion analysis, and stereoscopic production. At present, acquisition and reconstruction technologies based on multi-camera systems are difficult to obtain satisfactory reconstruction results due to problems in camera construction and synchronization, camera storage and transmission, high-dimensional data processing, and high-speed motion capture. Among them, to realize the capture of high-speed motion, one method is to use multiple high-speed cameras, but high-speed cameras are expensive and have limited storage capacity; Grouping, the same group of cameras sampling at the same time, and different groups of cameras interleaving sampling, so as to obtain sparsely sampled space-time information, and then achieve high frame rate reconstruction through interpolation methods. Stanford University (Wilburn B, Joshi N, Vaish V, et al. High-speed videography using a dense camera array. Proceedings of IEEE Conference on ComputerVision and Pattern Recognition, Washington, DC, USA, 2004.294-301.) through 52 frames The dense light field camera array with a rate of 30fps realizes high-speed scene reproduction from a single perspective. However, the result of this method is limited to a single perspective, and images from multiple perspectives at each moment cannot be obtained, and thus the 3D model at each moment cannot be reconstructed. The first inventor of the present invention once proposed a method (ZL200810103684.2) for modeling high-speed moving objects using a ring-shaped low frame rate camera array to achieve full-view 3D reconstruction, but this method only simply uses the visible shell model to calculate the intersection To perform interpolation and reconstruction, so the interpolation and reconstruction effects are very general and not robust. Although existing video interpolation and image fusion methods can be used to obtain uncaptured multi-view videos, the results will have blurred or non-smooth areas.

Contents of the invention

In order to overcome the deficiencies of the prior art and provide a simple and practical method for multi-view video interpolation and three-dimensional modeling, the technical solution adopted by the present invention is to combine space-time multi-view video interpolation and three-dimensional modeling method to group multi-camera arrays at intervals : There are n cameras with a frame rate of f frames per second, which are divided into m groups at even intervals, where n and m are positive integers, and n is an integer multiple of m; the same group of cameras can be synchronously collected to obtain n/m viewing angles at the same time Different groups of cameras interleaved at 1/(fm) seconds to collect videos at different moments; using the proposed spatio-temporal joint multi-view video interpolation and 3D modeling method to obtain videos of n viewing angles at all times, and then The three-dimensional model of the scene on each moment is reconstructed, and the specific method includes the following steps:

1) For each camera, use the optical flow method to obtain the forward optical flow and backward optical flow between two adjacent acquisition frames, and then interpolate the unacquired frames between the two frames, which is the interpolation frame in the instant domain;

2) For each acquisition moment, the model-assisted weighting method is used to obtain the image of the angle of view that is not collected at this moment, that is, the spatial interpolation frame;

3) calculate the time domain interpolation frame obtained by step 1) and the accumulated energy spectrum of the dual tree discrete wavelet domain of the space domain interpolation frame obtained by step 2), and extract key points;

4) Use the shape context to describe the extracted key points, and convert the key point matching problem based on the shape context into a square assignment, that is, a weighted bipartite graph matching problem, and solve it by the Hungarian method;

5) Obtain the final interpolation frame by solving the Poisson editing optimization problem;

6) At each moment, using images from all perspectives, including acquisition images and interpolation images, the multi-view stereo method is used to reconstruct the 3D model of the scene and render it.

The model-assisted weighting method specifically includes the following steps:

21) Extract the contour map of the three-dimensional object from the collected perspective image through simple difference or blue screen segmentation technology;

22) Using the contour map calculated in step 21), reconstruct a rough three-dimensional model, namely the visible shell model, by the EPVH method;

23) For each uncollected viewing angle i, weighted interpolation is performed using the images of the two closest adjacent collection viewing angles j and k, and the weight is calculated as follows:

(1)

为三维点p的法线与摄像机视线r_j的夹角，

为三维点p的法线与摄像机视线r_k的夹角；p为通过视角i图像上某一像素的视线与三维模型的交点。Among them, Θ and Ф are two constant angles, which respectively represent the maximum value of the angle between the allowable camera line of sight and the maximum angle between the three-dimensional point normal and the camera line of sight; θ ₁ is the camera line of sight r _i and the camera line of sight The included angle of r _j , θ ₂ is the included angle between camera line of sight r _i and camera line of sight r _k ,

is the angle between the normal of the 3D point p and the camera line of sight r _j ,

is the angle between the normal of the 3D point p and the camera line of sight r _k ; p is the intersection point of the line of sight of a certain pixel on the image passing through the viewing angle i and the 3D model.

Calculate the cumulative energy spectrum of the dual tree discrete wavelet domain and extract key points, the specific method includes the following steps:

31) performing dual-tree discrete wavelet transform on the spatial domain interpolation frame and the time domain interpolation frame, and decomposing them into S scales;

32) Calculate the key point energy spectrum {M _s } _1≤s≤S at each scale of the real part and imaginary part respectively, and the key point energy at each pixel position is calculated as:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 6</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Where {c ₁ , K, c ₆ } are the six subband coefficients of the pixels corresponding to the real or imaginary part, and the parameters α and β are used to adjust the importance of the scale in the accumulated energy spectrum;

33) The energy spectrum obtained in step 32) is interpolated into the original image size using a two-dimensional Gaussian kernel, and the interpolation spectrum under the scale s is defined as g _s (M _s );

并得到最终的积累能量谱为 $A=\sqrt{{A_r}^2+{A_i}^2};$ 34) Calculate the cumulative energy spectra A _r and A _i of the real and imaginary parts respectively as

And get the final accumulated energy spectrum as $A=\sqrt{{{\mathrm{<figure-callout\; id="2"\; label="A\; r"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">A</figure-callout>}}_r}^2+{{\mathrm{<figure-callout\; id="2"\; label="A\; i"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">A</figure-callout>}}_i}^2};$

35) Using the SIFT method to extract the key points of the accumulated energy spectrum obtained in step 34).

The key point matching method based on the shape context is specifically to obtain the final interpolation frame by solving the following optimization problem:

Δf | _Ω = divv $\mathrm{the\; s}.t.f{|}_{\&PartialD;\mathrm{\&Omega;}}=\hat{f}{|}_{\&PartialD;\mathrm{\&Omega;}}---\left(3\right)$

为拉普拉斯算子，v＝(u，v)为时域插值帧的梯度向量场，

为v的散度，为空域插值帧，

为闭集Ω的边界，s.t.表示“满足...条件”的意思，|_Ω是指在闭集Ω上，

表示在闭集Ω边界上。Among them, f is the unknown frame to be interpolated,

Be the Laplacian operator, v=(u, v) is the gradient vector field of the time domain interpolation frame,

is the divergence of v, Interpolate frames for the spatial domain,

is the boundary of the closed set Ω, st means "satisfies the ... condition", | _Ω refers to the closed set Ω,

Expressed on the closed set Ω boundary.

Features and effects of the method of the present invention:

The method of the invention avoids the need for expensive high-speed cameras and the problems of existing video interpolation and unsmooth image fusion methods. Through space-time sampling, space-time interpolation and space-time optimization, high frame rates under low frame rate camera acquisition conditions are realized. High-rate multi-view video restoration and 3D reconstruction of scenes. Has the following characteristics:

1. The program is simple and easy to implement.

2. Space-time sampling and interpolation for non-planar camera systems. Reasonably group the system composed of low frame rate cameras, so that each group of cameras is synchronized and evenly distributed. Different groups of cameras interleavedly capture dynamic scenes in the time domain. At each sampling moment, the unsampled camera images are interpolated in the spatial domain using a weighted method, and interpolated in the temporal domain using a bidirectional optical flow method.

3. The optimization method of shape context based on dual-tree discrete wavelet transform. Transforming the optimization of space-time information into the solution of Poisson editing problems for images. Using the time-shift invariance and direction selectivity of dual-tree discrete wavelet transform, key points of interest near edges (high-frequency information) are extracted, and then the shape context method is used to match these key points as boundary constraints.

The present invention can use a low frame rate camera system to realize three-dimensional reconstruction of dense dynamic scenes in time domain. The proposed method is very scalable: multi-view video restoration and 3D dynamic scene reconstruction with higher temporal resolution can be obtained simply by adding more cameras or adopting higher frame rate cameras.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flow chart of a spatio-temporal joint multi-view video interpolation and three-dimensional modeling method according to an embodiment of the present invention;

Fig. 2 adopts the proposed inventive method and other two methods to recover a certain uncollected frame result of sequence 1 in the embodiment of the present invention;

Fig. 3 is the visible shell model reconstructed by the embodiment of the present invention for sequence 1 and the result of the three-dimensional model reconstructed by the proposed inventive method;

Fig. 4 is the result of the dynamic three-dimensional model reconstructed for sequence 2 by using the proposed inventive method according to the embodiment of the present invention.

Detailed ways

The invention solves the problem of Poisson editing by transforming joint spatio-temporal multi-view video interpolation into images, wherein key point extraction and matching utilizes the directionality and shape context (shape) of dual-tree discrete wavelet transform (DDWT) context) to achieve multi-view video interpolation with high frame rate and high quality. The obtained result has the characteristics of good interpolation effect, high precision, accurate and complete reconstructed three-dimensional model, and can be realized under the condition of low frame rate camera array with group interleaved sampling.

The space-time joint multi-view video interpolation and three-dimensional modeling method of the present invention is characterized in that:

Group the multi-camera array at intervals (set n cameras with a frame rate of f frames per second, and divide them into m groups at even intervals, where n and m are positive integers, and n is an integer multiple of m), and the same group of cameras collects synchronously to obtain Videos of n/m angles of view at the same moment, different groups of cameras are interleaved at 1/(fm) seconds to collect videos at different moments; using the proposed spatio-temporal joint multi-angle video interpolation and 3D modeling method to obtain all moments Videos from n perspectives, and then reconstruct the 3D model of the scene at each moment. The specific method includes the following steps:

1) For each camera, use the optical flow method proposed by Brox et al. (Brox T, Bruhn A, Papenberg N, et al. High accuracy optical flow estimation based on a theory for warping. Proceedings of European Conference on Computer Vision, volume 3024, 2004.25-36.) Calculate the forward optical flow and backward optical flow between two adjacent acquisition frames, and then interpolate the m-1 frame between the two frames; the specific time domain interpolation method may include the following steps:

11) Assuming that the motion path is linear, that is, the position of the pixel on the motion path on the frame to be interpolated is proportional to the relative position of the frame between the two latest acquisition frames, according to the forward optical flow and backward optical flow results , calculate the forward optical flow interpolation and backward optical flow interpolation of the pixel corresponding to the frame to be interpolated;

12) For each pixel on the frame to be interpolated, take the average value of its forward optical flow interpolation and backward optical flow interpolation as the final estimation result;

13) Using an eight-neighborhood filtering method to fill in unassigned pixel holes in the frame to be interpolated;

2) For each collection moment, adopt the model-assisted weighting method to obtain the image of the angle of view that is not collected at this moment; the specific method may include the following steps:

21) Extract the contour map of the three-dimensional object from the collected perspective image through simple difference or blue screen segmentation technology;

22) Utilize the outline figure calculated in step 21) to reconstruct a rough three-dimensional model by EPVH (Exact Polyhedral Visual Hulls, FrancoJ S, Boyer E.Exact polyhedral visual hulls.Proceedings of British Machine Vision Conference, 1994.329-338.) method, That is, the visual shell model;

23) For each uncollected viewing angle i, weighted interpolation is performed using the images of the two closest adjacent collection viewing angles j and k, and the weight is calculated as follows:

(1)

(1)

为三维点p的法线与摄像机视线r_j的夹角，

为三维点p的法线与摄像机视线r_k的夹角；p为通过视角i图像上某一像素的视线与三维模型的交点；Among them, Θ and Ф are two constant angles, which respectively represent the maximum value of the angle between the allowable camera line of sight and the maximum angle between the three-dimensional point normal and the camera line of sight; θ ₁ is the camera line of sight r _i and the camera line of sight The included angle of r _j , θ ₂ is the included angle between camera line of sight r _i and camera line of sight r _k ,

is the angle between the normal of the 3D point p and the camera line of sight r _j ,

is the angle between the normal of the 3D point p and the camera line of sight r _k ; p is the intersection point of the line of sight of a certain pixel on the image passing through the viewing angle i and the 3D model;

3) calculate by step 1) the time domain interpolation frame that obtains and by step 2) the accumulative energy spectrum of the double tree wavelet transform domain of the space domain interpolation frame that obtains, and extract key point; Concrete method may comprise the following steps:

31) performing dual-tree discrete wavelet transform on the spatial domain interpolation frame and the time domain interpolation frame, and decomposing them into S scales;

32) Calculate the key point energy spectrum {M _s } _1≤s≤S at each scale of the real part and imaginary part respectively, and the key point energy at each pixel position is calculated as:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 6</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Where {c ₁ , K, c ₆ } are the six subband coefficients of the pixels corresponding to the real part or imaginary part, and the parameters α and β are used to adjust the importance of the scale in the accumulated energy spectrum;

33) The energy spectrum obtained in step 32) is interpolated into the original image size using a two-dimensional Gaussian kernel, and the interpolation spectrum under the scale s is defined as g _s (M _s );

并得到最终的积累能量谱为 $A=\sqrt{{A_r}^2+{A_i}^2};$ 34) Calculate the cumulative energy spectra A _r and A _i of the real and imaginary parts respectively as

And get the final accumulated energy spectrum as $A=\sqrt{{A_{\mathrm{<figure-callout\; id="2"\; label="r"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">r</figure-callout>}}}^2+{A_{\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">i</figure-callout>}}}^2};$

35) adopt the SIFT (Scale-Invariant Feature Transform) method to extract the key points of the accumulated energy spectrum obtained by step 34);

4) Use the shape context to describe the extracted key points, and convert the key point matching problem based on the shape context into a square assignment (weighted bipartite graph matching) problem and solve it by the Hungarian method;

5) Obtain the final interpolated frame by solving the following optimization problem:

Δf | _Ω = divv $\mathrm{the\; s}.t.f{|}_{\&PartialD;\mathrm{\&Omega;}}=\hat{f}{|}_{\&PartialD;\mathrm{\&Omega;}}---\left(3\right)$

为拉普拉斯算子，v＝(u，v)为时域插值帧的梯度向量场，

为v的散度，为空域插值帧，

为闭集Ω的边界。Among them, f is the unknown frame to be interpolated,

Be the Laplacian operator, v=(u, v) is the gradient vector field of the time domain interpolation frame,

is the divergence of v, Interpolate frames for the spatial domain,

is the boundary of the closed set Ω.

6) At each moment, use images from all perspectives (acquisition images and interpolation images) to reconstruct and render the 3D model of the scene using a multi-view stereo method.

The present invention proposes a method for spatio-temporal joint multi-view video interpolation and three-dimensional modeling, which is described in detail in conjunction with the accompanying drawings and embodiments as follows:

The structure of the embodiment of the system for realizing the method of the present invention is as follows: 20 cameras with a frame rate of 30 frames per second are distributed in a ring around the scene to be collected. The multi-camera array is divided into 4 groups at even intervals, and the cameras in the same group are synchronously collected to obtain videos from 5 viewing angles at the same time, and different groups of cameras are interleaved at 1/120 seconds to collect videos at different times; the proposed Combined spatio-temporal multi-view video interpolation and 3D modeling method to obtain videos from 20 viewing angles at all moments, and then reconstruct the 3D model of the scene at each moment. As shown in Figure 1, it is a flow chart of a spatio-temporal joint multi-view video interpolation and three-dimensional modeling method according to an embodiment of the present invention, including the following steps:

1) For each camera, use the optical flow method proposed by Brox et al. (Brox T, Bruhn A, Papenberg N, et al. High accuracy optical flow estimation based on a theory for warping. Proceedings of European Conference on Computer Vision, volume 3024 , 2004.25-36.) Calculate the forward optical flow and backward optical flow between two adjacent acquisition frames, and then interpolate 3 frames between the two frames; the specific time-domain interpolation method may include the following steps:

11) Assuming that the motion path is linear, that is, the position of the pixel on the motion path on the frame to be interpolated is proportional to the relative position of the frame between the two latest acquisition frames, according to the forward optical flow and backward optical flow results , calculate the forward optical flow interpolation and backward optical flow interpolation of the pixel corresponding to the frame to be interpolated;

12) For each pixel on the frame to be interpolated, take the average value of its forward optical flow interpolation and backward optical flow interpolation as the final estimation result;

13) Using an eight-neighborhood filtering method to fill in unassigned pixel holes in the frame to be interpolated;

2) For each collection moment, adopt the model-assisted weighting method to obtain the image of the angle of view that is not collected at this moment; the specific method may include the following steps:

21) Extract the contour map of the three-dimensional object from the collected perspective image through simple difference or blue screen segmentation technology;

22) Using the contour map calculated in step 21), a rough three-dimensional model is reconstructed by the EPVH (Exact Polyhedral Visual Hulls, Franco J S, Boyer E. Exact polyhedral visual hulls. Proceedings of British MachineVision Conference, 1994.329-338.) method, That is, the visual shell model;

23) For each uncollected viewing angle i, weighted interpolation is performed using the images of the two closest adjacent collection viewing angles j and k, and the weight is calculated as follows:

为三维点p的法线与摄像机视线r_j的夹角，为三维点p的法线与摄像机视线r_k的夹角；p为通过视角i图像上某一像素的视线与三维模型的交点；Among them, Θ=80° and Ф=70° are two constant angles, which respectively represent the maximum value of the angle between the allowable camera line of sight and the maximum value of the angle between the three-dimensional point normal and the camera line of sight; θ ₁ is the camera The angle between line of sight r _i and camera line of sight r _j , θ ₂ is the angle between camera line of sight r _i and camera line of sight r _k ,

is the angle between the normal of the 3D point p and the camera line of sight r _j , is the angle between the normal of the 3D point p and the camera line of sight r _k ; p is the intersection point of the line of sight of a certain pixel on the image passing through the viewing angle i and the 3D model;

3) calculate by step 1) the time domain interpolation frame that obtains and by step 2) the accumulative energy spectrum of the double tree wavelet transform domain of the space domain interpolation frame that obtains, and extract key point; Concrete method may comprise the following steps:

31) performing dual-tree discrete wavelet transform on the spatial domain interpolation frame and the time domain interpolation frame, and decomposing them into s scales;

32) Calculate the key point energy spectrum {M _s } _1≤s≤3 at each scale of the real part and imaginary part respectively, and the key point energy at each pixel position is calculated as:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 6</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Where {c ₁ , K, c ₆ } are the six subband coefficients of the pixels corresponding to the real or imaginary part, and the parameters α and β are used to adjust the importance of the scale in the accumulated energy spectrum, α=1,

33) The energy spectrum obtained in step 32) is interpolated into the original image size using a two-dimensional Gaussian kernel, and the interpolation spectrum under the scale s is defined as g _s (M _s );

并得到最终的积累能量谱为 $A=\sqrt{{A_r}^2+{A_i}^2};$ 34) Calculate the cumulative energy spectra A _r and A _i of the real and imaginary parts respectively as

And get the final accumulated energy spectrum as $A=\sqrt{{A_{\mathrm{<figure-callout\; id="2"\; label="r"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">r</figure-callout>}}}^2+{A_{\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000091126400000011.png"\; state="\{\{state\}\}">i</figure-callout>}}}^2};$

35) adopt the SIFT (Scale-Invariant Feature Transform) method to extract the key points of the accumulated energy spectrum obtained by step 34);

4) Use the shape context to describe the extracted key points, and convert the key point matching problem based on the shape context into a square assignment (weighted bipartite graph matching) problem and solve it by the Hungarian method;

5) Obtain the final interpolated frame by solving the following optimization problem:

Δf | _Ω = divv $\mathrm{the\; s}.t.f{|}_{\&PartialD;\mathrm{\&Omega;}}=\hat{f}{|}_{\&PartialD;\mathrm{\&Omega;}}---\left(3\right)$

为拉普拉斯算子，v＝(u，v)为时域插值帧的梯度向量场，

为v的散度，

为空域插值帧，为闭集Ω的边界。Among them, f is the unknown frame to be interpolated,

Be the Laplacian operator, v=(u, v) is the gradient vector field of the time domain interpolation frame,

is the divergence of v,

Interpolate frames for the spatial domain, is the boundary of the closed set Ω.

The present embodiment is to the final optimized interpolation result of sequence 1 and the comparison with other methods as shown in Figure 2, wherein (a) figure is to adopt wavelet-based SSIM fusion method (X.Luo, J.Zhang, and Q.Dai, "A classification-based image fusion scheme using wavelet transform," in Proc.SPIE 8064, no.806400, 2011.) Interpolated frame results; (b) The picture shows the two-dimensional empirical mode decomposition method (Y.Zheng and Z .Qin, "Region-based image fusion method using bidimensional empirical modedecomposition," Journal of Electronic Imaging, vol.18, no.1, p.013008, 2009.) obtained interpolation frame results; (c) the figure shows the use of the present invention method to get the interpolated frame result.

6) At each moment, use images from all perspectives (acquisition images and interpolation images) to reconstruct and render the 3D model of the scene using a multi-view stereo method.

As shown in Fig. 3, it is the result of the three-dimensional model reconstructed on sequence 1 by using the proposed inventive method. Wherein, the figure (a) is a visible shell model, and the figure (b) is a model reconstructed by the method of the present invention. The model is rendered with a normal map. As shown in Fig. 4, it is the result of the dynamic 3D model reconstructed on sequence 2 using the proposed inventive method. Among them, the first picture is the general picture that puts the models of each moment together, and the following pictures are the modeling results of each moment.

Claims (4)

1. A space-time joint multi-view video interpolation and three-dimensional modeling method is characterized in that a plurality of camera arrays are grouped at intervals: n cameras with frame rate of f frames/second are arranged and are uniformly divided into m groups at intervals, n and m are positive integers, and n is an integral multiple of m; the same group of cameras synchronously acquire n/m videos of visual angles at the same moment, and the different groups of cameras acquire the videos at different moments by time interpolation of 1/(fm) seconds; the method comprises the following steps of obtaining videos of n visual angles at all moments by adopting the proposed space-time joint multi-visual-angle video interpolation and three-dimensional modeling method, and further reconstructing a three-dimensional model of a scene at each moment, wherein the method comprises the following steps:

1) for each camera, forward optical flow and backward optical flow between two adjacent collected frames are obtained by adopting an optical flow method, and then the non-collected frames between the two frames, namely the frames subjected to the time domain interpolation, are interpolated;

2) for each acquisition moment, obtaining an image of an uncollected visual angle at the moment by adopting a model-assisted weighting method, namely a spatial domain interpolation frame;

3) calculating the accumulated energy spectrum of the dual-tree discrete wavelet domain of the time domain interpolation frame obtained in the step 1) and the spatial domain interpolation frame obtained in the step 2), and extracting key points;

4) describing the extracted key points by using shape context, converting a key point matching problem based on the shape context into square assignment, namely a weighted bipartite graph matching problem, and solving by using a Hungarian method;

5) obtaining a final interpolation frame by solving a Poisson editing optimization problem;

6) at each moment, images of all visual angles are utilized, including collected images and interpolation images, and a three-dimensional model of a scene is reconstructed and rendered by adopting a multi-visual angle stereo method.

2. The method of claim 1, wherein the model-assisted weighting method specifically comprises the steps of:

21) extracting a contour map of the three-dimensional object from the collected visual angle image by a differential or blue screen segmentation technology;

22) reconstructing a rough three-dimensional model, namely a visible shell model, by using the contour map obtained by calculation in the step 21) through an EPVH method;

23) for each non-collection visual angle i, performing weighted interpolation by using the images of two collection visual angles j and k adjacent to the non-collection visual angle i, wherein the weight is calculated as follows:

wherein, theta and phi are two constant angles which respectively represent the maximum value of an included angle between allowed camera sight lines and the maximum value of an included angle between a three-dimensional point normal and the camera sight line; theta₁Is the camera sight line r_iAnd the camera sight line r_jAngle of (a) of₂Is the camera sight line r_iAnd the camera sight line r_kThe angle of,

normal to the three-dimensional point p and the camera's line of sight r_jThe angle of,

normal to the three-dimensional point p and the camera's line of sight r_kThe included angle of (A); p is the intersection of the three-dimensional model and a line of sight through a pixel on the view i image.

3. The method of claim 1, wherein the accumulated energy spectrum of the dual-tree discrete wavelet domain is calculated and the key points are extracted, the method comprising the steps of:

31) performing double-tree discrete wavelet transform on the spatial domain interpolation frame and the time domain interpolation frame, and decomposing the spatial domain interpolation frame and the time domain interpolation frame into S scales;

32) respectively calculating the energy spectrum { M of the key point under each scale of the real part and the imaginary part_s}_1≤i≤SThe keypoint energy for each pixel position is calculated as:

wherein c₁,…c₆The parameters alpha and beta are used for adjusting the importance of the scale in the accumulated energy spectrum;

33) subjecting the energy obtained in step 32)The spectrum is interpolated by two-dimensional Gaussian kernel to obtain the original image size, and the interpolation spectrum at the scale s is defined as g_s(M_s)；

34) Calculating the accumulated energy spectrum A of the real part and the imaginary part respectively_rAnd A_iAs a result of

And obtaining a final accumulated energy spectrum of

35) Extracting key points of the accumulated energy spectrum obtained in the step 34) by adopting a SIFT method.

4. The method as claimed in claim 1, wherein the shape context based keypoint matching method is embodied by solving the following optimization problem to obtain the final interpolated frame:

wherein,ffor the unknown frame to be interpolated,

is Laplace operator, v ═ o (u，v) For the gradient vector field of the time-domain interpolated frame,

is the divergence of v and is the sum of the magnitudes of v,

in order to interpolate the frame in the spatial domain,

Ω is the boundary of the closed set Ω, s.t. means "condition … is satisfied", and_Ωmeans that over a closed set omega, calcuim

_ΩRepresented on the closed set omega boundary.

Images