CN115457182A - An Interactive Viewpoint Image Synthesis Method Based on Multi-plane Image Scene Representation - Google Patents

Abstract

In order to eliminate distortion and artifacts in a new viewpoint image and improve the synthesis quality of the new viewpoint image, a three-dimensional convolution neural network is used for capturing spatial features across a plurality of depth planes, and meanwhile, the capability of predicting an occlusion area on the depth plane is established. The homography transformation module is used for coding the position information of the input reference image pair and respectively re-projecting the input images into the target camera through the homography transformation matrix; the network framework uses a coder decoder framework based on three-dimensional convolution, and a coder and a decoder of each layer are connected into a U-shaped network through jumping, so that the capability of capturing context information is enhanced; the network output module generates a multi-plane image scene representation by mixing the weight predicted by the network and the alpha image, and the image of the target viewpoint is rendered by synthesizing the scene representation from back to front. The method can improve the accuracy of the new viewpoint image synthesis.

Description

An Interactive Viewpoint Image Synthesis Method Based on Multi-plane Image Scene Representation

technical field

The invention relates to the field of image processing, in particular to an interactive viewpoint image synthesis method based on multi-plane image scene representation.

Background technique

With the development of deep learning, view synthesis has gained a lot of attention in computer vision and computer graphics research because of its wide range of applications, such as providing user Real-time interactive experience. The challenge of view synthesis with explicit 3D representations is that it requires inferring an accurate geometric understanding of the scene from existing viewpoints. It turns out that MPI (Multiplanar Image) is a convenient volumetric representation that can efficiently estimate the geometry of a scene to synthesize new viewpoint images. MPI is a set of translucent images distributed at different depths, which can encode diffuse surfaces as well as non-Lambertian effects such as transparent and reflective areas. Given an MPI scene representation, new viewpoint images of the target image can be simply rendered by applying an inverse homography transformation and back-to-front alpha compositing.

Currently, many works focus on learning MPI scene representations from single or multiple inputs. Tucker et al. (Tucker R, Snavely N. Single-view view synthesis with multiplane images.) predict the MPI representation of the scene from a single input image, while the MINE method extends the MPI into a continuous depth plane by introducing a planar neural radiation field, namely Given a single image as input, planes of any depth can be predicted in MPI. Although the single-input method achieves good results, it needs to use point clouds as supervision to solve the scale ambiguity problem of monocular depth estimation. However, the process of predicting MPI is itself an estimation of depth; MPI based on multiple inputs The prediction method does not have the problem of scale ambiguity. Given an input stereo image pair captured by a narrow-baseline stereo camera, Zhou et al. (ZhouT, Tucker R, Flynn J, et al. Stereo magnification: Learning view synthesis using multiplane images.) use an end-to-end 2D deep learning network To infer MPI scene representations, the network uses a differentiable rendering module to generate images from new viewpoints, a problem known as stereoscopic upscaling. Building on this work, Tucker et al. (Srinivasan P P, Tucker R, Barron J T, et al. Pushing the boundaries of view extrapolation with multiplane images.) proposed a theoretical analysis that the range of views that can be rendered from MPI increases with The MPI disparity increases linearly with increasing sampling frequency, and a two-stage framework based on a 3D convolutional neural network uses optical flow to predict occluded content. Although increasing the number of layers of MPI can effectively expand the boundary of viewpoint extrapolation, due to the limitation of GPU, the number of layers of MPI cannot be increased infinitely. Flynn et al. (Flynn J, Broxton M, Debevec P, et al. Deepview: View synthesis with learned gradient descent.) regard the MPI prediction process as an inverse problem, which belongs to the over-fitting problem, so the gradient descent method is proposed to solve, but this method is computationally intensive.

Contents of the invention

In view of the above-mentioned background technology, the existing method cannot capture the feature connection of MPI across the depth plane, resulting in the problem that the synthesized new view image often has obvious distortion and artifacts. The present invention provides a new view synthesis based on MPI scene representation method, which utilizes a 3D convolutional neural network to capture spatial features across multiple depth planes, establishes the ability of MPI to predict occluded and hidden regions in depth planes, and improves the prediction accuracy of plane transparency values. It is verified on the Spaces dataset and the RealEstate 10K dataset. Experiments prove that the invention can effectively synthesize new viewpoint images, and has better performance than existing methods.

An interactive viewpoint image synthesis method based on multi-plane image scene representation, comprising the following steps:

Step 1, obtain training data and preprocessing;

Step 2, inputting the training image data obtained in step 1 into a three-dimensional convolutional neural network based on multi-plane image scene representation for training;

The three-dimensional convolutional neural network comprises a homography transformation module, a three-dimensional convolution codec architecture and a network output module;

The homography transformation module encodes the position information of the reference image pair acquired by the input reference camera, and re-projects the input image into the target camera respectively through the homography transformation matrix;

The three-dimensional convolutional codec architecture includes a preprocessing block and a four-layer encoder-decoder structure, and the encoder and decoder of each layer are connected as a U-shaped network by skipping;

The network output module generates a multi-plane image scene representation by mixing the weight and alpha image predicted by the network, and the image of the target viewpoint is rendered by synthesizing the multi-plane image from back to front;

Step 3, based on the trained network, input the test image for detection, and obtain the final new viewpoint image synthesis result.

Further, in step 1, the data preprocessing includes performing data enhancement on the training images.

Further, in step 2, the homography transform uses the same depth to infer the geometry of the scene by comparing different input images.

Further, in the homography transformation module, based on the parameters of the reference camera, the position information of the input reference image pair I ₁ and I ₂ is encoded, and the corresponding plane scanning volume PSV (Plane Sweeping Volume) is calculated, and the reference image Re-project to the target camera at a set of fixed depths D, where the projection points in the reference view and the target view are connected through the homography matrix,

Further, in step 2, the 3D convolutional codec architecture captures spatial features across multiple depth planes, and predicts occluded and hidden areas of multi-plane images in depth planes.

Further, in the three-dimensional convolution codec architecture, the convolution operation is performed on the 3N color channels of D consecutive images with a resolution of H×W (where N is the number of input images); the first in the preprocessing block One convolutional layer uses a convolution kernel of size 7×7×7, and the kernel size used by the remaining three-dimensional convolutions is 3×3×3.

Further, each layer of the encoder-decoder structure contains an encoding block and a decoding block; the encoding block consists of four three-dimensional convolutions, and there is a skip connection between each two convolutions, except for the encoding block of the first layer All others downsample the input tensor; the decoding block consists of two 3D convolutional layers with a kernel size of 3×3×3 and an upsampling layer.

The beneficial effects that the present invention reaches are:

1) Build a 3D convolutional neural network with an encoder-decoder architecture to capture spatial features across multiple depth planes, eliminate distortion and artifacts in new viewpoint images, and improve the synthesis quality of new viewpoint images;

2) The ability to predict occluded regions on the depth plane is established.

3) Improve the accuracy of new viewpoint image synthesis.

Description of drawings

FIG. 1 is a flowchart of a view synthesis algorithm using a three-dimensional convolutional neural network based on MPI scene representation in an embodiment of the present invention.

FIG. 2 is a schematic diagram of a three-dimensional convolutional neural network architecture in an embodiment of the present invention.

Fig. 3 is a schematic diagram of the structure of the coding block 2 in the embodiment of the present invention.

Fig. 4 is a schematic diagram of the architecture of the decoding block 2 in the embodiment of the present invention.

detailed description

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings.

The overall structure of the present invention is shown in Figure 1, and a viewpoint synthesis framework based on MPI scene representation is proposed. This method uses a three-dimensional convolutional neural network to improve the extraction of multi-planes and mix the spatial features across multiple planes to reconstruct high-quality images. The MPI scene representation is trained in an end-to-end manner. This method specifically comprises the following steps:

Step 1. Obtain training image data and preprocess.

Since the network needs multiple iterations of training and needs to be adapted to various application scenarios, the amount of training data prepared needs to meet a certain order of magnitude requirement. The Spaces dataset and the RealEstate10K dataset are used as training image data. The RealEstate10k dataset contains about 7,500 indoor and outdoor scenes extracted from YouTube videos, and the internal parameters and relative positions of the cameras are calibrated; the Spaces dataset contains 100 indoor and outdoor scenes. The scene is shot with 16 cameras, the interval between each camera is about 10 cm, and the internal and external parameters of the cameras are calibrated using the motion recovery structure SFM (Structure-from-Motion). Use 90 scenes in this dataset for training and evaluate on the remaining 10 scenes, and the image resolution is set to 800×480. It allows deep learning methods to use large-scale data to train their architectures.

Step 2. The flow of view synthesis algorithm using 3D convolutional neural network based on MPI scene representation is shown in Figure 1, which consists of three parts: homography transformation module, network framework and network output module. The homography transformation module is to encode the position information of the input reference image pair, and re-project the input image to the target camera through the homography transformation matrix; the network framework uses a three-dimensional convolution-based encoder-decoder architecture, The encoder and decoder of each layer are connected into a U-shaped network by jumping, which enhances the ability to capture context information; the network output module generates an MPI scene representation by mixing the weights predicted by the network and the alpha image, and the image of the target viewpoint is obtained from the MPI. The back-to-front composition is rendered.

Step 21. Given two input images I ₁ and I ₂ , known camera parameters C ₁ =(A ₁ ,[R ₁ ,t ₁ ]) and C ₂ =(A ₂ ,[R ₂ ,t ₂ ]) , where A _i represents the internal parameters of the camera, and [R _i , T _i ] (i=1,2) represents the external parameters of the camera (ie, rotation matrix and translation vector). As shown in Figure 1, in order to encode the position information of the input reference image pair I ₁ and I ₂ , a pair of plane scanning volume PSV (Plane Sweeping Volume) is calculated, that is, the reference image pair is at a fixed depth D respectively reprojected into the target camera. Consider a pixel point p _i (u _i , v _i ,1) in the reference view I _i (i=1,2) (where (u _i , v _i ,1) is the coordinate of the pixel point p _i ) and Refer to the corresponding voxel at depth _zi in the camera coordinate system. If the depth of this voxel in the target camera coordinate system is expressed as z _v , then the pixel point p _i (u _i , v _i , 1) projected on the target view I _t is the pixel point p _v (u _v , v _v , 1) (where (u _v , v _v , 1) is the coordinate of pixel point p _v ) can be expressed as:

Among them, A _v represents the internal parameters of the target camera, and [R _v , T _v ] represents the external parameters of the target camera (ie, rotation matrix and translation vector). A 3D scene can be split into multiple planes that are at the same distance (i.e. disparity value) from the reference camera. For points on such a depth plane, their projected point p _i in the reference view and projected point p _v in the target view can be related by the homography matrix H _vi,z , that is, H _vi,z can be simplified by (1) get:

Since a series of homography matrices H _vi,z are applied to the reference view, a set of homography warped views P _i (i.e. planar scan volume PSV) can be obtained respectively, i.e. reprojected on different depth planes result. The size of each PSV tensor is [3, D, H, W], and the two PSVs are connected along the color channel to obtain a [3N, D, H, W] tensor as a three-dimensional convolutional neural network input of. where H and W are the height and width of the image respectively, D is the number of depth planes, and N is the number of input images. A 3D convolutional neural network learns to infer the geometry of a scene by comparing PSVs from two different views.

Step 22. As shown in Figure 2, the 3D convolutional neural network architecture consists of two parts: a preprocessing block and a four-layer encoder-decoder structure. Using 3D convolutions in training to extract spatial features across depths can effectively learn the spatial relationship between each plane. The three-dimensional convolution operation is performed on the 3N color channels of D consecutive images with a resolution of H×W. We use the number of depth planes D=32 and N=2 input images with a resolution of 480×800 For example, the input of the three-bit convolutional neural network is expressed as 6@32,480,800. The preprocessing module samples the input tensor to 32@16,240,400 by extracting spatial features across 32 depth planes. Except that the first convolution layer in the preprocessing block uses a convolution kernel of size 7×7×7, all other three-dimensional convolutions use a kernel size of 3×3×3. The encoding block and decoding block in each layer are connected as a U-network through skip connections, which enhances the ability to capture contextual information.

Each layer of the encoder-decoder architecture consists of an encoding block and a decoding block, as shown in Figure 3 and Figure 4, taking encoding block 2 and decoding block 2 as examples, respectively. Coding block 2 is composed of four 3D convolutions, and there is a skip connection between every two convolutions. A 3D convolution with a convolution kernel size of 1×1×1 is applied to the first skip connection of coding block 2, Downsample the input tensor. It should be noted that only encoding block 1 does not down-sample the input feature tensor (the structure of the remaining encoding blocks is basically similar to that of encoding block 2 shown in Figure 3, and in addition, the input of encoding block 1 in Figure 2 is not down-sampled, and the output downsampling exists); decoding block 2 consists of two 3D convolutional layers with a kernel size of 3×3×3 and an upsampling layer, and the other decoding blocks are the same as decoding block 2. In Figure 2, the parameters of each module represent changes in the size of the tensor 3N@D, H, and W. In the figure, "upsampling 2×" means that the resolution and depth channel parameters are doubled.

Step 23, network output module As shown in FIG. 1 , the three-dimensional convolutional neural network directly predicts an MPI opacity α _i and two mixing weights w _i (i=1, 2). While the RGB values of MPI can be well modeled by mixing weights and _Pi , where Pi is the planar scan volume PSV obtained by the _homography matrix described in step 21. Therefore, for each plane in MPI, compute its RGB image c as:

c＝∑w _i ΘP _i (i＝1,2) (3)

Finally, the target image I _t can be rendered from the MPI scene representation M={c _i ,α _i }(i=1,2...N) through alpha compositing. Among them, the rendering process is differentiable, and the process of synthesizing the target image I _t is defined as:

The present invention uses view synthesis as supervision to train the MPI prediction network, and uses VGG-19 perceptual loss normalized in the channel dimension as the loss function:

是目标图像I_t的真值，φ_l是VGG-19的一组层，权重超参数λ_l被设定为VGG-19(该VGG-19是在计算损失的过程中用到的二维卷积神经网络，使用其中的φ_l层提取

和I_t的特征并计算两者的损失)中神经元数量的倒数。in

is the true value of the target image I _t , φ _l is a set of layers of VGG-19, and the weight hyperparameter λ _l is set to VGG-19 (the VGG-19 is the two-dimensional volume used in the process of calculating the loss product neural network, using the φ _l layer to extract

and It _features and calculate the loss of both) the reciprocal of the number of neurons in .

Step 3, based on the trained network, input the test image for detection, and obtain the final new viewpoint image synthesis result.

In order to evaluate the ability of this model in inferring the target view, experiments were carried out on the RealEstate 10K dataset and the Spaces dataset. For a fair comparison, this method uses the same number of depth planes as other methods. When experimenting on the Spaces dataset, the input views are adjusted to 4. Experimental results show that this framework can capture spatial features across multiple depth planes to infer correct scene geometry and content of occluded regions, thereby synthesizing high-quality target views; at the same time, this method can handle thin and complex structures, rendering produce sharper object edges than previous methods.

In the ablation experiments, more depth planes (D=8, 16, 24, 32, 40) were set to verify their importance to the method. Two different baseline configurations are trained on the Spaces dataset, i.e. the baseline distance of the input viewpoint is about 20 cm and the baseline is about 40 cm, and 2 views are used as input. The dataset is enlarged by a factor of 16 by using data augmentation. Experimental results show that the performance of our model in synthesizing novel viewpoint images increases with the number of depth planes and is not affected by the baseline distance. The performance improvement can be attributed to the ability of the model to learn more accurate inference of scene geometry by sampling depth more densely during training; while the performance of synthesizing new viewpoint images decreases with the same number of depth planes. Decreases as the baseline distance increases, because as the baseline distance increases, it is difficult to propagate volumetric visibility relying on the network, so the performance of MPI suffers.

In summary, the present invention proposes a viewpoint synthesis method based on multi-plane scene representation. In order to eliminate the distortion and artifacts in the new viewpoint image and improve the synthesis quality of the new viewpoint image, the framework uses a three-dimensional convolutional neural network to capture cross- Spatial features of multiple depth planes, while building the ability to predict occluded regions on depth planes. Our method also achieves high-quality synthesis results on special regions, such as specular regions in the scene. Experimental results on two datasets demonstrate that the quality of new view synthesis outperforms previous algorithms. As the number of depth planes increases in the ablation experiments, the rendering quality of images from new viewpoints also improves.

The above descriptions are only preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments, but all equivalent modifications or changes made by those of ordinary skill in the art according to the disclosure of the present invention should be included within the scope of protection described in the claims.

Claims (7)

1. A method for synthesizing interactive viewpoint images based on multi-plane image scene representation, characterized in that: comprising the following steps: Step 1, obtain training data and preprocessing; Step 2, inputting the training image data obtained in step 1 into a three-dimensional convolutional neural network based on multi-plane image scene representation for training; The three-dimensional convolutional neural network comprises a homography transformation module, a three-dimensional convolution codec architecture and a network output module; The homography transformation module encodes the position information of the reference image pair acquired by the input reference camera, and re-projects the input image into the target camera respectively through the homography transformation matrix; The three-dimensional convolutional codec architecture includes a preprocessing block and a four-layer encoder-decoder structure, and the encoder and decoder of each layer are connected as a U-shaped network by skipping; The network output module generates a multi-plane image scene representation by mixing the weight and alpha image predicted by the network, and the image of the target viewpoint is rendered by synthesizing the multi-plane image from back to front; Step 3, based on the trained network, input the test image for detection, and obtain the final new viewpoint image synthesis result. 2 . The method for synthesizing interactive viewpoint images based on multi-plane image scene representations according to claim 1 , wherein in step 1, the data preprocessing includes performing data enhancement on the training images. 3 . 3. A method of interactive viewpoint image synthesis based on multi-plane image scene representation according to claim 1, characterized in that: in step 2, the homography transformation uses the same depth, and the scene is inferred by comparing different input images geometry structure. 4. a kind of interactive viewpoint image synthesis method based on multi-plane image scene representation according to claim 1, is characterized in that: in the homography transformation module, based on the parameter of reference camera, to the input reference image pair I ₁ and The position information of I ₂ is encoded to calculate the corresponding plane scanning volume PSV, and the reference image pair is re-projected to the target camera at a set of fixed depths D, wherein the projection points in the reference view and the target view are passed through a single Responsive matrix to connect. 5. A method of interactive viewpoint image synthesis based on multi-plane image scene representation according to claim 1, characterized in that: in step 2, the three-dimensional convolutional codec architecture captures spatial features across multiple depth planes, and predicts multiple Planar images occlude and hide regions in the depth plane. 6. A method of interactive viewpoint image synthesis based on multi-plane image scene representation according to claim 1, characterized in that: in the three-dimensional convolution codec architecture, in the 3N of D consecutive images with a resolution of H×W Convolution operations are performed on color channels, where the number of N-bit input images; the first convolution layer in the preprocessing block uses a convolution kernel of size 7×7×7, and the kernel used by the rest of the three-dimensional convolution All are 3×3×3 in size. 7. A kind of interactive viewpoint image synthesis method based on multi-plane image scene representation according to claim 1, is characterized in that: each layer of encoder-decoder structure all comprises an encoding block and a decoding block; The encoding block consists of It consists of four three-dimensional convolutions, and there is a skip connection between each two convolutions. Except for the encoding block of the first layer, the input tensor is down-sampled; the decoding block consists of a convolution kernel with a size of 3×3×3 It consists of two 3D convolutional layers and an upsampling layer.

Images