CN102592275A - Virtual viewpoint rendering method - Google Patents

Abstract

The invention relates to a virtual viewpoint rendering method, which belongs to the field of image processing and stereo imaging and can achieve high-quality virtual viewpoint rendering. The technical scheme adopted by the virtual viewpoint rendering method comprises the following steps: (1) preprocessing depth maps; (3) performing image fusion on the two virtual viewpoint images by using distance weighted values, eliminating most of hollows and producing a fusional initial virtual view; and (4) filling the hollows of the initial virtual view by a method on the basis of figure-ground segregation and obtaining a final virtual viewpoint image. The virtual viewpoint rendering method is mainly applicable to image processing and stereo imaging.

Description

Virtual viewpoint rendering method

Technical field

The invention belongs to Flame Image Process and three-dimensional imaging field, relate to a kind of virtual viewpoint rendering method based on the degree of depth.

Background technology

Traditional planar video is described three-dimensional scenic with plane picture, can not satisfy the requirement of people's stereo sense.Compare three-dimensional video-frequency with traditional planar video and have more intuitive visual expression effect, it can make people experience vivid depth perception and feeling of immersion.The virtual viewpoint rendering technology refers to the video image through the resulting Same Scene of two or more video camera viewpoints; Generating the process of other each these scene video images of video camera viewpoint, is to realize one of three-dimensional video-frequency key technologies for application such as free stereo TV and free view-point TV.

The virtual viewpoint rendering technology mainly comprise drafting based on model (Model Based Rendering, MBR) and based on the drafting of image (Image Based Rendering, IBR).Traditional drafting based on model utilizes computer graphics techniques; Scene is defined; Thereby set up three-dimensional scene models, utilize processing procedures such as computer realization scene model rendering, painted, illumination and projection then, finally generate the image of given viewpoint.Can the geometric model that therefore, accurately construct scene realistically to final imaging effect generation decisive influence.Because the complexity of scene often is not easy to control, be not very desirable through the MBR imaging effect under the complex scene, can't satisfy the constantly demand to the image sense of reality and feeling of immersion of development of user.How to adopt known visual point image to come directly to generate the correspondence image under the virtual view based on the main research of the drafting of image; Its drawing process has been avoided complicated Geometric Modeling and evaluation work; Therefore have advantage quick, simple, true to nature, become the current direction of active research the most.

Different IBR technology can be regarded as full light function (Plenoptic Function) concrete manifestation under certain condition.Full light function is expressed as scene at moment τ, is (V from three dimensional space coordinate _x, V _y, V _z) viewpoint set out, the visual angle does

The employing wavelength is the information summation that the light of λ is sampled.Can scene be expressed as one 7 dimension function thus,

is called full light function.IBR can be described as the sample set of given full light function, generates the process of its continuous representation.Based on this description, the IBR process can be decomposed into the sampling of full light function, three processes of rebuilding and resample.But under actual conditions, can't obtain the complete or collected works of the full light function of scene in advance, 7 dimension functional forms need huge storage and calculated amount simultaneously, so the researcher has launched big quantity research to application and simplification.The employing scene geometric information what, can the IBR technology be divided three classes: the IBR method of 1) not having how much information according to; 2) use the IBR method that implies geological information; 3) the IBR method of the clear and definite geological information of use.

There is not the intensive sampling drawing three-dimensional scene of the IBR method of how much information based on scene; The panorama sketch technology (Panoramas) that mainly comprises 2 dimensions; The concentric picture mosaic technology (Concentric Mosaics) of 3 dimensions; Light field technology (Light fields) and the ray space implementation method (Ray Space Representation) of 4 dimensions, the full light modeling technique method for drafting such as (Plenoptic Modeling) of 5 dimensions.

Use the IBR method of implicit geological information only to need input picture seldom, can under certain error condition, come to generate fast the scene picture with less cost, the picture quality that is generated simultaneously is very high.Use the IBR method of implicit geological information mainly to comprise view deformation technology (View Morphing) and view interpolation technology (View Interpolation) etc.The view deformation technology utilizes the geometric projection principle to go out the image of each new viewpoint on the photocentre line from the image reconstruction of two different points of view.The view interpolation technology is based on the continuity between the adjacent image, utilizes the depth information of each pixel of image to set up the corresponding relation of pixel between the neighbouring sample point photo-realistic images through the vision ultimate principle.

(Layered Depth Image is LDI) with based on the 3D rendering deformation technology (3D Image Warping) of depth image etc. to use the IBR method of clear and definite geological information mainly to comprise texture (Text Mapping), layered depth image technology.The essence of texture is that the image replay is mapped on the surface of 3D scene, comes show complex mold surface details with texture.To image layered method, the geological information that is utilized is the depth information on some branch aspects to the employing of layered depth image technology according to depth information.Drafting (Depth Image Based Rendering based on depth image; DIBR) owing to the depth information with scene is incorporated among the IBR; Utilize source images and corresponding depth image thereof to generate the virtual view view through the 3-D view mapping; Thereby significantly reduced the number of reference picture, saved storage space and transmission bandwidth.

DIBR technology development in recent years is rapid, is the focus of current research.McMillan has proposed 3D image warping equation the earliest, obtains virtual view through image mapped, greatly reduces the complexity that generates new view, but also is faced with the problem of how to carry out effective hole-filling.Cooke has proposed to utilize sampling density figure to choose many view synthesizing methods that best surface generates virtual view; Because the influence in cavity in the single view of not considering to generate; Can not obtain complete sampling density figure, among the scene that is applied to complicacy that can not be successful.Muller has proposed the method by the parameter derivation virtual view parameter of two reference-view around the method for 3D mapping, thereby can obtain the virtual view of any viewpoint between two reference-view, but still does not effectively solve empty problem.Mori is through using 3D warping and carry out virtual viewpoint rendering and through depth map being projected to the virtual image plane and the depth map that obtains after the mapping being carried out the back Filtering Processing solve empty problem; Blur depth map but the depth map that obtains is carried out the back Filtering Processing, can influence the rendering quality of virtual view.

Summary of the invention

The present invention is intended to overcome the deficiency of prior art, and a kind of virtual viewpoint rendering method is provided, and realizes high-quality virtual viewpoint rendering.For achieving the above object, the technical scheme that the present invention takes is that virtual viewpoint rendering method comprises the following steps:

(1) adopt border restructing algorithm that depth map is carried out pre-service based on rim detection;

(2) utilize two viewpoint reference-view and corresponding depth map thereof, obtain the virtual view view through the 3-D view conversion:

When drawing virtual visual point image; Adopt 3D Image Warping method; Utilize the camera parameter of depth image and demarcation; Pixel in the reference picture is mapped in the target image; The camera parameter re-projection that at first pixel of source reference image is utilized its depth information and reference camera is to their corresponding three dimensions point positions, utilizes the camera parameter of position and the virtual view of virtual view that these three dimensions points are projected to the virtual camera plane more then and is carried out to picture, obtains virtual visual point image;

(3) two width of cloth virtual visual point images are carried out image co-registration and eliminate most of cavity, generate a width of cloth initial virtual view:

Utilize distance weighted value to merge two width of cloth virtual views, thus the initial virtual view that obtains merging;

(4) utilization of initial virtual view is carried out hole-filling based on the method for background separation, obtain final virtual visual point image.

Said employing in the step (1) is carried out pre-service based on the border restructing algorithm of rim detection to depth map and is specially:

At first utilize the canny operator to carry out rim detection, extract the edge of foreground object in the depth map, carry out depth boundary reconstruct filtering then the edge that extracts is handled, adopt following cost function: J depth map _Recon(k)=J _F(k)+J _S(k)+J _C(k),, find best brightness value to replace the brightness value of current pixel with maximum cost to the neighbor cost value of the edge calculations current pixel that extracts, wherein, the brightness value of k represent pixel, J _ReconCost function when the represent pixel value is k, first filial generation valency function J _FRepresent the frequency of occurrences of each pixel, it is obtained by the occurrence number of pixel; Second sub-cost function J _SRepresent current pixel point to be adjacent the similarity of the brightness between the pixel; The 3rd sub-cost function J _CRepresent between current pixel and the neighbor near degree, last, for the depth map after the smooth boundary reconstruct, utilize medium filtering and morphology to handle the edge of depth of smoothness figure, thereby acquisition quality depth map preferably.

Said in the step (4) carries out hole-filling to the utilization of initial virtual view based on the method for background separation, obtains final virtual visual point image and is specially:

At first utilize the depth value of the depth map that merges the back virtual view, choose threshold value γ, depth map is divided, be about to background area and foreground object and be separated from each other; Fill through the method for neighborhood interpolation isolated then background area; At last foreground area is reset in the virtual view,, utilizes the method for neighborhood interpolation that whole virtual view is carried out hole-filling again, obtain final virtual view for guaranteeing that all cavities are all filled.

Said border restructing algorithm cost function in the step (1) is specially:

First filial generation valency function J _FDefine as follows:

$N_{\mathrm{OC}}\left(k\right)=\underset{i=0}{\overset{n\×n-1}{\mathrm{\Σ}}}\mathrm{\δ}[k,w_{n\×n}\left(i\right)],$

$J_F\left(k\right)=\frac{N_{\mathrm{OC}}\left(k\right)-N_{\mathrm{OC}}\left(\min \right)}{N_{\mathrm{OC}}\left(\max \right)-N_{\mathrm{OC}}\left(\min \right)}$

Wherein, w _{N * n}(i) represent window w _{N * n}The brightness value of middle pixel i, N _OC(max) and N _OC(min) represent minimum and maximum N respectively _OCValue;

Second sub-cost function J _SDefine as follows:

S(k)＝|I _cur-I _k|

$J_S\left(k\right)=\frac{S\left(\max \right)-S\left(k\right)}{S\left(\max \right)-S\left(\min \right)}$

Wherein, S (max) and S (min) represent minimum and maximum S (k) value respectively;

The 3rd sub-cost function J _CDefine as follows:

$C\left(k\right)=\frac{1}{N_{\mathrm{OC}}\left(k\right)}\underset{i=0}{\overset{N_{\mathrm{OC}}\left(k\right)-1}{\mathrm{\Σ}}}\sqrt{{(x_{\mathrm{cur}}-x_i)}^2+{(y_{\mathrm{cur}}-y_i)}^2}$

$J_C\left(k\right)=\frac{C\left(\max \right)-C\left(k\right)}{C\left(\max \right)-C\left(\min \right)}$

Wherein, C (max) and C (min) represent minimum and maximum C (k) value respectively.

The present invention has following technique effect:

The present invention adopts the border restructing algorithm based on rim detection that depth map is carried out pre-service, has improved the quality of depth map; And adopt based on the cavity in the hole-filling algorithm process virtual view of background separation, realized high-quality virtual viewpoint rendering.

Description of drawings

Fig. 1 has provided the not part amplification of the virtual view view of process embodiment of the invention method drafting;

Fig. 2 has provided the part of the virtual view view of drawing through embodiment of the invention method and has amplified;

Fig. 3 has provided the virtual view view full figure of drawing through embodiment of the invention method;

Fig. 4 has provided virtual viewpoint rendering flow process of the present invention.

Embodiment

The objective of the invention is to overcome the deficiency of prior art, propose a kind of new method of virtual viewpoint rendering.The present invention at first adopts the border restructing algorithm based on rim detection that depth map is carried out pre-service to improve the quality of depth map; Utilize the mapping of left and right sides reference-view and pretreated depth map to obtain the virtual view images of positions then, and utilize image co-registration the synthetic width of cloth view of two width of cloth virtual images; Adopt at last based on the cavity in the hole-filling algorithm process virtual view of background separation, realize high-quality virtual viewpoint rendering.

A kind of virtual viewpoint rendering method said method comprising the steps of:

(1) adopt border restructing algorithm that depth map is carried out pre-service based on rim detection;

At first utilize the canny operator to carry out rim detection, extract the edge of foreground object in the depth map depth map.Carry out depth boundary reconstruct filtering then the edge that extracts is handled, the design cost function is following: J _Recon(k)=J _F(k)+J _S(k)+J _C(k), the neighbor cost value to the edge calculations current pixel that extracts finds the best brightness value with maximum cost to replace the brightness value of current pixel.Wherein, the brightness value of k represent pixel, J _ReconCost function when the represent pixel value is k, first filial generation valency function J _FRepresent the frequency of occurrences of each pixel, it is obtained by the occurrence number of pixel; Second sub-cost function J _SRepresent current pixel point to be adjacent the similarity of the brightness between the pixel; The 3rd sub-cost function J _CRepresent between current pixel and the neighbor near degree.At last,, utilize medium filtering and morphology to handle the edge of depth of smoothness figure, thereby obtain quality depth map preferably for the depth map after the smooth boundary reconstruct.

(2) utilize two viewpoint reference-view and corresponding depth map thereof, obtain the virtual view view through the 3-D view conversion;

When drawing virtual visual point image, adopt 3D Image Warping method, utilize the camera parameter of depth image and demarcation, the pixel in the reference picture is mapped in the target image.At first the pixel of source reference image is utilized the three dimensions point position of the camera parameter re-projection of its depth information and reference camera to their correspondences; Utilize the camera parameter of position and the virtual view of virtual view that these three dimensions points are carried out to picture projecting to the virtual camera plane then, obtain virtual visual point image.

In the process of carrying out 3D warping, exist a plurality of reference views to be mapped to the phenomenon of same virtual visual point image position; Can cause the pixel of back projection to cover the pixel of first projection; Thereby caused serious distortion for the virtual view that generates, choose the pixel of the less pixel of depth value as needs in order to address this is that the method that adopts Z-Buffer to compare depth value.

(3) two width of cloth virtual visual point images are carried out image co-registration and eliminate most of cavity, generate a width of cloth initial virtual view;

Obtain in the process of virtual visual point image in the purpose image, becoming visible because the relation of mapping makes sightless zone in reference picture in reference picture mapping, these zones can not find corresponding Pixel Information in reference picture.Therefore, utilize distance weighted value to merge two width of cloth virtual views, thus the initial virtual view that obtains merging.

(4) utilization of initial virtual view is carried out hole-filling based on the method for background separation, obtain final virtual visual point image;

At first, utilize the depth value of the depth map that merges the back virtual view, choose threshold value γ, depth map is divided, be about to background area and foreground object and be separated from each other.Then, fill through the method for neighborhood interpolation isolated background area, so just can the background area of former hollow sectors before this utilized the pixel of adjacent background area fill.At last, foreground area is reset in the virtual view,, utilizes the method for neighborhood interpolation that whole virtual view is carried out hole-filling again, obtain final virtual view for guaranteeing that all cavities are all filled.

For making the object of the invention, technical scheme and advantage clearer, will combine accompanying drawing that embodiment of the present invention is done to describe in detail further below.

(1) adopt border restructing algorithm that depth map is carried out pre-service based on rim detection;

At first utilize the canny operator to carry out rim detection, extract the edge of foreground object in the depth map depth map.Carrying out depth boundary reconstruct filtering then handles the edge that extracts.Through consider the frequency of occurrences, similarity, near three measuring methods of property, definition cost function J _Recon(k)=J _F(k)+J _S(k)+J _C(k), the neighbor cost value to the edge calculations current pixel that extracts finds the best brightness value with maximum cost to replace the brightness value of current pixel.Wherein, the brightness value of k represent pixel, J _ReconCost function when the represent pixel value is k.First filial generation valency function J _FRepresent the frequency of occurrences of each pixel, it is obtained by the occurrence number of pixel, defines as follows:

$N_{\mathrm{OC}}\left(k\right)=\underset{i=0}{\overset{n\×n-1}{\mathrm{\Σ}}}\mathrm{\δ}[k,w_{n\×n}\left(i\right)],$

$J_F\left(k\right)=\frac{N_{\mathrm{OC}}\left(k\right)-N_{\mathrm{OC}}\left(\min \right)}{N_{\mathrm{OC}}\left(\max \right)-N_{\mathrm{OC}}\left(\min \right)}$

Wherein, w _{N * n}(i) represent window w _{N * n}The brightness value of middle pixel i, N _OC(max) and N _OC(min) represent minimum and maximum N respectively _OCValue.

Second sub-cost function J _SMean current pixel point I _CurBe adjacent pixel I _kBetween the similarity of brightness, definition as follows:

S(k)＝|I _cur-I _k|

$J_S\left(k\right)=\frac{S\left(\max \right)-S\left(k\right)}{S\left(\max \right)-S\left(\min \right)}$

Wherein, S (max) and S (min) represent minimum and maximum S (k) value respectively.

The 3rd sub-cost function J _CRepresent current pixel (x _Cur, y _Cur) and neighbor (x _i, y _i) between distance feature, the definition as follows:

$C\left(k\right)=\frac{1}{N_{\mathrm{OC}}\left(k\right)}\underset{i=0}{\overset{N_{\mathrm{OC}}\left(k\right)-1}{\mathrm{\Σ}}}\sqrt{{(x_{\mathrm{cur}}-x_i)}^2+{(y_{\mathrm{cur}}-y_i)}^2}$

$J_C\left(k\right)=\frac{C\left(\max \right)-C\left(k\right)}{C\left(\max \right)-C\left(\min \right)}$

Wherein, C (max) and C (min) represent minimum and maximum C (k) value respectively.

At last,, utilize medium filtering and morphology to handle the edge of depth of smoothness figure, thereby obtain quality depth map preferably for the depth map after the smooth boundary reconstruct.

(2) utilize two viewpoint reference-view and corresponding depth map thereof, obtain the virtual view view through the 3-D view conversion;

When drawing virtual visual point image; Adopt 3D Image Warping method; 3D image warping has described the transformation relation of the respective pixel of the projection of any three dimensions point on different imaging planes, utilizes depth image, and the pixel in the reference picture is mapped in the target image.At first the pixel of source reference image is utilized the three dimensions point position of the camera parameter re-projection of its depth information and reference camera to their correspondences; According to the position of virtual view and virtual view camera parameter these three dimensions points are projected to the virtual camera plane more then and be carried out to picture, obtain virtual visual point image.Specific algorithm is realized as follows:

$\left\{\begin{array}{c}{P}_{11}^{r}X+P_{12}^{r}Y+P_{13}^{r}Z+P_{14}^r=\mathrm{wu}\\ P_{21}^{r}X+P_{22}^{r}Y+P_{23}^{r}Z+P_{24}^r=\mathrm{wu}\\ P_{31}^{r}X+P_{32}^{r}Y+P_{33}^{r}Z+P_{34}^r=w\end{array}\right.$

$\left\{\begin{array}{c}{u}_v=\frac{P_{11}^{v}X+P_{12}^{v}Y+P_{13}^{v}Z+P_{14}^v}{P_{31}^{v}X+P_{32}^{v}Y+P_{33}^{v}Z+P_{34}^v}\\ v_v=\frac{P_{21}^{v}X+P_{22}^{v}Y+P_{23}^{v}Z+P_{24}^v}{P_{31}^{v}X+P_{32}^{v}Y+P_{33}^{v}Z+P_{34}^v}\end{array}\right.$

Wherein, (u, v) ^TBe the pixel coordinate in the reference picture coordinate system, (u _v, v _v) ^TBe the coordinate points in the virtual view that obtains, w is a zoom factor, (X, Y, Z, 1) ^TFor the coordinate points in the world coordinate system (X, Y, Z) ^TCorresponding homogeneous coordinates point,

(i=1,2,3; J=1,2,3,4) be the element in the reference view projection matrix,

(i=1,2,3; J=1,2,3,4) be the element in the virtual view projection matrix, projection matrix P is by confidential reference items matrix K (3 * 3 matrixes), and rotation matrix R (3 * 3 matrixes) and translation matrix t (3 * 1 vectors) unite and obtain.

In the process of carrying out 3D warping, exist a plurality of reference views to be mapped to the phenomenon of same virtual visual point image position; Can cause the pixel of back projection to cover the pixel of first projection; Thereby caused serious distortion for the virtual view that generates, choose the pixel of the less pixel of depth value as needs in order to address this is that the method that adopts Z-Buffer to compare depth value.

(3) two width of cloth virtual visual point images are carried out image co-registration and eliminate most of cavity, generate a width of cloth initial virtual view;

Obtain in the process of virtual visual point image in the purpose image, becoming visible because the relation of mapping makes sightless zone in reference picture in reference picture mapping, these zones can not find corresponding Pixel Information in reference picture.Therefore, utilize distance weighted value to merge two width of cloth virtual views, thus the initial virtual view that obtains merging.Fusion formula is following:

Wherein,

(u v) is virtual visual point image coordinate (u, the pixel value of v) locating, I to I _L, I _RThe reference picture of representing two contiguous cameras respectively, (u _L, v _L) and (u _R, v _R) be respectively virtual visual point image pixel (u, v) corresponding point on the target image that generates respectively by two reference pictures, Z _RAnd Z _LRepresent I respectively _RMiddle coordinate is (u _R, v _R) and I _LMiddle coordinate is (u _L, v _L) depth value.

(4) utilization of initial virtual view is carried out hole-filling based on the method for background separation, obtain final virtual visual point image;

Image co-registration has been eliminated the most of cavity in the virtual view, but some cavity all is positioned at invisible area in reference picture, use the method for image co-registration not eliminate.In order to carry out high-quality hole-filling, avoid foreground pixel is filled on the background, design following hole-filling method:

At first, utilize the depth value of the depth map that merges the back virtual view, choose threshold value γ, depth map is divided, be about to background area and foreground object and be separated from each other.Then, fill through the method for neighborhood interpolation isolated background area, so just can the background area of former hollow sectors before this utilized the pixel of adjacent background area fill.At last, foreground area is reset in the virtual view,, utilizes the method for neighborhood interpolation that whole virtual view is carried out hole-filling again, obtain final virtual view for guaranteeing that all cavities are all filled.

For the effect that makes the embodiment of the invention has comparability; Adopt cam3 and two viewpoints of cam5 of Ballet database; Draw out the image of cam4 viewpoint; The part amplification of the virtual view view of process embodiment of the invention method drafting is not as shown in Figure 1, and the part amplification of the virtual view view that process embodiment of the invention method is drawn is as shown in Figure 2, and the virtual view view full figure of process embodiment of the invention method drafting is as shown in Figure 3.

Fig. 1 is not for using the inventive method, and the part of the virtual visual point image of directly drawing is amplified, and marked region exists foreground pixel to be filled into the phenomenon on the background among the figure, causes the visual effect of virtual view to descend.

Fig. 2 amplifies for the part of the virtual visual point image that use the inventive method is drawn, and marked region has been avoided foreground pixel is filled into the phenomenon on the background among the figure, and the rendering quality of virtual view is better.

The virtual view view full figure that Fig. 3 is to use the inventive method to draw, visible from figure, obtained virtual viewpoint rendering effect preferably.

Claims (4)

1. a virtual viewpoint rendering method is characterized in that, comprises the following steps:

(1) adopt border restructing algorithm that depth map is carried out pre-service based on rim detection;

(2) utilize two viewpoint reference-view and corresponding depth map thereof, obtain the virtual view view through the 3-D view conversion:

When drawing virtual visual point image; Adopt 3D Image Warping method; Utilize the camera parameter of depth image and demarcation; Pixel in the reference picture is mapped in the target image; The camera parameter re-projection that at first pixel of source reference image is utilized its depth information and reference camera is to their corresponding three dimensions point positions, utilizes the camera parameter of position and the virtual view of virtual view that these three dimensions points are projected to the virtual camera plane more then and is carried out to picture, obtains virtual visual point image;

(3) two width of cloth virtual visual point images are carried out image co-registration and eliminate most of cavity, generate a width of cloth initial virtual view:

Utilize distance weighted value to merge two width of cloth virtual views, thus the initial virtual view that obtains merging;

(4) utilization of initial virtual view is carried out hole-filling based on the method for background separation, obtain final virtual visual point image.

2. the method for claim 1 is characterized in that, the said employing in the step (1) is carried out pre-service based on the border restructing algorithm of rim detection to depth map and is specially:

At first utilize the canny operator to carry out rim detection, extract the edge of foreground object in the depth map, carry out depth boundary reconstruct filtering then the edge that extracts is handled, adopt following cost function: J depth map _Recon(k)=J _F(k)+J _S(k)+J _C(k),, find best brightness value to replace the brightness value of current pixel with maximum cost to the neighbor cost value of the edge calculations current pixel that extracts, wherein, the brightness value of k represent pixel, J _ReconCost function when the represent pixel value is k, first filial generation valency function J _FRepresent the frequency of occurrences of each pixel, it is obtained by the occurrence number of pixel; Second sub-cost function J _SRepresent current pixel point to be adjacent the similarity of the brightness between the pixel; The 3rd sub-cost function J _CRepresent between current pixel and the neighbor near degree, last, for the depth map after the smooth boundary reconstruct, utilize medium filtering and morphology to handle the edge of depth of smoothness figure, thereby acquisition quality depth map preferably.

3. the method for claim 1 is characterized in that, said in the step (4) carries out hole-filling to the utilization of initial virtual view based on the method for background separation, obtains final virtual visual point image and is specially:

At first utilize the depth value of the depth map that merges the back virtual view, choose threshold value γ, depth map is divided, be about to background area and foreground object and be separated from each other; Fill through the method for neighborhood interpolation isolated then background area; At last foreground area is reset in the virtual view,, utilizes the method for neighborhood interpolation that whole virtual view is carried out hole-filling again, obtain final virtual view for guaranteeing that all cavities are all filled.

4. the method for claim 1 is characterized in that, the said border restructing algorithm cost function in the step (1) is specially: first filial generation valency function J _FDefine as follows:

$N_{\mathrm{OC}}\left(k\right)=\underset{i=0}{\overset{n\×n-1}{\mathrm{\Σ}}}\mathrm{\δ}[k,w_{n\×n}\left(i\right)],$

$J_F\left(k\right)=\frac{N_{\mathrm{OC}}\left(k\right)-N_{\mathrm{OC}}\left(\min \right)}{N_{\mathrm{OC}}\left(\max \right)-N_{\mathrm{OC}}\left(\min \right)}$

Wherein, w _{N * n}(i) represent window w _{N * n}The brightness value of middle pixel i, N _OC(max) and N _OC(min) represent minimum and maximum N respectively _OCValue;

Second sub-cost function J _SDefine as follows:

S(k)＝|I _cur-I _k|

$J_S\left(k\right)=\frac{S\left(\max \right)-S\left(k\right)}{S\left(\max \right)-S\left(\min \right)}$

Wherein, S (max) and S (min) represent minimum and maximum S (k) value respectively;

The 3rd sub-cost function J _CDefine as follows:

$C\left(k\right)=\frac{1}{N_{\mathrm{OC}}\left(k\right)}\underset{i=0}{\overset{N_{\mathrm{OC}}\left(k\right)-1}{\mathrm{\Σ}}}\sqrt{{(x_{\mathrm{cur}}-x_i)}^2+{(y_{\mathrm{cur}}-y_i)}^2}$

$J_C\left(k\right)=\frac{C\left(\max \right)-C\left(k\right)}{C\left(\max \right)-C\left(\min \right)}$

Wherein, C (max) and C (min) represent minimum and maximum C (k) value respectively.

Images