CN106791770A - A kind of depth map fusion method suitable for DIBR preprocessing process - Google Patents

Abstract

The invention discloses a kind of depth map fusion method for DIBR preprocessing process, comprise the following steps：Gradient analysis is carried out to original depth-map, the hole region in new viewpoint image is estimated out；Initial effects degree distribution map is generated according to hole region distribution is estimated；Initial effects degree distribution map is diffused using texture image under dimension transformation space；The depth map of original depth-map and pre-smoothed is merged based on the disturbance degree distribution map after diffusion, the depth map after generation optimization.The method of the present invention is under efficient dimension transformation space, merged come the depth map to original depth-map and pre-smoothed by the disturbance degree distribution map spread based on textural characteristics, so as to realize self-adaptative adjustment of the depth map after optimization in different zones smoothed intensity, the smoothing effect of hole region is not only effectively retained to eliminate empty influence, and effectively prevent the extra twisted phenomena caused to occurring without smoothing excessively for hole region.

Description

A kind of depth map fusion method suitable for DIBR preprocessing process

Technical field

It is particularly a kind of to be applicable the invention belongs to 3 D video technical field, and in particular to the Video Quality Metric technology of 2D/3D In the depth map fusion method of DIBR preprocessing process.

Background technology

At present, three-dimensional (3D) video is gradually popularized, and Chinese Central Television (CCTV) also pilots when New Year's Day in 2012 3D channels, 3D videos have been increasingly becoming a kind of trend of current development.However, not enough the turning into of video source restricts this product The Main Bottleneck that industry is risen.In this case, 2D videos are switched into the effective way that 3D videos are this problems of solution.

2D videos are switched into 3D videos and generally speaking renders mode in the presence of two kinds：One of which is by using some way The right and left eyes image pair with parallax is directly reconstructed out from single frame of video；Another kind is rendering based on depth map (Depth Image-based Rendering, DIBR), its transformation result is that each frame is addition of on the basis of former video Corresponding depth map, being finally converted to after binocular tri-dimensional video by the display terminal output of embedded DIBR processing modules just can be with Viewed and admired (referring to " film 2D/3D switch technologies summarize [J] ", Liu Wei, Wu Yihong, Hu Zhanyi,《CAD with Graphics journal》, 2012,24 (1):14-28).Compared with the former, three original features that the latter has with it：Efficient pressure The depth of field that the compatibility of contracting efficiency of transmission and existing 2D technologies and distinct device has by force and in real time tridimensional video generation The technical advantage such as adjustment and Fast rendering synthesis, in the absolute leading position of market share such as emerging 3DTV, 3D mobile terminal, It is the direction of 3D Rendering future developments.

It is the important step being based in the 2D/3D conversion methods of depth map that DIBR is rendered, and it can utilize depth information wash with watercolours Virtual three-dimensional video-frequency is dyed, so as to be finally completed 2D to 3D " fundamental change ".Although this technology has many advantages, Still there is its limitation.Because the mapping relations that DIBR is changed from reference picture according to depth map fictionalize right and left eyes image, The change of viewpoint may cause the part background area blocked by foreground object in original image to be exposed in new images, and this Subregion does not have corresponding texture mapping in conversion process, therefore cavitation will be produced on target image.This Problem is DIBR technologies study hotspot in recent years, is also the importance for improving 3D rendering quality.It is current for this problem Frequently with handling process as shown in Figure 1, link is pre-processed and after DIBR based on figure by adding depth map before DIBR Filling up for cavity is completed as recovery technique.

Depth map pretreatment is typically with all kinds of wave filters and depth map is smoothed, and so draws the new viewpoint for obtaining In will be conducive to further filling up comprising less cavity.This kind of method operational efficiency is high, and effect is obvious, but smooth filter The object edges areas (the especially edge of vertical direction) that ripple may result in virtual image produce geometric deformation.Therefore exist Depth map pretreatment link cannot be effectively ensured undistorted turn of virtual image in existing DIBR technologies in 2D/3D Video Quality Metrics Synthesis is changed, so as to have impact on the actual converted effect of 3D videos.

The content of the invention

In view of this, the purpose of the present invention is directed to the deficiency of existing DIBR depth maps preconditioning technique link, by dimension The disturbance degree distribution map that generation is spread under degree transform domain is merged come the depth map to original depth-map and pre-smoothed, so that Self-adaptative adjustment of the depth map after optimization in different zones smoothed intensity is realized, the smooth of hole region has not only been effectively retained Effect is showed with eliminating empty influence, and effectively prevent the extra distortion caused to occurring without smoothing excessively for hole region As from lifting 3D virtual image rendering effects.

To reach above-mentioned purpose, the present invention uses following technical scheme：

A kind of depth map fusion method suitable for DIBR preprocessing process, comprises the following steps：

A) in original depth-map D_OriIn, change in depth rule of the operator according to right and left eyes virtual image is estimated to producing using cavity Raw hole region R_HEstimated；

B) generation initial effects degree distribution map I_f-init；

C) to initial effects degree distribution map I under dimension transform domain_f-initIt is diffused, the disturbance degree distribution map after generation optimization I_f；

D) with the disturbance degree distribution map I after optimization_fTo original depth-map D_OriDepth map with pre-smoothed carries out fusion D_Pre, it is raw Into the depth map D after optimization_Fin。

Wherein, step A) described in cavity estimate operator and be：

Wherein, R_HThe cavity of prediction is represented, r (x, y) is represented in original depth-map D_OriThe depth value at middle coordinate (x, y) place, D_max Be setting generation virtual image parallax maximum number of pixels, α be normalization factor (in 8bit gray level images, α= 255), D_widthIt is the number of pixels of picture traverse, λ_HIt is default threshold factor；If the virtual view of new synthesis is that left eye is regarded Scheme, then i=l, otherwise, i=r.

Wherein, step B) in initial effects degree distribution map I_f-initIt is specifically defined as：

Wherein, R_HRepresent the cavity of prediction, D_eP () represents point p to the distance for estimating empty edge.

Wherein, step C) under dimension transform domain to initial effects degree distribution map I_f-initIt is diffused specially:Diffusion letter Number is defined as follows：

I_f[n]=(1-a^d)I_f-init[n]+a^dI_f[n-1]

Wherein, I_f-init[n] represents the pixel value of initial effects degree distribution map lastrow or a row, and a ∈ (0,1) are spread functions Feedback factor, d represents adjacent sample x in dimension transform domain_nAnd x_n-1The distance between.

Wherein, adjacent sample x in dimension transform domain_nAnd x_n-1The distance between be defined as：

D=ct (x_n)-ct(x_n-1)

Wherein, ct (u) represents dimension transform domain, and dimension conversion process is:

Wherein, I_textureX () represents the texture image of input, | I '_texture(x) | represent the gradient intensity of texture image, σ_sAnd σ_r It is respectively transmission device space and codomain parameter, for adjusting the influence of propagation.σ_sSpan is 200~2500, σ_rSpan It is 0.1~10.

Wherein, be diffused as iterative process, and to realize symmetric propagation, if spread in an iteration according to from a left side to The right side, order from top to bottom is propagated in the picture, then spread in next iteration and passed according to order from right to left, from top to bottom Broadcast.Iterations is 2~10 times.

Wherein, step D) in depth map fusion formula be：

D_Fin=I_fD_Pre+(1-I_f)D_Ori。

The beneficial effects of the invention are as follows：Disturbance degree distribution map in the present invention is based on line under efficient dimension transformation space Reason feature and the original cavity diffusion of prediction, can reflect the distributed intelligence of structuring.Depth integration is carried out with this, can be with Traditional Man parameter limiting mode is replaced to determine that the filtering for strengthening area to be repaired is imitated come more efficient with self adaptation diffusion way Really, and the filter effect for reducing non-hole region is excluded, so as to overcome the depth map in conventional depth figure preprocessing process to distort With smoothing problasm excessively, the virtual rendering effects of 3D are significantly improved while realizing that cavity is repaired.

Brief description of the drawings

Fig. 1 is existing DIBR systems process chart；

Fig. 2 is flow chart of the method for the present invention；

Fig. 3 is the depth map and virtual image effect contrast figure using the inventive method.

Specific embodiment

The invention will be further described with reference to the accompanying drawings and examples：

Fig. 1 shows existing DIBR systems handling process, for the original depth-map being input into, first in pretreatment link to depth Degree figure carries out smothing filtering, reduces the generation in most of cavity when rendering by the optimization of depth map part-structure with this；Then, Processing method is rendered using DIBR, using depth image and the camera parameter demarcated, the pixel in reference picture is mapped to In target image；Finally, using hole-filling method, a small amount of cavity retained in render process is repaired, and export what is changed out Right and left eyes virtual image.

Wherein, it is the important step in 2D/3D conversion methods that DIBR renders treatment, and it describes an accurate point-to-point Mapping relations, it is possible to use depth information renders virtual three-dimensional video-frequency, so as to be finally completed 2D to 3D " fundamental change ". Although this technology there are many advantages, still there is its limitation.Because DIBR turns from reference picture according to depth map The mapping relations changed fictionalize right and left eyes image, and the change of viewpoint may cause the part blocked by foreground object in original image to be carried on the back Scene area is exposed in new images, and this subregion does not have corresponding texture mapping in conversion process, therefore will Cavitation is produced on target image.This problem is DIBR technologies study hotspot in recent years, is also to improve 3D rendering matter The importance of amount.

There are three class solutions for what this problem was commonly used at present：

1) depth of seam division video (LDV) form.Such method by new data Layer fundamentally solve due to block and The cavitation produced in depth map.But using special equipment during this technical requirements video acquisition, so being not appropriate for In 2D/3D conversions；

2) hole-filling.Such method is the hole-filling link after the DIBR shown in Fig. 1.The main thought of such method It is the textural characteristics according to image, chooses sizeable texture block, most phase therewith is then found around region to be repaired Near Texture Matching block substitutes the texture block.This kind of method can repair large area region cavity, but when repairing block matching Based on greedy search, obvious reparation mistake is likely to result in.In addition, such method amount of calculation is larger, therefore often auxiliary Depth map pre-processes link to repair a small amount of cavity of reservation；

3) pretreatment of depth image, such method is the link before the DIBR shown in Fig. 1.This kind of method energy depth of smoothness So as to reduce the cavity in depth map, the intensity for increasing gaussian filtering can be improved in discontinuous (depth acute variation) region in figure Generate the quality of stereotome.Because this kind of method computational efficiency is higher, can be eliminated in advance by the local optimum of depth map The major part cavity being likely to occur, only comprising less cavity, is conducive to further filling up, therefore is cavity in DIBR systems The important step filled up.But on the other hand filtering easily causes the torsional deformation of object rectilinear direction fringe region.Although mesh Before asymmetric smothing filtering and bilateral filtering have been proposed alleviate this problem, but these global parameters set wave filter Still local excessively smooth phenomenon fully effective cannot be avoided, when local smoothing effect is excessive, the new of synthesis is still resulted in The subregion object of viewpoint view produces geometric deformation.

Therefore the depth map pretreatment link in existing DIBR systems in 2D/3D Video Quality Metrics still cannot have completely Effect ensures the undistorted conversion synthesis of virtual image, so as to have impact on the actual converted effect of 3D videos.Therefore, the inventive method Secondary fusion is carried out to pretreated depth map and original depth-map by introducing disturbance degree distribution map, self adaptation is realized Part filter, carries out DIBR and renders with the depth map after further being optimized.

The inventive method is the pre-smoothed obtained with texture image, original depth image and by certain filtering method Depth map generates the depth map after fusion optimization after treatment as the data source of input.Fig. 2 is method of the present invention flow Figure, is described with reference to Fig. 2 to specific embodiment of the invention.

If smooth discontinuous (depth acute variation) region that may only occur in cavity of depth map is carried out, then non- The depth plot quality of hole region can be just effectively maintained, and crossing smoothing effect also can preferably be suppressed.It is based on This thought, the present invention proposes a kind of new depth map fusion method, adaptive to realize by the disturbance degree distribution map for spreading The local smoothing method answered, specifically includes following steps：

A) in original depth-map D_OriIn, change in depth rule of the operator according to right and left eyes virtual image is estimated to producing using cavity Raw hole region R_HEstimated.If defining in depth map the distance nearer apart from observer with the larger value of numerical value come table Show, represented with the less value of numerical value apart from observer's larger distance.So specifically, it is empty in left eye virtual image Hole concentrates on the region of the ascending acute variation of depth value；In right eye virtual image, cavity is concentrated in depth value by big To the region of small acute variation.Based on this, cavity estimates operator definitions and is：

Wherein, R_HThe cavity of prediction is represented, r (x, y) is represented in original depth-map D_OriThe depth value at middle coordinate (x, y) place, D_max Be setting generation virtual image parallax maximum number of pixels, α be normalization factor (such as in 8bit gray level images, α= 255), D_widthIt is the number of pixels of picture traverse, λ_HIt is default threshold factor, span is 1~5, λ in emulation experiment_H Take 2；If the virtual view of new synthesis is left-eye view, i=l, otherwise, i=r；

B) generation initial effects degree distribution map I_f-init.To eliminate cavity influence, the disturbance degree distribution map of present invention definition is around pre- The hole region estimated is launched.Based on this, initial effects degree distribution map I_f-initIt is specifically defined as：

Wherein, R_HRepresent the cavity of prediction, D_eP () represents point p to the distance for estimating empty edge.From D_eP the definition of () can be seen Go out, closer to the region of hollow center, the depth map of its corresponding pre-smoothed is bigger to eliminating empty effect；In non-NULL Hole region, the depth map of its corresponding pre-smoothed is to eliminating empty effect very little；

C) to initial effects degree distribution map I under dimension transform domain_f-initIt is diffused；

Because initial effects degree distribution map only only accounts for the direct distribution in cavity, if there is edge line to pass through in virtual image Near cavity, then may have influence on edge to empty smoothing and be allowed to produce deformation.In order to solve this problem, it is necessary to right Initial effects degree distribution map carries out appropriate diffusion according to the textural characteristics of image, and nearby structure similar area has phase to make cavity As disturbance degree, the generation of deformation is overcome while local smoothing method is realized with this；

Spread function is defined as follows：

I_f[n]=(1-a^d)I_f-init[n]+a^dI_f[n-1]

Wherein, I_f-init[n] represents the pixel value of initial effects degree distribution map lastrow or a row, and a ∈ (0,1) are spread functions Feedback factor, d=ct (x_n)-ct(x_n-1) represent adjacent sample x in dimension transform domain_nAnd x_n-1The distance between.Here dimension Degree transform domain is in article " Domain transform for edge- with Eduardo S.L.Gastal in 2011 et al. The transformation space that the method proposed in aware image and video processing " is obtained, its sharpest edges be Hyperspace is being reduced to the one-dimensional space on the premise of ensure that image texture characteristic, so as to substantially increase computational efficiency. Specifically, ct (u) represents dimension transform domain, and dimension conversion process is:

Wherein, I_textureX () represents the texture image of input, | I '_texture(x) | represent the gradient intensity of texture image, σ_sAnd σ_r It is respectively transmission device space and codomain parameter, for adjusting the influence of propagation.σ_sSpan is 200~2500, σ_rSpan It is 0.1~10；

It can be seen that, in dimension conversion process, the scene structure feature reflected in texture image is taken into account, and becomes just The important evidence of beginning disturbance degree distribution map diffusion.The similar two-sided filter of whole diffusion process effect, disturbance degree distribution map with The scene characteristic of image is propagated further diffusion near cavity, but due to the reduction of dimension under dimension transformation space, it Operation efficiency is far longer than traditional two-sided filter, and traditional two-sided filter is run under two bit spaces, above-mentioned fixed Although the dimension conversion process of justice substantially increases operation efficiency, but wave filter simply under the one-dimensional space.It is same in order to reach Effect, in the particular embodiment, diffusion is realized with the mode of iteration.Again because dimension conversion process defined above is non- Symmetrically, so to realize symmetric propagation, if spread in an iteration according to from left to right, order from top to bottom is in image Middle propagation, then spread according to sequence spread from right to left, from top to bottom in next iteration.Iterations is 2~10 times, one As 3 diffusion effects of iteration can reach stabilization, in emulation experiment iterations be 3 times；

D) with the disturbance degree distribution map I after optimization_fTo original depth-map D_OriDepth map with pre-smoothed carries out fusion D_Pre, it is raw Into the depth map D after optimization_Fin.Depth map fusion formula is：

D_Fin=I_fD_Pre+(1-I_f)D_Ori

Using the disturbance degree distribution map after diffusion, the depth map and original depth-map after smooth are carried out into secondary fusion, retained Under favourable smooth region is eliminated to cavity.Due to disturbance degree distribution map I_fCodomain may change in diffusion process, because This needs to normalize to region [0,1] when implementing；

Depth map after fusion optimization has inhibited a large amount of cavities to produce, so last carry out DIBR wash with watercolours with it and texture image Dye, then by filling up the cavity of a small amount of reservation using simple interpolation method, it is possible to generate right and left eyes virtual image.

It is below the experimental verification of DIBR virtual images restorative procedure of the invention；

1) experiment condition：

It is in CPUCore^TM2 Quad CPU Q9400@2.66GHz, are carried out in the system of internal memory 4G, Windows 7 Experiment；

2) experiment content：

To specifically describe realize details according to the experiment of the inventive method and render quality to 3D and bring referring to Fig. 3 and carry Rise.

Fig. 3 is situation when processing one group of experimental image.Wherein, Fig. 3 (a) is original texture image, here we It directly as right eye virtual image, then need to generate left eye virtual image by DIBR methods.Fig. 3 (b) show not The left eye virtual image for obtaining directly is rendered with DIBR by depth map pretreatment and hole-filling, in newly-generated virtual image Black discontinuously part is cavity, it is seen that cavity is mainly distributed in Fig. 3 (c) in original depth-map depth value from small to large The foreground object left side edge of drastic change.Fig. 3 (d) is the depth map of the pre-smoothed obtained by bilateral filtering, it is seen that after smooth The region of depth value drastic change is significantly less in depth map, therefore, it is possible to substantially suppress the generation in cavity.But such as Fig. 3 B shown in (), cavity is concentrated mainly on foreground object left side edge, therefore smoothing to cavity on the right side of foreground object in Fig. 3 (d) Elimination has no obvious effect.The inventive method propose diffusion after disturbance degree distribution map such as Fig. 3 (e) shown in, it is seen that it divide , with hole region as core, the architectural feature that can reflect according to image is diffused for cloth and intensity.For example in image man is left Texture in the background of side on wall is vertically distributed, and the background shutter texture on the left of Ms is distributed in the horizontal direction, Embodiment well is obtained in these features disturbance degree distribution map after the diffusion.Fig. 3 (f) is by original depth based on Fig. 3 (e) Depth map after the optimization obtained after the secondary fusion of depth map Fig. 3 (d) of degree figure Fig. 3 (c) and pre-smoothed, with Fig. 3 (d) phases Smooth region is crossed than, it can be seen that foreground object right side to be suppressed, and the smooth effect of hole region is then according to around scheming As feature is effectively maintained.Fig. 3 (g) and Fig. 3 (h) are respectively to be input into obtain as depth map with Fig. 3 (d) and Fig. 3 (f) DIBR rendering results, it can be seen that non-hole region crosses oval tag slot of the smoothing effect in Fig. 3 (g) in Fig. 3 (d) More obvious pattern distortion is generated in domain, and the depth map after being processed through the inventive method in Fig. 3 (h) renders image then not There are problems that this.As can be seen here depth map fusion method proposed by the present invention in DIBR depth map preprocessing process to improve The castering action that 3D visual effect quality is played.

Finally illustrate, the above embodiments are merely illustrative of the technical solutions of the present invention and it is unrestricted, this area is common Other modifications or equivalent that technical staff is made to technical scheme, without departing from technical solution of the present invention Spirit and scope, all should cover in the middle of scope of the presently claimed invention.

Claims (7)

1. a kind of depth map fusion method for DIBR preprocessing process, it is characterised in that：Comprise the following steps：

A) in original depth-map D_OriIn, change in depth rule of the operator according to right and left eyes virtual image is estimated to producing using cavity Raw hole region R_HEstimated；

B) generation initial effects degree distribution map I_f-init；

C) to initial effects degree distribution map I under dimension transform domain_f-initIt is diffused, the disturbance degree distribution map after generation optimization I_f；

D) with the disturbance degree distribution map I after optimization_fTo original depth-map D_OriDepth map with pre-smoothed carries out fusion D_Pre, it is raw Into the depth map D after optimization_Fin。

2. a kind of depth map fusion method for DIBR preprocessing process as claimed in claim 1, it is characterised in that step A operator is estimated in the cavity described in)：

$R_H=\left\{r(x,y)\right|{\mathrm{\η}}_i(x,y)-r(x,y)>\mathrm{\α}\·\frac{{\mathrm{\λ}}_H\·D_{max}}{D_{width}}\}$

${\mathrm{\η}}_i(x,y)=\left\{\begin{array}{cc}r(x+1,y)& ifi=l\\ r(x-1,y)& ifi=r\end{array}\right.$

Wherein, R_HThe cavity of prediction is represented, r (x, y) is represented in original depth-map D_OriThe depth value at middle coordinate (x, y) place, D_maxIt is The number of pixels of setting generation virtual image parallax maximum, α be normalization factor (in 8bit gray level images, α=255), D_widthIt is the number of pixels of picture traverse, λ_HIt is default threshold factor；If the virtual view of new synthesis is left-eye view, I=l, otherwise, i=r.

3. a kind of depth map fusion method for DIBR preprocessing process as claimed in claim 1, it is characterised in that step B the initial effects degree distribution map I described in)_f-initIt is specifically defined as：

$I_{f-init}\left(p\right)=\left\{\begin{array}{cc}1-\exp (-D_e(p\left)\right)& ifx\∈R_H\\ 0& ifx\∉R_H\end{array}\right.$

Wherein, R_HRepresent the cavity of prediction, D_eP () represents point p to the distance for estimating empty edge.

4. a kind of depth map fusion method for DIBR preprocessing process as claimed in claim 1, it is characterised in that step C described in) under dimension transformation space to initial effects degree distribution map I_f-initIt is diffused specially:Spread function is defined such as Under：

I_f[n]=(1-a^d)I_f-init[n]+a^dI_f[n-1]

Wherein, I_f-init[n] represents the pixel value of initial effects degree distribution map lastrow or a row, and a ∈ (0,1) are spread functions Feedback factor, d represents adjacent sample x in dimension transform domain_nAnd x_n-1The distance between.

5. a kind of depth map fusion method for DIBR preprocessing process as claimed in claim 4, it is characterised in that

Adjacent sample x in the dimension transform domain_nAnd x_n-1The distance between be defined as：

D=ct (x_n)-ct(x_n-1)

Wherein, ct (u) represents dimension transform domain, and dimension conversion process is:

$ct\left(u\right)={\∫}_0^{t}1+\frac{{\mathrm{\σ}}_s}{{\mathrm{\σ}}_r}\left|I_{texture}^{\′}\left(x\right)\right|dx$

Wherein, I_textureX () represents the texture image of input, | I '_texture(x) | represent the gradient intensity of texture image, σ_sAnd σ_r It is respectively transmission device space and codomain parameter, for adjusting the influence of propagation, σ_sSpan is 200~2500, σ_rSpan It is 0.1~10.

6. a kind of depth map fusion method for DIBR preprocessing process as described in claim 1 or claim 4, it is special Levy and be,

It is described to be diffused as iterative process, and to realize symmetric propagation, if spread in an iteration according to from left to right, from upper Order under is propagated in the picture, then spread according to sequence spread from right to left, from top to bottom, iteration in next iteration Number of times is 2~10 times.

7. a kind of depth map fusion method for DIBR preprocessing process as claimed in claim 1, it is characterised in that step D the depth map fusion formula described in) is：

D_Fin=I_fD_Pre+(1-I_f)D_Ori。