CN103955954A - Reconstruction method for high-resolution depth image in combination with space diagram pairs of same scene - Google Patents

Abstract

The invention relates to a reconstruction method for a high-resolution depth image in combination with space diagram pairs of the same scene. More and more applications depend on accurate and rapid observation and analysis of depth images of a real scene. A flight time depth camera can obtain the depth images of the scene in real time; however, due to limitation of hardware conditions, collected depth images are low in resolution and can not meet practical application requirements. A stereo matching algorithm is a classic method for obtaining the depth images; however, due to blocking between the left image and the right image and the influence of a texture-free region, the stereo matching algorithm has a large limitation in practical application. According to the reconstruction method, the advantages of the flight time depth camera and the advantages of the stereo matching algorithm are made full use of, the reconstruction method for the high-resolution depth image combines the flight time depth camera and the space diagram pairs of the same scene, the defects in the prior art can be well overcome, and the high-resolution and high-quality depth image can be reconstructed.

Description

A kind of combination is with the right high resolving power depth image method for reconstructing of scene stereographic map

Technical field

The invention belongs to computer vision field, be specifically related to a kind of combination with the right high resolving power depth image method for reconstructing of scene stereographic map.

Background technology

The depth image that obtains scene is the vital task of computer vision.Each pixel value in depth image represents is point in scene and the distance between camera.Increasing application as three-dimensional reconstruction, collision detection, gesture identification, robot navigation, industrial automation and at film and in playing to the design setting model of virtual scene etc., all depend on real scene depth image observation and analysis accurately.At present, the obtaining means of depth image mainly contains: 1) method by Stereo matching calculates depth image; 2) by direct surveying instrument observation, obtain depth image.

Thereby Stereo Matching Algorithm is calculated the depth image that the parallax relation of each corresponding point between the image of left and right obtains scene, Stereo Matching Algorithm is a class classical way that obtains scene depth image.But owing to blocking between the image of left and right and without the impact of texture region, Stereo Matching Algorithm exists significant limitation in actual applications.Flight time depth camera is the major equipment of image of directly fathoming, it is by sending light pulse to scene, use high-speed shutter to calculate the distance of determining object in scene two-way time of light pulse, time-of-flight camera can obtain the depth information of view picture scene fast.But the scene depth figure resolution obtaining due to the restriction time-of-flight camera of hardware is lower, be difficult to meet the needs of practical application.

Summary of the invention

Object of the present invention overcomes the deficiencies in the prior art exactly, has proposed a kind of combination with the right high resolving power depth image method for reconstructing of scene stereographic map.The low resolution depth image that this method collects in conjunction with flight time depth camera and with the high-resolution color stereographic map of scene to rebuilding the high-quality depth image of high resolving power.Concrete steps are:

Step (1) is constructed non local filter weights, and the low resolution depth image of input is carried out to filtering:

G represents the low resolution depth image that flight time depth camera collects, and size is n * m; L represents the left image of same scene high-resolution color of the left collected by camera of CCD, and size is rn * rm; R represents the right image of high-resolution color of the corresponding right collected by camera of CCD, and size is rn * rm;

Construct as follows non local filter weights:

$w_{\mathrm{ij}}^N=\exp (-\frac{{\left|\right|M_{i}L-M_{j}L\left|\right|}_2^2}{K\·h_L^2})\·\exp (-\frac{{\left|\right|M_{i}I-M_{j}I\left|\right|}_2^2}{K\·h_I^2})$

Wherein I is that the low resolution depth image of inputting carries out the result after bilinear interpolation, and size is rn * rm; M _il represents with L _icentered by localized mass, M _ii represents with I _icentered by localized mass, K is the pixel number that localized mass contains, h _l, h _ibe the parameter of wave filter, be used for controlling the rate of decay of the exponential expression of weighted calculation;

By above-mentioned, calculate non local filter weights , by following formula, bilinear interpolation result I is carried out to filtering:

$F_i^{\′}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}^N\·I_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}^N}$

The initial high resolution depth image obtaining is designated as F ', and size is rn * rm;

Step (2) be take left image as reference picture, and right image is target image, and in conjunction with initial high resolution depth image F ', structure sectional perspective is mated weights, calculates the disparity map D of left image ^l:

(a) in conjunction with F ', construct as follows sectional perspective coupling weights:

$W_{\mathrm{ij}}^L=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|F_i^{\′}-F_j^{\′}\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _cand σ _dbe used for the rate of decay of control characteristic expression formula, choose as follows:

1. calculate the color data error between interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

2. calculate the depth difference between interior other points of the current local window of F ' and central point using the intermediate value of current depth difference as current σ _d;

(b) construct as follows the coupling cost function between left image L and right image R:

$E(i,{\stackrel{\‾}{i}}_d)=\frac{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L\·e(j,{\stackrel{\‾}{j}}_d)}{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L}$

Wherein mate cost while representing that some i parallax in left image is d, the corresponding point in i and right image between matching degree, j ∈ Ω (i) represents that j is in the local window of i, represent ? local window in, colour and the gradient difference chosen between corresponding point define matching error:

$e(j,{\stackrel{\‾}{j}}_d)=\mathrm{\α}\min \left(\right||L_j-R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_1)+(1-\mathrm{\α})\min \left(\right||\▿L_j-\▿R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_2)$

Wherein parameter alpha is used for balance colour and gradient error, τ ₁, τ ₂for error threshold;

(c), according to coupling cost function, calculate as follows the parallax of each point in left image:

$d^{*}={\arg \min }_{d\∈S_d}E(i,{\stackrel{\‾}{i}}_d)$

S wherein _d={ d _min..., d _maxexpression disparity range; Thereby obtain the anaglyph D of left image ^l;

Step (3) be take right image as reference picture, and left image is target image, calculates the parallax value of each point in right image, utilizes the right image parallactic figure D calculating ^rto D ^lcarry out left and right occlusion detection:

Take right image as reference diagram, when left image is target figure, due to F ' and left image registration, take right image during as reference diagram, cannot utilize F ' to construct coupling weights, therefore take right image, construct as follows local matching weights during as reference diagram:

$W_{\mathrm{ij}}^R=\begin{array}{}\end{array}\left\{\begin{array}{c}\exp (-\frac{{\left|\right|R_i-R_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _sand σ _cchoose as follows:

1. σ _select the radius size of local window as;

2. calculate the color data error of interior other points of the current local window of R and central point using the intermediate value of current color difference as current σ _c;

The local matching weights that utilization calculates construct the coupling cost function between right image R and left image L, calculate and take the disparity map of right image during as reference diagram, be designated as D ^r;

Use D ^rto D ^lcarry out left and right occlusion detection, if parallax point meet following formula, this point be labeled as and meet left and right occlusion detection:

$D_i^L=D_{i-D_i^L}^R$

By S set, represent D ^linterior all points that meet left and right occlusion detection;

The depth image F ' that step (4) obtains in conjunction with filtering and coupling obtain anaglyph D ^lmerge and obtain final high resolving power depth image:

(a) as follows by F ' and D ^lmerge:

$F_i=\left\{\begin{array}{cc}\frac{b-f}{D_i^L},& i\∈S\\ F_i^{\′},& \mathrm{else}\end{array}\right.$

Wherein b is baseline distance between the camera of left and right, and f is the focal length of CCD camera;

(b) structure part filter weights are revised merging the depth image obtaining:

2 observations based on following:

1., probably there is similar depth value in two regions that have Similar color in coloured image;

2. the surface of real scene is substantially all piecewise smooth;

Part filter weights are decomposed into three parts: color similarity, distance similarity and degree of depth similarity, construct as follows part filter weights:

$w_{\mathrm{ij}}=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2})\·\exp (-\frac{{\left|\right|F_i-F_j\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _c, σ _sand σ _dchoose as follows:

1. σ _select the radius size of local window as;

2. calculate the color data error of interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

3. calculate the depth difference of interior other points of the current local window of F and central point using the intermediate value of current depth difference as current σ _d;

By the above-mentioned part filter weight w that calculates _ij, as follows fusion results F is carried out to filtering:

$F_i^{*}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}\·F_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}}$

Thereby obtain final high resolving power depth image, be designated as F ^*, size is rn * rm.

The present invention, by the low resolution depth image that collects in conjunction with flight time depth camera with the high-resolution left and right of scene anaglyph pair, can rebuild the depth image that obtains quality, high resolution.

A kind of building method of non local weights wave filter is disclosed according to a first aspect of the invention.

According to a second aspect of the invention, a kind of method that combination initial depth image is constructed sectional perspective coupling weights is disclosed.

According to a third aspect of the invention we, the method that anaglyph that a kind of depth image that flight time depth camera is collected and Stereo Matching Algorithm calculate merges is disclosed.

According to a forth aspect of the invention, a kind of combination is disclosed with the idiographic flow of the right high resolving power depth image method for reconstructing of scene stereographic map.Mainly comprise: construct the depth image that non local filter weights collects flight time depth camera and carry out filtering; Structure sectional perspective coupling weights calculate the anaglyph based on Stereo Matching Algorithm; Final fusion obtains the high-quality depth image of high resolving power.

Beneficial effect of the present invention: the inventive method well, in conjunction with Stereo Matching Technology and flight time depth camera advantage separately, has well overcome the deficiencies in the prior art by a kind of fusion method, can rebuild the depth image of quality, high resolution.

Concrete implementation step

Step (1) is constructed non local filter weights, and the low resolution depth image of input is carried out to filtering:

G represents the low resolution depth image that flight time depth camera collects, and size is n * m; L represents the left image of same scene high-resolution color of the left collected by camera of CCD, and size is rn * rm; R represents the right image of high-resolution color of the corresponding right collected by camera of CCD, and size is rn * rm;

Construct as follows non local filter weights:

$w_{\mathrm{ij}}^N=\exp (-\frac{{\left|\right|M_{i}L-M_{j}L\left|\right|}_2^2}{K\·h_L^2})\·\exp (-\frac{{\left|\right|M_{i}I-M_{j}I\left|\right|}_2^2}{K\·h_I^2})$

Wherein I is that the low resolution depth image of inputting carries out the result after bilinear interpolation, and size is rn * rm, M _il represents with L _icentered by localized mass, M _ii represents with I _icentered by localized mass, K is the pixel number that localized mass contains, localized mass size elects 5 * 5 as; Parameter h _lelect 15 as, parameter h _ielect 20 as;

According to by following formula, bilinear interpolation result I is carried out to filtering:

$F_i^{\′}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}^N\·I_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}^N}$

The initial high resolution depth image obtaining is designated as F ', and size is rn * rm;

Step (2) be take left image as reference picture, and right image is target image, and in conjunction with initial high resolution depth image F ', structure sectional perspective is mated weights, calculates the disparity map D of left image ^l:

(a) in conjunction with F ', construct as follows sectional perspective coupling weights:

$W_{\mathrm{ij}}^L=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|F_i^{\′}-F_j^{\′}\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, and local window size elects 9 * 9 as, parameter σ _cand σ _dchoose as follows:

1. calculate the color data error between interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

2. calculate the depth difference between interior other points of the current local window of F ' and central point using the intermediate value of current depth difference as current σ _d;

(b) construct as follows the coupling cost function between left image L and right image R:

$E(i,{\stackrel{\‾}{i}}_d)=\frac{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L\·e(j,{\stackrel{\‾}{j}}_d)}{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L}$

Wherein mate cost while representing that some i parallax in left image is d, the corresponding point in i and right image between matching degree, j ∈ Ω (i) represents that j is in the local window of i, represent ? local window in, colour and the gradient difference chosen between corresponding point define matching error:

$e(j,{\stackrel{\‾}{j}}_d)=\mathrm{\α}\min \left(\right||L_j-R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_1)+(1-\mathrm{\α})\min \left(\right||\▿L_j-\▿R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_2)$

Wherein parameter alpha is used for balance colour and gradient error, elects 0.2 as; τ ₁, τ ₂for error threshold, elect respectively 7 and 2 as;

(c), according to coupling cost function, calculate as follows the parallax of each point in left image:

$d^{*}={\arg \min }_{d\∈S_d}E(i,{\stackrel{\‾}{i}}_d)$

S wherein _d={ d _min..., d _maxexpression disparity range; Thereby obtain the anaglyph D of left image ^l;

Step (3) be take right image as reference picture, and left image is target image, calculates the disparity map D of right image ^r, to D ^lcarry out left and right occlusion detection:

Take right image as reference diagram, when left image is target figure, due to F ' and left image registration, take right image during as reference diagram, cannot utilize F ' to construct coupling weights, therefore take right image, construct as follows local matching weights during as reference diagram:

$W_{\mathrm{ij}}^R=\begin{array}{}\end{array}\left\{\begin{array}{c}\exp (-\frac{{\left|\right|R_i-R_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, and local window size elects 9 * 9 as, parameter σ _sand σ _cchoose as follows:

1. σ _ssize elects 4 as;

2. calculate the color data error of interior other points of the current local window of R and central point using the intermediate value of current color difference as current σ _c;

The sectional perspective coupling weights that utilization calculates construct the coupling cost function between right image R and left image L, calculate and take the disparity map of right image during as reference diagram, be designated as D ^r;

Use D ^rto D ^lcarry out left and right occlusion detection, if parallax point meet following formula, this point be labeled as and meet left and right occlusion detection:

$D_i^L=D_{i-D_i^L}^R$

By S set, represent D ^linterior all points that meet left and right occlusion detection;

The depth image F ' that step (4) obtains in conjunction with filtering and coupling obtain anaglyph D ^lmerge and obtain final high resolving power depth image:

(a) as follows by F ' and D ^lmerge:

$F_i=\left\{\begin{array}{cc}\frac{b-f}{D_i^L},& i\∈S\\ F_i^{\′},& \mathrm{else}\end{array}\right.$

Wherein b is baseline distance between the camera of left and right, and f is the focal length of CCD camera;

(b) structure part filter weights are revised merging the depth image obtaining:

Construct as follows part filter weights:

$w_{\mathrm{ij}}=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2})\·\exp (-\frac{{\left|\right|F_i-F_j\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, and local window size elects 9 * 9 as, parameter σ _c, σ _sand σ _dchoose as follows:

1. σ _ssize elects 4 as;

2. calculate the color data error of interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

3. calculate the depth difference of interior other points of the current local window of F and central point using the intermediate value of current depth difference as current σ _d;

By the above-mentioned part filter weight w that calculates _ij, as follows fusion results F is carried out to filtering:

$F_i^{*}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}\·F_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}}$

Thereby obtain final high resolving power depth image F ^*, size is rn * rm.

Claims (1)

1. combination, with the right high resolving power depth image method for reconstructing of scene stereographic map, is characterized in that the concrete steps of the method are:

Step (1) is constructed non local filter weights, and the low resolution depth image of input is carried out to filtering:

G represents the low resolution depth image that flight time depth camera collects, and size is n * m; L represents the left image of same scene high-resolution color of the left collected by camera of CCD, and size is rn * rm; R represents the right image of high-resolution color of the corresponding right collected by camera of CCD, and size is rn * rm;

Construct as follows non local filter weights:

$w_{\mathrm{ij}}^N=\exp (-\frac{{\left|\right|M_{i}L-M_{j}L\left|\right|}_2^2}{K\·h_L^2})\·\exp (-\frac{{\left|\right|M_{i}I-M_{j}I\left|\right|}_2^2}{K\·h_I^2})$

Wherein I is that the low resolution depth image of inputting carries out the result after bilinear interpolation, and size is rn * rm; M _il represents with L _icentered by localized mass, M _ii represents with I _icentered by localized mass, K is the pixel number that localized mass contains, h _l, h _ibe the parameter of wave filter, be used for controlling the rate of decay of the exponential expression of weighted calculation;

By above-mentioned, calculate non local filter weights , by following formula, bilinear interpolation result I is carried out to filtering:

$F_i^{\′}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}^N\·I_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}^N}$

The initial high resolution depth image obtaining is designated as F ', and size is rn * rm;

Step (2) be take left image as reference picture, and right image is target image, and in conjunction with initial high resolution depth image F ', structure sectional perspective is mated weights, calculates the disparity map D of left image ^l:

(a) in conjunction with F ', construct as follows sectional perspective coupling weights:

$W_{\mathrm{ij}}^L=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|F_i^{\′}-F_j^{\′}\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _cand σ _dbe used for the rate of decay of control characteristic expression formula, choose as follows:

1. calculate the color data error between interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

2. calculate the depth difference between interior other points of the current local window of F ' and central point using the intermediate value of current depth difference as current σ _d;

(b) construct as follows the coupling cost function between left image L and right image R:

$E(i,{\stackrel{\‾}{i}}_d)=\frac{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L\·e(j,{\stackrel{\‾}{j}}_d)}{{\mathrm{\Σ}}_{j\∈\mathrm{\Ω}\left(i\right),{\stackrel{\‾}{j}}_d\∈\mathrm{\Ω}\left({\stackrel{\‾}{i}}_d\right)}{W}_{\mathrm{ij}}^L}$

Wherein mate cost while representing that some i parallax in left image is d, the corresponding point in i and right image between matching degree, j ∈ Ω (i) represents that j is in the local window of i, represent ? local window in, colour and the gradient difference chosen between corresponding point define matching error:

$e(j,{\stackrel{\‾}{j}}_d)=\mathrm{\α}\min \left(\right||L_j-R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_1)+(1-\mathrm{\α})\min \left(\right||\▿L_j-\▿R_{{\stackrel{\‾}{j}}_d}||,{\mathrm{\τ}}_2)$

Wherein parameter alpha is used for balance colour and gradient error, τ ₁, τ ₂for error threshold;

(c), according to coupling cost function, calculate as follows the parallax of each point in left image:

$d^{*}={\arg \min }_{d\∈S_d}E(i,{\stackrel{\‾}{i}}_d)$

S wherein _d={ d _min..., d _maxexpression disparity range; Thereby obtain the anaglyph D of left image ^l;

Step (3) be take right image as reference picture, and left image is target image, calculates the parallax value of each point in right image, utilizes the right image parallactic figure D calculating ^rto D ^lcarry out left and right occlusion detection:

Take right image as reference diagram, when left image is target figure, due to F ' and left image registration, take right image during as reference diagram, cannot utilize F ' to construct coupling weights, therefore take right image, construct as follows sectional perspective coupling weights during as reference diagram:

$W_{\mathrm{ij}}^R=\begin{array}{}\end{array}\left\{\begin{array}{c}\exp (-\frac{{\left|\right|R_i-R_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _sand σ _cchoose as follows:

1. σ _select the radius size of local window as;

2. calculate the color data error of interior other points of the current local window of R and central point using the intermediate value of current color difference as current σ _c;

The local matching weights that utilization calculates , construct the coupling cost function between right image R and left image L, calculate and take the disparity map of right image during as reference diagram, be designated as D ^r;

Use D ^rto D ^lcarry out left and right occlusion detection, if parallax point meet following formula, this point be labeled as and meet left and right occlusion detection:

$D_i^L=D_{i-D_i^L}^R$

By S set, represent D ^linterior all points that meet left and right occlusion detection;

The depth image F ' that step (4) obtains in conjunction with filtering and coupling obtain anaglyph D ^lmerge and obtain final high resolving power depth image:

(a) as follows by F ' and D ^lmerge:

$F_i=\left\{\begin{array}{cc}\frac{b-f}{D_i^L},& i\∈S\\ F_i^{\′},& \mathrm{else}\end{array}\right.$

Wherein b is baseline distance between the camera of left and right, and f is the focal length of CCD camera;

(b) structure part filter weights are revised merging the depth image obtaining:

2 observations based on following:

1., probably there is similar depth value in two regions that have Similar color in coloured image;

2. the surface of real scene is substantially all piecewise smooth;

Part filter weights are decomposed into three parts: color similarity, distance similarity and degree of depth similarity, construct as follows part filter weights:

$w_{\mathrm{ij}}=\left\{\begin{array}{c}\exp (-\frac{{\left|\right|L_i-L_j\left|\right|}_2^2}{{\mathrm{\σ}}_c^2})\·\exp (-\frac{{\left|\right|i-j\left|\right|}_2^2}{{\mathrm{\σ}}_s^2})\·\exp (-\frac{{\left|\right|F_i-F_j\left|\right|}_2^2}{{\mathrm{\σ}}_d^2}),j\∈\mathrm{\Ω}\left(i\right)\\ 0,\mathrm{else}\end{array}\right.$

Wherein j ∈ Ω (i) represents that j is in the local window of i, parameter σ _c, σ _sand σ _dchoose as follows:

1. σ _select the radius size of local window as;

2. calculate the color data error of interior other points of the current local window of L and central point using the intermediate value of current color difference as current σ _c;

3. calculate the depth difference of interior other points of the current local window of F and central point using the intermediate value of current depth difference as current σ _d;

By the above-mentioned part filter weight w that calculates _ij, as follows fusion results F is carried out to filtering:

$F_i^{*}=\frac{\mathrm{\Σ}{w}_{\mathrm{ij}}\·F_j}{\mathrm{\Σ}{w}_{\mathrm{ij}}}$

Thereby obtain final high resolving power depth image F ^*, size is rn * rm.