CN102098526A - Depth map calculating method and device - Google Patents

Abstract

The invention discloses a depth map calculating method which comprises the steps of: inputting plane image; pretreating; selectively carrying out basic depth map calculation, basic depth map real-time calculation, depth map calculation based on color segmentation and rapid depth map calculation based on color segmentation on the plane image; carrying out depth map improvement based on reference frame and color distribution and space position improvement based on current frame on the depth map obtained by calculation; and outputting the depth map obtained by calculation, the improved depth map based on the reference frame, the improved color distribution and space position depth map based on the current frame and the plane image. The invention can be used for improving the quality of the depth map, meets the needs of depth map calculation with multiple time and quality requirements, and is wider in application scope. The invention also discloses a depth map calculating device.

Description

A kind of depth map acquiring method and device

Technical field

The present invention relates to technical field of computer vision, particularly a kind of depth map acquiring method and device.

Background technology

Existing various video medium generally is that the form with the plane exists, but above-mentioned video medium has lost the depth information of concrete scene when obtaining.

The three-dimensional video-frequency technology is as following Development of Multimedia Technology direction, and being a kind ofly can provide relief novel video technique.Compare with the single channel video, three-dimensional video-frequency generally has two video channels, and data volume will be far longer than the single channel video, so the efficient compression of stereoscopic video is particularly important.Particularly, three-dimensional video-frequency not only comprises the surface information about scene of conventional two-dimensional video, but also comprises the 3 D stereo information relevant with the scene particular location.Compare with traditional two-dimensional video, three-dimensional video-frequency is a kind of more effective, more real expression way, the one-sidedness of two-dimensional video and the shortcoming of passivity have been overcome, can satisfy visual perception's demand of people more fully, a lot of fields such as, advertisement media live in Interactive Free viewpoint video (FVV), virtual reality, 3DTV, 3D recreation, physical culture have a wide range of applications.

In stereovision technique, the extraction of depth map is then particularly important as the depth information that obtains concrete scene.Though existing various depth extraction method is a lot, all has the computation complexity height, calculates characteristics such as consuming time.Especially, for the depth map acquisition algorithm in the binocular stereo vision, be divided into two kinds of the overall situation and local algorithms.The complexity of above-mentioned algorithm generally is directly proportional with the geometric progression of the number of pixels of image or frame of video, and particularly Global Algorithm has related to the iterative process of global optimization, and complexity is higher.

Especially, existing depth map extracting method is not high for the real-time of the real-time image data processing on ordinary consumption type host computer.

Summary of the invention

Purpose of the present invention is intended to solve at least one of above-mentioned technological deficiency, has proposed a kind of depth map acquiring method and device especially.

For achieving the above object, the embodiment of first aspect present invention has proposed a kind of depth map acquiring method, comprises the steps:

The input plane image, wherein, described plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Described plane picture is carried out preliminary treatment;

Judge whether pretreated plane picture needs to carry out that color of image is cut apart and whether need to move the moving average algorithm, to described plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates and calculates fast based on the depth map that color is cut apart;

The depth map that calculates is carried out improving and based on the improvement of present frame distribution of color and locus based on the depth map of reference frame, wherein, the described depth map that calculates comprise the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation; And

Export the described depth map that calculates and improve after the depth map based on reference frame, depth map and the described plane picture after improving based on present frame distribution of color and locus.

According to the depth map acquiring method of the embodiment of the invention, adopt and to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and by utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Improve algorithm by depth map based on bilateral filtering, both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map, and the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

The embodiment of second aspect present invention has proposed a kind of depth map and has asked for device, comprise the plane picture input module, described plane picture input module is used for the input plane image, and wherein, described plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence; Pretreatment module, described pretreatment module are used for described plane picture is carried out preliminary treatment; The depth map computing module, described depth map computing module is used to judge whether the pretreated plane picture of described pretreatment module needs to carry out that color of image is cut apart and whether need to move the moving average algorithm, to described plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates and calculates fast based on the depth map that color is cut apart; Depth map improves module, described depth map improves module and is used for the depth map that described depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame distribution of color and locus, wherein, the described depth map that calculates comprises the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation; And output module, described output module be used to export the described depth map that calculates and improve after the depth map based on reference frame, depth map and the described plane picture after improving based on present frame distribution of color and locus.

Depth map according to the embodiment of the invention is asked for device, adopts to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and by utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Improve algorithm by depth map based on bilateral filtering, both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map, and the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize by practice of the present invention.

Description of drawings

Above-mentioned and/or additional aspect of the present invention and advantage are from obviously and easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:

Fig. 1 is the FB(flow block) according to the depth map acquiring method of the embodiment of the invention;

Fig. 2 is the flow chart according to the input plane image of the embodiment of the invention;

Fig. 3 is for carrying out pretreated flow chart according to the embodiment of the invention to plane picture;

Fig. 4 is the flow chart according to the depth map calculating of the embodiment of the invention;

Fig. 5 is according to the improved flow chart of the depth map of the embodiment of the invention;

Fig. 6 is the schematic flow sheet according to the depth map output of the embodiment of the invention;

Fig. 7 is for asking for the structural representation of device according to the depth map of the embodiment of the invention; And

Fig. 8 is the structural representation according to the depth map computing module of the embodiment of the invention.

Embodiment

Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Below by the embodiment that is described with reference to the drawings is exemplary, only is used to explain the present invention, and can not be interpreted as limitation of the present invention.

Below with reference to the depth map acquiring method of Fig. 1 to Fig. 6 description according to the embodiment of the invention.

As shown in Figure 1, the depth map acquiring method according to the embodiment of the invention comprises the steps:

S101: input plane image;

As shown in Figure 2, plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence.Wherein, plane picture can change 3D video system, other two-way videos or image sequence output interface or other single channel videos or image sequence output interface from binocular solid collection and three-dimensional Play System, full-automatic 2D video.

For binocular solid collection and three-dimensional Play System, two-way video that end collects or the two-way picture frame plane picture as input will be taken.Change the 3D video system for full-automatic 2D video, directly with the plane picture of two frames of the front and back on the time shaft in the 2D video as input.

When the plane picture of input is single channel video or two-way video, need carry out decoding processing.Based on different video compression format different video encoding/decoding methods is arranged.Wherein, the decoding to video comprises that reading data flow is converted to suitable frame of video form again from known video.If the frame of video form of decoding output and follow-up depth map are asked for interface and be not inconsistent, then need to carry out the conversion operations of frame of video form.

When the plane picture of input is two-way video or two-way image sequence, synchronization picture frame for dual input, with the picture frame of wherein one road video or image sequence as present frame, the picture frame of another road video or image sequence as the reference frame, can be provided with the order of two-path video according to application request.Particularly,, the picture frame of a left side (or right) road video or image sequence is called present frame, the picture frame in same moment of right (or left side) road video or image sequence is called reference frame for binocular solid collection and three-dimensional Play System.

When the plane picture of input is single channel video or single channel image sequence, for example change the 3D video system for full-automatic 2D video, be input as single channel 2D video, then the present frame of single channel video on time shaft be called present frame, the back frame on the time shaft is called the default reference frame.

S102: plane picture is carried out preliminary treatment;

As shown in Figure 3, when the plane picture of input when being single channel video or image sequence, single channel video or image sequence are carried out preliminary treatment comprise the steps:

S1021: judge whether to carry out key frame and detect, then carry out S1022 if desired; Otherwise carry out S103;

Change the 3D video for real-time 2D video, be input as two picture frames of the front and back on the time shaft in the 2D video, need do a key frame to back one frame this moment and judge, with the image of a frame after confirming and the similitude of current frame image.If the diversity of back one two field picture and current frame image surpasses the threshold value under certain specific tolerance, think that then back one two field picture is a key frame, otherwise think that then back one two field picture is not key frame.

S1022: key frame detects;

At first, front and back two frames are carried out horizontal central line simultaneously cut apart with median vertical line and cut apart, thereby two frames are divided into four little image blocks respectively before and after making.In one embodiment of the invention, the length of four little image blocks after cutting apart and wide size are original image frame length and roomy little by 1/2.

Secondly, four little image blocks of front and back two frame correspondence positions are carried out the image block pixel value respectively subtract each other, the absolute value of getting difference to obtain new image block, is asked for pixel average and variance to four new image blocks as new pixel value.

Once more, the pixel average and the variance of four new images pieces are carried out threshold decision respectively.If individual number average that average and variance surpass threshold value in four new images pieces, judges then that afterwards the relative present frame of a frame is a new key frame greater than preset value a.In one embodiment of the invention, preset value a can be 3.

In one embodiment of the invention, pixel average and variance threshold values can be the arbitrary value within [20,30].

At last, if one frame relative present frame in back is new key frame, then the reference frame of back one frame as the depth map calculating of present frame can be able to not be chosen the reference frame of the former frame of present frame as the depth map calculating of present frame this moment.

When the plane picture of input when being two-way video or two-way image sequence, two-way video or two-way image sequence are carried out preliminary treatment comprise the steps:

S1023: judge whether to carry out outer level line and proofread and correct, then carry out S1024 if desired, otherwise carry out S103;

For binocular solid collection and three-dimensional Play System, two-way video or the two-way image sequence of importing carried out preliminary treatment, need proofread and correct the outer level of two-path video frame or image sequence.Outer level line is proofreaied and correct and can be made certain the delegation's pixel of sustained height in the corresponding reference frame of present frame pixel, has eliminated the deviation on the vertical direction.Thereby can reduce in the corresponding reference frame of this pixel with the search dimension of the corresponding pixel points in the time chart picture frame, reduce amount of calculation.

S1024: the outer level of image line is proofreaied and correct.

A certain pixel in the identical time chart picture frame in any pixel in a certain time chart picture frame in the road video in the two-way video and another road video is unique corresponding, thus the jobbie point in the space is gathered in common expression.

If but corresponding degree of depth the unknown of some pixels in a certain picture frame in one road video, this pixel may appear on the optional position of a certain straight line in one picture frame of back with the unique corresponding pixel points in the time chart picture frame in another road video so.Be on this straight line in the back picture frame all pixels all may with same object point in the corresponding space of this pixel.This straight line is the outer level of this pixel correspondence line.Generally speaking, outer level line is not a horizontal linear, but an oblique line.Proofread and correct the corresponding outer level line that can make in corresponding another picture frame of this pixel by outer level line and be transformed to horizontal direction, thereby eliminated departing from the vertical direction, can dwindle in corresponding another road video of this pixel search dimension, thereby reduce amount of calculation with the unique corresponding pixel points in the time chart picture frame.

Certainly it will be appreciated by persons skilled in the art that and to adopt other outer level line bearing calibrations the outer level line correction of two-way video or image sequence.

As can be seen from Figure 3, the plane picture of input also can be without preliminary treatment and the input of directly calculating as depth map.

S103: compute depth figure;

As shown in Figure 4, in an embodiment of the present invention, the calculating of depth map is found any one pixel of present frame corresponding pixel in reference frame by the window matching algorithm, obtains the depth information of current frame pixel point then by the position relation of two corresponding pixels.

S1031: judge whether to carry out color of image and cut apart, if then carry out S1035; Otherwise carry out S1032;

To from the plane picture of the direct input of step S101 or judge whether to carry out color of image from the plane picture after the improvement of step S102 and cut apart.

S1032: judge whether to use the moving average algorithm, if then carry out S1034; Otherwise carry out S1033;

When judging that plane picture does not need to carry out color of image when cutting apart, and judges further whether plane picture needs to use the moving average algorithm.

S1033: calculate basic depth map;

When judging that plane picture does not need to use the moving average algorithm, calculate the basic depth map of plane picture, comprise the steps:

The first step is provided with window and search volume.

The size of window at first is set, and the window width that window is set is odd number N, includes N*N pixel in window width is the window of N.With candidate's respective pixel on the region of search in any one pixel in the present frame and the reference frame respectively as center pixel.Wherein, center pixel is positioned at the center of window.All pixels that comprise in the window become (comprising center pixel itself) the support pixel of center pixel.In one embodiment of the invention, window width N can be 15.Certainly it will be appreciated by persons skilled in the art that window width N can also be other odd number width.

When the ranks pixel of the pixel at four angles of each image or four peripheries is set up the window of pre-set dimension as center pixel, will run into this pixel around do not have the situation of pixel.At this moment, can adopt one of following dual mode that window is set.

1) do not exist the pixel value of pixel to be set to 0 around this pixel.

2) window size of directly setting up the corresponding window of above-mentioned pixel is set to 1 * 1 pixel (being about to this pixel itself as a window).

Certainly it will be appreciated by persons skilled in the art that when adopt additive method solve this pixel around when not having under the situation of pixel the problem of window to be set, above-mentioned additive method also falls into protection scope of the present invention.

For any one pixel of present frame, the hunting zone that the region of search is promptly set the candidate corresponding pixel points of this pixel in reference frame is set.The region of search is represented in the present frame relative position relation of candidate's respective pixel in the pixel and reference frame arbitrarily.In one embodiment of the invention, the region of search can be according to actual concrete condition of attending with flexible processing.

Eliminate later present frame and the reference frame of vertical missing to proofreading and correct through the outer level of image line, the region of search can be set to [d _Min, d _Max]=[0,32], wherein d is the difference of waiting arbitrarily in reference frame candidate corresponding pixel points and the present frame to ask between the horizontal coordinate of degree of depth pixel, d _MinThe coordinate of candidate's respective pixel waits to ask the pixel coordinate of the degree of depth consistent with present frame in=0 expression reference frame, and two pixels are in the same position in reference frame and the present frame respectively.

In second step, plane picture is carried out calculating based on the pixel coupling of window according to window;

Pixel coupling calculating based on window can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈N_p,{\stackrel{\‾}{q}}_d\∈N_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, p represents any one pixel in the present frame;

Expression is candidate's corresponding pixel points in the reference frame of d apart from current frame pixel point p relative position; N _pExpression is all pixels that window was comprised (comprising pixel p itself) of center pixel with pixel p;

Expression is with candidate pixel point in the reference frame

For all pixels that window comprised of center pixel (comprise pixel

Itself); P with

Corresponding window width is N; Q represents in the present frame with p to be any one pixel in the window of center pixel,

Represent in the reference frame with the candidate pixel point

Be any one pixel in the window of center pixel,

Relative position in window is identical with q,

Remarked pixel point q with

The absolute value of margin of image element.

The 3rd step, obtain each pixel pixel parallax in the present frame according to window matching value and search volume, obtain the basic depth map of present frame correspondence.

At first obtain the pixel parallax of each pixel in the present frame.For any one the pixel p in the present frame, at the region of search [d _Min, d _Max] in choose candidate pixel point in one of them d corresponding reference frame, the absolute difference and the summation of calculating all pixels in these two central pixel point place windows obtain E _d, to interval [d _Min, d _Max] interior all corresponding E of any d _d, select all E at last _dThe parallax value of minimum value correspondence as the parallax value d of current pixel point in the present frame _s, simultaneously with parallax value d _sCandidate pixel point in the corresponding reference frame

As the final respective pixel of pixel p, wherein,

The employing said method can obtain the time difference of all pixels of present frame, according to the parallax d of all pixels of present frame that obtain _{(i, j)}=d _s, d wherein _{(i, j)}Be corresponding pixel points (i, parallax j).According to the depth map of the present frame correspondence that obtains,

$Z_{(i,j)}=\frac{d_{(i,j)}-d_{\min }}{(d_{\max }-d_{\min })}*255$

Z wherein _{(i, j)}Be corresponding pixel points (i, degree of depth j).

S1034: calculate basic real-time deep figure;

When judging that plane picture need use the moving average algorithm, calculate the basic real-time deep figure of plane picture, comprise the steps:

For the region of search [d _Min, d _Max] in each parallax d, for each pixel in the present frame, all need to calculate in its place window the pixel value difference of the respective pixel in the window of candidate pixel point place in each pixel and reference frame.

When being input as long and wide the be present frame of W*H and reference frame, window size is N*N, search length is L, wherein for each pixel in the present frame and the candidate pixel point in the reference frame in the hunting zone, every calculating once, amount of calculation is N*N time a subtraction, N*N-1 time addition, and final amount of calculation is W*H*L* (N*N) * (N*N-1).This amount of calculation is directly proportional with the biquadratic of window width N, because N is variable, when N increased, then amount of calculation will be geometric progression increased.

The first step, initialization calculating that line slip on line direction is average and the average recursive calculation of line slip, and the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain the real-time window matching value Q of any pixel correspondence of described present frame _{(i, j)}

At first calculate in the initialization of the up moving average of line direction.First pixel (i, 0) to each row in the present frame is interior to pixel value difference absolute value C in window line width scope [0, N] _{(i, j)}Carry out accumulation calculating.Wherein,

j∈[0，N]，A _(i，0)＝A _(i，0)+C _(i，j)

Wherein,

Any i of delegation, A are got in expression _{(i, 0)}The C of first pixel (i, 0) in window line width scope that represents any i of delegation _{(i, j)}Accumulation calculating result, C _{(i, j)}Support pixel in the expression window line width scope (i, j) with reference frame in the pixel value difference absolute value of respective pixel, N represents window width.

Then line slip is averaged recursive calculation.Particularly, adopt following formula that any capable i is carried out the average recursive calculation of line slip:

$A_{(i,j)}=A_{(i,j-1)}+C_{(i,j+\frac{N}{2})}-C_{(i,j-\frac{N}{2}-1)},$

Wherein, for the center pixel of the capable j of i row (i, j), the sliding average A on its corresponding row direction _{(i, j)}Line slip mean value A by previous pixel _{(i, j-1)}Add the current window right endpoint Result of calculation

And, deduct the left end point of previous window again Result of calculation

Recursive calculation obtains.

Again, the initialization of the above-listed moving average of column direction is calculated.Particularly, to first pixel of any row j in the present frame (0, j) adopt following formula in the window column wide region, line slip mean value to be carried out accumulation calculating:

i∈[0，N]，Q _(0,j)＝Q _(0,j)+A _(i，j)，

Wherein Represent to appoint and get a j, Q _{(0, j)}Represent that (0, j) A is calculated in the accumulation to line slip mean value to any first pixel of row j in the window column wide region _{(i, j)}(N represents window width to pixel for i, line slip mean value j) in the expression present frame.

Be listed as the average recursive calculation of slip at last.Particularly, adopt following formula to carry out column direction row moving average recursive calculation for any row j:

$Q_{(i,j)}=Q_{(i-1,j)}+A_{(i+\frac{N}{2},j)}-A_{(i-\frac{N}{2}-1,j)},$

Q wherein _{(i, j)}Represent any one row j the capable pixel of i (i, j) the recursive calculation result of row sliding average in the window column wide region is by the row sliding average Q of a last pixel _{(i-1, j)}With the current window lower extreme point

Line slip mean value and, deduct a window upper extreme point again

Line slip mean value recursive calculation obtain.

Second the step, any pixel in present frame (i, j) corresponding to each relative position among the d of particular search space to a plurality of described real-time window matching value Q should be arranged _{(i, j)}, for the region of search [d _Min, d _Max] interior a plurality of real-time window matching value Q _{(i, j)}In choose the corresponding parallax value of minimum value as current pixel point in the present frame (i, parallax value d j) _S1Thereby, can obtain the depth value of respective pixel.

$d_{s1}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{Q_{(i,j)}{|}_d\right\}.$

In the 3rd step, ask for the real-time pixel parallax of each pixel of present frame, thereby can obtain the basic real-time deep figure of present frame correspondence.

S1035: judge whether to use the moving average algorithm, if then carry out S1037; Otherwise carry out S1036;

Because in the ordinary course of things, the corresponding same object of the image-region of same color, depth value is same or similar; And the corresponding different objects of the image-region of different colours, the degree of depth is inconsistent.Therefore can utilize color of image to cut apart the classified information that obtains, based on the window calculation depth map time, increase belongs to the shared weight of of a sort other pixels with the window center pixel, reduces not belong to the shared weight of of a sort other pixels with the window center pixel.

S1036: calculate depth map based on image segmentation.

The first step, (i j), is connected to 8 limits with each pixel (i) of importing present frame with its 8 pixels (j) on every side to set up graph model V.Wherein, two end points on every limit be two pixels (i, j), the weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) |, obtain thus a series of limit edge (i, j).

In second step, under initial situation, (i) is a class with each pixel, and to set the initial merger threshold value of class this moment be threshold (i)=c, sorted from small to large according to weights in every limit in the above-mentioned graph model then.

The 3rd step, to pairing two pixels in limit after the ordering (i, j) the place class is carried out merger in twos operation, wherein, merger comprises the steps:

As a limit edge (i, j) weight w (i, j) than two end points (i on this limit, j) the merger threshold value threshold (i) of place class, threshold (j) hour, then with this limit edge (i, the two class merger at two pixel places j) are a class, the merger threshold value of upgrading the new class obtain simultaneously is:

$\mathrm{threshold}(i,j)=w(i,j)+\frac{c}{\mathrm{num}(i,j)},$

Wherein, num (i, j) the number of pixels addition of the original place of remarked pixel point i and j class.

The 4th step, the fewer class of the number of pixels that obtains after the merger is forced the merger operation, travel through all limit edge (i, j), if exist two pixels on any limit not at same time-like, and the number of pixels of the class at two pixel places all is less than a specified value minSize, so just two classes at two pixel places carried out the merger operation, and the color that finally obtains after each pixel is sorted out is cut apart figure.

The 5th goes on foot, and the plane picture after according to window color being cut apart carries out calculating based on the pixel coupling of window;

Can adopt following formula to calculate based on the pixel coupling calculating that color is cut apart,

$E_d={\mathrm{\Σ}}_{q\∈{\mathrm{NS}}_p,{\stackrel{\‾}{q}}_d\∈{\mathrm{NS}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d)+\mathrm{\λ}\×{\mathrm{\Σ}}_{q\∈{N\stackrel{\‾}{S}}_p,{\stackrel{\‾}{q}}_d\∈{N\stackrel{\‾}{S}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, NS _pExpression with pixel p be center pixel window was comprised and pixel p belongs to the pixel (comprising pixel p itself) of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that belongs to same color block (comprises pixel

Itself);

Expression with pixel p be center pixel window was comprised and pixel p does not belong to the pixel of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that does not belong to same color block; λ represents not belong to center pixel the accumulation calculating weight of the area pixel of same block.In one embodiment of the invention, λ can be 0.01.

The 6th step, obtain each pixel pixel parallax in the present frame according to window matching value and search volume, obtain the basic depth map of present frame correspondence.

$d_s=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{E_d\right\}.$

S1037: calculate real-time deep figure based on image segmentation;

The first step is calculated based on the initialization of the line slip of the carve information of image segmentation.

For each row of present frame, make the line slip of all blocks on average be initially zero, i.e. T _S=0, wherein s is the block under the pixel.Then each row is carried out calculating based on the average initialization of the line slip of carve information, promptly in window line width scope to pixel value difference according to the place block, carry out accumulation calculating.

j∈[0，N]，T _S(i，j)＝T _S(i，j)+C _(i，j)

In the following formula, T _{S (i, j)}(i, j) the affiliated sliding average of block on line direction is by belonging to block S (i, C j) for remarked pixel _{(i, j)}Adding up obtains, C _{(i, j)}Pixel in the expression present frame (i, j) with reference frame in the pixel value difference absolute value of respective pixel.Calculate by following formula, can obtain any delegation based on the average initial calculation result of the line slip of carve information.

Second step is based on the average recursive calculation of line slip of the carve information of image segmentation

Adopt following formula that any delegation is carried out based on the average recursive calculation of the line slip of carve information:

$T_{S(i,j+\frac{N}{2})}=T_{S(i,j+\frac{N}{2})}+C_{S(i,j+\frac{N}{2})}$

$T_{S(i,j-\frac{N}{2}-1)}=T_{S(i,j-\frac{N}{2}-1)}-C_{S(i,j-\frac{N}{2}-1)},$

In following formula, (i, j), the result of calculation of the right endpoint of its corresponding window is the center pixel that is listed as for the capable j of i The result of calculation of the left end point of the previous window of its correspondence is

Respectively according to the block at its place

Carry out addition and subtraction, upgrade block T _SThe average recursive calculation result of line slip.Finally for each center pixel (i, j), obtain based on the line slip average computation result of carve information be

$A_{(i,j)}^r=T_{S(i,j)}.$

In the 3rd step, calculate based on the initialization of the row moving average of the carve information of image segmentation

For each row of present frame, make the row moving average of all blocks be initially zero, i.e. G _S=0, wherein s is the block under the pixel, and G is a row moving average result of calculation.Then each row is carried out calculating based on the row moving average initialization of carve information, promptly in the window column wide region to based on the line slip average computation result of carve information according to the place block, carry out accumulation calculating.

i∈[0，N]， $G_{S(i,j)}=G_{S(i,j)}+A_{(i,j)}^r$

In following formula, G _{S (i, j)}Remarked pixel (i, j) the window moving average on the affiliated block column direction,

Pixel (i, j) the line slip average computation result on line direction in the expression present frame based on carve information.Calculate by following formula, can obtain for the initial calculation result of any rows of directions based on the row moving average of carve information.

The 4th step is based on the row moving average recursive calculation of the carve information of image segmentation.

Adopt following formula to carry out row moving average recursive calculation to any row based on carve information:

$G_{S(i+\frac{N}{2},j)}=G_{S(i+\frac{N}{2},j)}+A_{S(i+\frac{N}{2},j)}^r$

$G_{S(i-\frac{N}{2}-1,j)}=G_{S(i-\frac{N}{2}-1,j)}-A_{S(i-\frac{N}{2}-1,j)}^r,$

In following formula, (i, j), the line slip average computation result of its corresponding window lower extreme point is to be listed as the capable center pixel of i for j

The line slip average computation result of the upper extreme point of a window is on its respective column direction

Respectively according to the block at its place

Carry out addition and subtraction, recurrence is upgraded block G _SRow moving average result.(i, j) the window accumulation result of calculation based on carve information is G finally to obtain any pixel _{S (i, j)}

In the 5th step, ask for parallax and depth map based on the carve information of image segmentation.

By obtaining any pixel (i, basic window accumulation result of calculation Q j) _{(i, j)}With window accumulation result of calculation G based on the color carve information _{S (i, j)}, utilize The above results to adopt the basic depth map of following improvement to ask for algorithm in real time:

M(i，j)| _d＝λ(Q _(i，j)| _d-G _S(i，j)| _d)+G _S(i，j)| _d

In following formula | _dExpression is for d specific in the hunting zone, and (i, j) the final window of expression is accumulated result of calculation, Q to M _{(i, j)}-G _{S (i, j)}When expression window accumulation is calculated and center pixel (i j) does not belong to the accumulation calculating section of same block, G _{S (i, j)}(i j) belongs to the accumulation calculating section of same block, and λ represents that the accumulation that does not belong to the area pixel of same block with center pixel calculates weight for expression and center pixel.In one embodiment of the invention, λ can be 0.01.(i, the parallax value of minimum value correspondence j) is the parallax d of current frame pixel finally to get M _S2

$d_{s2}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{M(i,j){|}_d\right\}.$

In one embodiment of the invention, based on the calculating of the algorithm of four kinds of different characteristics to the depth map of present frame, be that the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculate and calculate fast based on the depth map that color is cut apart, four kinds of algorithms are separate.Each algorithm can be as independently depth map calculating.Certainly the independence or the comprehensive use that it will be appreciated by persons skilled in the art that above-mentioned four kinds of algorithms all belong to protection scope of the present invention.

The real-time calculating of the basic depth map of the embodiment of the invention improves on computation complexity the calculating of basic depth map, has reached the real-time requirement.The depth map calculating of cutting apart based on color of the embodiment of the invention further improves on the calculating accuracy the calculating of basic depth map, the quick calculating of cutting apart based on color of depth map of the embodiment of the invention all improves on calculating accuracy and real-time, has obtained the higher depth map of quality.

S104: improve depth map;

Because the quick calculating of the calculating of the depth map that utilizes the real-time calculating of the calculating of above-mentioned basic depth map, basic depth map, cuts apart based on color and the depth map cut apart based on color is extracted the depth map of present frame, there is the pixel matching error, perhaps there is not corresponding matched pixel point in some pixel of present frame in reference frame, therefore need the depth map that calculate among the step S103 be improved.

As shown in Figure 5, the depth map that calculates among the step S103 is carried out based on the improvement of the depth map of reference frame with based on the improvement of present frame distribution of color and locus.Wherein, the depth map that calculates comprise the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation.

S1041: judge whether to carry out improvement based on the reference frame depth map; If then carry out S1042; Otherwise carry out S1044;

S1042: calculate the reference frame depth map;

When judging the improvement to carry out based on the reference frame depth map, the depth map that calculates is asked for, searches, checked and improves.Particularly, comprise the steps:

The first step is asked for the depth map of reference frame according to the depth map that calculates.Only need depth map calculate interface will before reference frame as present present frame, with before present frame as present reference frame, the depth map of reference frame correspondence before can calculating.

Second step is by certain the pixel P in the present frame _C(i, degree of depth Z j) _C(i j) can find pixel P corresponding in the reference frame _R(i, j), then the corresponding degree of depth of this pixel in the reference frame depth map is Z _R(i, j).

S1043: based on the improvement of reference frame depth map;

The first step is checked depth map.If Z _C(i, j) and Z _R(i, j) difference of two depth values is less than or equal to predetermined threshold t, then represents two pixel P _C(i, j) and P _R(i, j) corresponding same object point can judge that then depth calculation is accurate; If Z _C(i, j) and Z _R(i when j) difference of two depth values is greater than predetermined threshold t, represents that then two pixels are not corresponding, can assert that present frame is at pixel P _C(i, the depth calculation mistake of j) locating.

Predetermined threshold t can reach needs as the case may be and set.In one embodiment of the invention, predetermined threshold t can be 1.

In second step, depth map improves.If the depth calculation mistake then can be got Z _C(i, j) and Z _R(i, j) smaller between the two is current frame pixel point P _C(i, degree of depth j), thus reach the purpose of improving the depth map quality.

S1044: judge whether to carry out improvement based on present frame distribution of color and locus; If then carry out S1045; Otherwise carry out S105;

S1045: based on the improvement of reference frame depth map.

Refer to that based on the present frame distribution of color and the improvement of locus the ID value can carry out bilateral filtering by distribution of color in the pixel value place window area of its corresponding present frame and locus and improve.

For the pixel Z (p) of any the expression degree of depth among the ID figure, its respective pixel in present frame is p, is center pixel with the pixel p in the present frame, and window width is that all pixels in the window of N are q.The pixel value difference Gauss weight factor w of all pixel q and center pixel p in the calculation window ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q), and the depth value Z (q) of two weight factors and pixel q multiplied each other, obtain the depth value Z (q) of cum rights repeated factor.Pixel value difference Gauss weight factor w ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q) be respectively:

$w_1(V_p,V_q)=e^{-\frac{{(V_p-V_q)}^2}{2{\mathrm{\σ}}_1^2}}$

$w_2(S_p,S_q)=e^{-\frac{{(p\left(x\right)-q\left(x\right))}^2}{2{\mathrm{\σ}}_2^2}}*e^{-\frac{{(p\left(y\right)-q\left(y\right))}^2}{2{\mathrm{\σ}}_2^2}},$

Wherein,

Be the gaussian filtering function, V remarked pixel value, p (x), p (y), q (x), q (y) is the horizontal ordinate of remarked pixel p, q respectively.σ ₁, σ ₂Expression gaussian filtering variance.Wherein, σ ₁, σ ₂Can reach as the case may be needs and is provided with.In one embodiment of the invention, σ ₁=15, σ ₂=5.

Depth value Z (q) to the cum rights repeated factor adds up, and weight is carried out normalized, and the degree of depth that obtains is the degree of depth of improving the back pixel p

$\stackrel{\‾}{Z}\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)Z\left(q\right)}{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)}.$

From following formula as can be known, p is a pixel of present frame, and q is for being the center with current pixel point p, and window width is any pixel in the window of N, N _pThe expression window size.In one embodiment of the invention, window width N is 7, and then window size is 7*7.Z (q) is the depth value of pixel q in the present frame,

Be the depth value after the pixel p improvement in the present frame, w ₁(V _p, V _q) be Gauss's weight factor of the pixel value difference of pixel p and pixel q, w ₂(S _p, S _q) be Gauss's weight factor of the pixel space distance of pixel p and pixel q.

Noise or erroneous matching when the depth map after adopting above-mentioned depth map to improve one's methods to improve not only can filter out depth calculation, and keep the boundary information of present frame ID.

The depth map based on reference frame that the embodiment of the invention provides improves and can use separately or use simultaneously based on the improvement of present frame distribution of color and locus.

S105: output depth map.

As shown in Figure 6, exporting after the depth map after the improvement that to export by the depth map that calculates and the step S104 of step S103 output, the improvement based on reference frame based on the depth map of present frame distribution of color and locus and the plane picture of original input.

According to the depth map acquiring method of the embodiment of the invention, adopt and to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and by utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Improve algorithm by depth map based on bilateral filtering, both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map, and the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

As shown in Figure 7, the embodiment of the invention provides a kind of depth map to ask for device 700, comprise plane picture input module 710, wherein, plane picture input module 710 is used for the input plane image, wherein, plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence; Pretreatment module 720, wherein pretreatment module 720 is used for plane picture is carried out preliminary treatment; Depth map computing module 730, wherein said depth map computing module 730 depth map computing modules are used to judge whether the pretreated plane picture of described pretreatment module needs to carry out that color of image is cut apart and whether need to move the moving average algorithm, to described plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates fast or calculates based on the depth map that color is cut apart; Depth map improves module 740, wherein said depth map improves module 740 and is used for the depth map that described depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame distribution of color and locus, wherein, the described depth map that calculates comprises the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation; With output module 750, wherein said output module 750 be used to export the described depth map that calculates and improve after the depth map based on reference frame, depth map and the described plane picture after improving based on present frame distribution of color and locus.

In one embodiment of the invention, plane picture can change 3D video system, other two-way videos or image sequence output interface or other single channel videos or image sequence output interface from binocular solid collection and three-dimensional Play System, full-automatic 2D video.For binocular solid collection and three-dimensional Play System, two-way video that end collects or the two-way picture frame plane picture as input will be taken.Change the 3D video system for full-automatic 2D video, directly with the plane picture of two frames of the front and back on the time shaft in the 2D video as input.

When the plane picture of plane picture input module 710 inputs is single channel video or two-way video, need carry out decoding processing.Based on different video compression format different video encoding/decoding methods is arranged.Wherein, the decoding to video comprises that reading data flow is converted to suitable frame of video form again from known video.If the frame of video form of decoding output and follow-up depth map are asked for interface and be not inconsistent, then need to carry out the conversion operations of frame of video form.

When the plane picture of plane picture input module 710 inputs is two-way video or two-way image sequence, synchronization picture frame for dual input, with the picture frame of wherein one road video or image sequence as present frame, the picture frame of another road video or image sequence as the reference frame, can be provided with the order of two-path video according to application request.Particularly,, the picture frame of a left side (or right) road video or image sequence is called present frame, the picture frame in same moment of right (or left side) road video or image sequence is called reference frame for binocular solid collection and three-dimensional Play System.

When the plane picture of plane picture input module 710 inputs is single channel video or single channel image sequence, for example change the 3D video system for full-automatic 2D video, be input as single channel 2D video, then the present frame of single channel video on time shaft is called present frame, the back frame on the time shaft is called the default reference frame.

When the plane picture of plane picture input module 710 inputs was single channel video or image sequence, pretreatment module 720 judged whether to carry out key frame and detects.Change the 3D video for real-time 2D video, be input as two picture frames of the front and back on the time shaft in the 2D video, need do a key frame to back one frame this moment and judge, with the image of a frame after confirming and the similitude of current frame image.If the diversity of back one two field picture and current frame image surpasses the threshold value under certain specific tolerance, think that then back one two field picture is a key frame, otherwise think that then back one two field picture is not key frame.

When judging that need carry out key frame detects, at first 720 pairs of front and back two frames of pretreatment module carry out horizontal central line simultaneously and cut apart with median vertical line and cut apart, thereby two frames are divided into four little image blocks respectively before and after making.In one embodiment of the invention, the length of four little image blocks after cutting apart and wide size are original image frame length and roomy little by 1/2.Secondly, pretreatment module 720 is carried out the image block pixel value respectively with four little image blocks of front and back two frame correspondence positions and is subtracted each other, and the absolute value of getting difference to obtain new image block, is asked for pixel average and variance to four new image blocks as new pixel value.Once more, the pixel average of 720 pairs of four new images pieces of pretreatment module and variance are carried out threshold decision respectively.If individual number average that average and variance surpass threshold value in four new images pieces, judges then that afterwards the relative present frame of a frame is a new key frame greater than preset value a.In one embodiment of the invention, preset value a can be 3.In one embodiment of the invention, pixel average and variance threshold values can be the arbitrary value within [20,30].At last, if one frame relative present frame in back is new key frame, then can not be with the reference frame of back one frame as the depth map calculating of present frame, this moment, pretreatment module 720 can be chosen the reference frame of the former frame of present frame as the depth map calculating of present frame.

When the plane picture of plane picture input module 710 inputs was two-way video or two-way image sequence, pretreatment module 720 judged whether to carry out outer level line and proofreaies and correct.For binocular solid collection and three-dimensional Play System, two-way video or the two-way image sequence of importing carried out preliminary treatment, need proofread and correct the outer level of two-path video frame or image sequence.Outer level line is proofreaied and correct and can be made certain the delegation's pixel of sustained height in the corresponding reference frame of present frame pixel, has eliminated the deviation on the vertical direction.Thereby can reduce in the corresponding reference frame of this pixel with the search dimension of the corresponding pixel points in the time chart picture frame, reduce amount of calculation.

Level line timing outside judgement need be carried out, pretreatment module 720 is unique corresponding with a certain pixel in the identical time chart picture frame in any pixel in a certain time chart picture frame in the road video in the two-way video and another road video, thereby the jobbie point in the space is gathered in common expression.If but corresponding degree of depth the unknown of some pixels in a certain picture frame in one road video, this pixel may appear on the optional position of a certain straight line in one picture frame of back with the unique corresponding pixel points in the time chart picture frame in another road video so.Be on this straight line in the back picture frame all pixels all may with same object point in the corresponding space of this pixel.This straight line is the outer level of this pixel correspondence line.Generally speaking, outer level line is not a horizontal linear, but an oblique line.Proofread and correct the corresponding outer level line that can make in corresponding another picture frame of this pixel by outer level line and be transformed to horizontal direction, thereby eliminated departing from the vertical direction, can dwindle in corresponding another road video of this pixel search dimension, thereby reduce amount of calculation with the unique corresponding pixel points in the time chart picture frame.Certainly it will be appreciated by persons skilled in the art that and to adopt other outer level line bearing calibrations the outer level line correction of two-way video or image sequence.

In one embodiment of the invention, the plane picture of plane picture input module 710 input also can be without the preliminary treatment of pretreatment module 720 and directly as the input of depth map computing module 730.

As shown in Figure 8, depth map computing module 730 comprise judging unit 731, the basic real-time computing unit 732 of depth map, basic depth map computing unit 733, the depth map computing unit of cutting apart based on color 734 and the quick computing unit of cutting apart based on color 735 of depth map.

At first, 731 pairs of plane pictures from plane picture input module 710 or pretreatment module 720 of judging unit judge whether to carry out color of image and cut apart.When judging that plane picture does not need to carry out color of image when cutting apart, and judges further whether plane picture needs to use the moving average algorithm.

When judging unit 731 judged that plane picture does not need to use the moving average algorithm, basic depth map computing unit 733 calculated the basic depth map of plane picture.

At first basic depth map computing unit 733 is provided with the size of window, and the window width that window is set is odd number N, includes N*N pixel in window width is the window of N.With candidate's respective pixel on the region of search in any one pixel in the present frame and the reference frame respectively as center pixel.Wherein, center pixel is positioned at the center of window.All pixels that comprise in the window become (comprising center pixel itself) the support pixel of center pixel.In one embodiment of the invention, window width N can be 15.Certainly it will be appreciated by persons skilled in the art that window width N can also be other odd number width.

Basic depth map computing unit 733 will run into the situation that does not have pixel on every side of this pixel when the ranks pixel of the pixel at four angles of each image or four peripheries is set up the window of pre-set dimension as center pixel.At this moment, can adopt one of following dual mode that window is set.

1) do not exist the pixel value of pixel to be set to 0 around this pixel.

2) window size of directly setting up the corresponding window of above-mentioned pixel is set to 1 * 1 pixel (being about to this pixel itself as a window).

Certainly it will be appreciated by persons skilled in the art that when adopt additive method solve this pixel around when not having under the situation of pixel the problem of window to be set, above-mentioned additive method also falls into protection scope of the present invention.

For any one pixel of present frame, basic depth map computing unit 733 is provided with the hunting zone that the region of search is promptly set the candidate corresponding pixel points of this pixel in reference frame.The region of search is represented in the present frame relative position relation of candidate's respective pixel in the pixel and reference frame arbitrarily.In one embodiment of the invention, the region of search can be according to actual concrete condition of attending with flexible processing.

The outer level of basic depth map computing unit 733 pairs of processes image line is proofreaied and correct and is eliminated later present frame and the reference frame of vertical missing, and the region of search can be set to [d _Min, d _Max]=[0,32], wherein d is the difference of waiting arbitrarily in reference frame candidate corresponding pixel points and the present frame to ask between the horizontal coordinate of degree of depth pixel, d _MinThe coordinate of candidate's respective pixel waits to ask the pixel coordinate of the degree of depth consistent with present frame in=0 expression reference frame, and two pixels are in the same position in reference frame and the present frame respectively.

Basic depth map computing unit 733 calculates based on the pixel coupling of window and can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈N_p,{\stackrel{\‾}{q}}_d\∈N_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, p represents any one pixel in the present frame;

Expression is candidate's corresponding pixel points in the reference frame of d apart from current frame pixel point p relative position; N _pExpression is all pixels that window was comprised (comprising pixel p itself) of center pixel with pixel p;

Expression is with candidate pixel point in the reference frame

For all pixels that window comprised of center pixel (comprise pixel

Itself); P with

Corresponding window width is N; Q represents in the present frame with p to be any one pixel in the window of center pixel,

Represent in the reference frame with the candidate pixel point

Be any one pixel in the window of center pixel,

Relative position in window is identical with q, Remarked pixel point q with

The absolute value of margin of image element.

Basic depth map computing unit 733 at first obtains the pixel parallax of each pixel in the present frame.For any one the pixel p in the present frame, at the region of search [d _Min, d _Max] in choose candidate pixel point in one of them d corresponding reference frame, the absolute difference and the summation of calculating all pixels in these two central pixel point place windows obtain E _d, to interval [d _Min, d _Max] interior all corresponding E of any d _d, select all E at last _dThe parallax value of minimum value correspondence as the parallax value d of current pixel point in the present frame _s, simultaneously with parallax value d _sCandidate pixel point in the corresponding reference frame

As the final respective pixel of pixel p, wherein,

The employing said method can obtain the time difference of all pixels of present frame, according to the parallax d of all pixels of present frame that obtain _{(i, j)}=d _s, d wherein _{(i, j)}Be corresponding pixel points (i, parallax j).According to the depth map of the present frame correspondence that obtains,

$Z_{(i,j)}=\frac{d_{(i,j)}-d_{\min }}{(d_{\max }-d_{\min })}*255$

Z wherein _{(i, j)}Be corresponding pixel points (i, degree of depth j).

When judging unit 731 judged that plane picture need use the moving average algorithm, the real-time computing unit 732 of basic depth map calculated the basic real-time deep figure of plane picture.For the region of search [d _Min, d _Max] in each parallax d, for each pixel in the present frame, all need to calculate in its place window the pixel value difference of the respective pixel in the window of candidate pixel point place in each pixel and reference frame.

When being input as long and wide the be present frame of W*H and reference frame, window size is N*N, search length is L, wherein for each pixel in the present frame and the candidate pixel point in the reference frame in the hunting zone, every calculating once, amount of calculation is N*N time a subtraction, N*N-1 time addition, and final amount of calculation is W*H*L* (N*N) * (N*N-1).This amount of calculation is directly proportional with the biquadratic of window width N, because N is variable, when N increased, then amount of calculation will be geometric progression increased.

Basic average initialization calculating and the average recursive calculation of line slip of the line slip of the real-time computing unit 732 of depth map on line direction, and the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain the real-time window matching value Q of any pixel correspondence of described present frame _{(i, j)}

At first calculate in the initialization of the up moving average of line direction.First pixel (i, 0) to each row in the present frame is interior to pixel value difference absolute value C in window line width scope [0, N] _{(i, j)}Carry out accumulation calculating.Wherein,

j∈[0，N]，A _(i，0)＝A _(i，0)+C _(i，j)

Wherein,

Any i of delegation, A are got in expression _{(i, 0)}The C of first pixel (i, 0) in window line width scope that represents any i of delegation _{(i, j)}Accumulation calculating result, C _{(i, j)}Support pixel in the expression window line width scope (i, j) with reference frame in the pixel value difference absolute value of respective pixel, N represents window width.

The basic real-time computing unit 732 of depth map averages recursive calculation to line slip then.Particularly, adopt following formula that any capable i is carried out the average recursive calculation of line slip:

$A_{(i,j)}=A_{(i,j-1)}+C_{(i,j+\frac{N}{2})}-C_{(i,j-\frac{N}{2}-1)},$

Wherein, for the center pixel of the capable j of i row (i, j), the sliding average A on its corresponding row direction _{(i, j)}Line slip mean value A by previous pixel _{(i, j-1)}Add the current window right endpoint

Result of calculation

And, deduct the left end point of previous window again

Result of calculation Recursive calculation obtains.

Again, the initialization of basic 732 pairs of above-listed moving averages of column direction of the real-time computing unit of depth map is calculated.Particularly, to first pixel of any row j in the present frame (0, j) adopt following formula in the window column wide region, line slip mean value to be carried out accumulation calculating:

i∈[0，N]，Q _(0，j)＝Q _(0，j)+A _(i，j)，

Wherein Represent to appoint and get a j, Q _{(0, j)}Represent that (0, j) A is calculated in the accumulation to line slip mean value to any first pixel of row j in the window column wide region _{(i, j)}(N represents window width to pixel for i, line slip mean value j) in the expression present frame.

The last basic real-time computing unit 732 of depth map is listed as the average recursive calculation of slip.Particularly, adopt following formula to carry out column direction row moving average recursive calculation for any row j:

$Q_{(i,j)}=Q_{(i-1,j)}+A_{(i+\frac{N}{2},j)}-A_{(i-\frac{N}{2}-1,j)},$

Q wherein _{(i, j)}Represent any one row j the capable pixel of i (i, j) the recursive calculation result of row sliding average in the window column wide region is by the row sliding average Q of a last pixel _{(i-1, j)}With the current window lower extreme point

Line slip mean value and, deduct a window upper extreme point again

Line slip mean value recursive calculation obtain.

Any pixel in present frame of the basic real-time computing unit 732 of depth map (i, j) corresponding to each relative position among the d of particular search space to a plurality of described real-time window matching value Q should be arranged _{(i, j)}, for the region of search [d _Min, d _Max] interior a plurality of real-time window matching value Q _{(i, j)}In choose the corresponding parallax value of minimum value as current pixel point in the present frame (i, parallax value d j) _S1Thereby, can obtain the depth value of respective pixel.

$d_{s1}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{Q_{(i,j)}{|}_d\right\}.$

The basic real-time computing unit 732 of depth map is asked for the real-time pixel parallax of each pixel of present frame, thereby can obtain the basic real-time deep figure of present frame correspondence.

Because in the ordinary course of things, the corresponding same object of the image-region of same color, depth value is same or similar; And the corresponding different objects of the image-region of different colours, the degree of depth is inconsistent.Therefore can utilize color of image to cut apart the classified information that obtains, based on the window calculation depth map time, increase belongs to the shared weight of of a sort other pixels with the window center pixel, reduces not belong to the shared weight of of a sort other pixels with the window center pixel.

The depth map that the depth map computing unit of cutting apart based on color 734 calculates based on image segmentation.

At first, the depth map computing unit of cutting apart based on color 734 is set up graph model V (i j), is connected to 8 limits with each pixel (i) of input present frame and 8 pixels (j) around it.Wherein, two end points on every limit be two pixels (i, j), the weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (i) |, obtain thus a series of limit edge (i, j).

Under initial situation, the depth map computing unit of cutting apart based on color 734 is a class with each pixel (i), and setting the initial merger threshold value of class this moment is threshold (i)=c, is sorted from small to large according to weights in every limit in the above-mentioned graph model then.

Secondly, and pairing two pixels in limit after 734 pairs of orderings of depth map computing unit of cutting apart based on color (i, j) the place class is carried out the operation of merger in twos, and wherein, merger comprises the steps:

As a limit edge (i, j) weight w (i, j) than two end points (i on this limit, j) the merger threshold value threshold (i) of place class, threshold (j) hour, then with this limit edge (i, the two class merger at two pixel places j) are a class, the merger threshold value of upgrading the new class obtain simultaneously is:

$\mathrm{threshold}(i,j)=w(i,j)+\frac{c}{\mathrm{num}(i,j)},$

Wherein, num (i, j) the number of pixels addition of the original place of remarked pixel point i and j class.

Once more, the fewer class of number of pixels that obtains after 734 pairs of merger of depth map computing unit of cutting apart based on color is forced the merger operation, travel through all limit edge (i, j), if exist two pixels on any limit not at same time-like, and the number of pixels of the class at two pixel places all is less than a specified value minSize, so just two classes at two pixel places carried out the merger operation, and the color that finally obtains after each pixel is sorted out is cut apart figure.

Then, the plane picture after the depth map computing unit of cutting apart based on color 734 is cut apart color according to window carries out calculating based on the pixel coupling of window;

Can adopt following formula to calculate based on the pixel coupling calculating that color is cut apart,

$E_d={\mathrm{\Σ}}_{q\∈{\mathrm{NS}}_p,{\stackrel{\‾}{q}}_d\∈{\mathrm{NS}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d)+\mathrm{\λ}\×{\mathrm{\Σ}}_{q\∈{N\stackrel{\‾}{S}}_p,{\stackrel{\‾}{q}}_d\∈{N\stackrel{\‾}{S}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, NS _pExpression with pixel p be center pixel window was comprised and pixel p belongs to the pixel (comprising pixel p itself) of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that belongs to same color block (comprises pixel

Itself);

Expression with pixel p be center pixel window was comprised and pixel p does not belong to the pixel of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that does not belong to same color block; λ represents not belong to center pixel the accumulation calculating weight of the area pixel of same block.In one embodiment of the invention, λ can be 0.01.

At last, obtain each pixel pixel parallax in the present frame, obtain the basic depth map of present frame correspondence according to window matching value and search volume.

$d_s=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{E_d\right\}.$

The real-time deep figure that the quick computing unit of cutting apart based on color 735 of depth map calculates based on image segmentation.For each row of present frame, the quick computing unit of cutting apart based on color 735 of depth map makes the line slip of all blocks on average be initially zero, i.e. T _S=0, wherein s is the block under the pixel.Then each row is carried out calculating based on the average initialization of the line slip of carve information, promptly in window line width scope to pixel value difference according to the place block, carry out accumulation calculating.

j∈[0，N]，T _S(i，j)＝T _S(i，j)+C _(i，j)

In the following formula, T _{S (i, j)}(i, j) the affiliated sliding average of block on line direction is by belonging to block S (i, C j) for remarked pixel _{(i, j)}Adding up obtains, C _{(i, j)}Pixel in the expression present frame (i, j) with reference frame in the pixel value difference absolute value of respective pixel.Calculate by following formula, can obtain any delegation based on the average initial calculation result of the line slip of carve information.

The quick computing unit of cutting apart based on color 735 of depth map is based on the average recursive calculation of the line slip of the carve information of image segmentation.Adopt following formula that any delegation is carried out based on the average recursive calculation of the line slip of carve information:

$T_{S(i,j+\frac{N}{2})}=T_{S(i,j+\frac{N}{2})}+C_{S(i,j+\frac{N}{2})}$

$T_{S(i,j-\frac{N}{2}-1)}=T_{S(i,j-\frac{N}{2}-1)}-C_{S(i,j-\frac{N}{2}-1)},$

In following formula, (i, j), the result of calculation of the right endpoint of its corresponding window is the center pixel that is listed as for the capable j of i

The result of calculation of the left end point of the previous window of its correspondence is Respectively according to the block at its place

Carry out addition and subtraction, upgrade block T _SThe average recursive calculation result of line slip.Finally for each center pixel (i, j), obtain based on the line slip average computation result of carve information be

$A_{(i,j)}^r=T_{S(i,j)}.$

The quick computing unit of cutting apart based on color 735 of depth map calculates based on the initialization of the row moving average of the carve information of image segmentation.For each row of present frame, make the row moving average of all blocks be initially zero, i.e. G _S=0, wherein s is the block under the pixel, and G is a row moving average result of calculation.Then each row is carried out calculating based on the row moving average initialization of carve information, promptly in the window column wide region to based on the line slip average computation result of carve information according to the place block, carry out accumulation calculating.

i∈[0，N]， $G_{S(i,j)}=G_{S(i,j)}+A_{(i,j)}^r$

In following formula, G _{S (i, j)}Remarked pixel (i, j) the window moving average on the affiliated block column direction, Pixel (i, j) the line slip average computation result on line direction in the expression present frame based on carve information.Calculate by following formula, can obtain for the initial calculation result of any rows of directions based on the row moving average of carve information.

The quick computing unit of cutting apart based on color 735 of depth map is based on the row moving average recursive calculation of the carve information of image segmentation.Adopt following formula to carry out row moving average recursive calculation to any row based on carve information:

$G_{S(i+\frac{N}{2},j)}=G_{S(i+\frac{N}{2},j)}+A_{S(i+\frac{N}{2},j)}^r$

$G_{S(i-\frac{N}{2}-1,j)}=G_{S(i-\frac{N}{2}-1,j)}-A_{S(i-\frac{N}{2}-1,j)}^r,$

In following formula, (i, j), the line slip average computation result of its corresponding window lower extreme point is to be listed as the capable center pixel of i for j

The line slip average computation result of the upper extreme point of a window is on its respective column direction

Respectively according to the block at its place

Carry out addition and subtraction, recurrence is upgraded block G _SRow moving average result.(i, j) the window accumulation result of calculation based on carve information is G finally to obtain any pixel _{S (i, j)}

Ask for parallax and depth map based on the quick computing unit 735 of depth map that color is cut apart based on the carve information of image segmentation.By obtaining any pixel (i, basic window accumulation result of calculation Q j) _{(i, j)}With window accumulation result of calculation G based on the color carve information _{S (i, j)}, utilize The above results to adopt the basic depth map of following improvement to ask for algorithm in real time:

M(i，j)| _d＝λ(Q _(i，j)| _d-G _S(i，j)| _d)+G _S(i，j)| _d

In following formula | _dExpression is for d specific in the hunting zone, and (i, j) the final window of expression is accumulated result of calculation, Q to M _{(i, j)}-G _{S (i, j)}When expression window accumulation is calculated and center pixel (i j) does not belong to the accumulation calculating section of same block, G _{S (i, j)}(i j) belongs to the accumulation calculating section of same block, and λ represents that the accumulation that does not belong to the area pixel of same block with center pixel calculates weight for expression and center pixel.In one embodiment of the invention, λ can be 0.01.(i, the parallax value of minimum value correspondence j) is the parallax d of current frame pixel finally to get M _S2

$d_{s2}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{M(i,j){|}_d\right\}.$

In one embodiment of the invention, based on the calculating of the algorithm of four kinds of different characteristics to the depth map of present frame, the quick calculating of the calculating of the depth map that is the real-time calculating of the calculating of basic depth map, basic depth map, cuts apart based on color and the depth map cut apart based on color, four kinds of algorithms are separate, and the promptly basic real-time computing unit 732 of depth map, basic depth map computing unit 733, the depth map computing unit of cutting apart based on color 734 and the quick computing unit of cutting apart based on color 735 of depth map can independent operating compute depth figure.Certainly depth map computing unit 734 that it will be appreciated by persons skilled in the art that the real-time computing unit 732 of basic depth map, basic depth map computing unit 733, cuts apart based on color and the quick computing unit 735 of depth map cut apart based on color can independent operating compute depth figure independence or comprehensive use all belong to protection scope of the present invention.

732 pairs of basic depth map computing units 733 of the real-time computing unit of the basic depth map of the embodiment of the invention improve on computation complexity, have reached the real-time requirement.The depth map computing unit of cutting apart based on color 734 of the embodiment of the invention further improves calculating on the accuracy the real-time computing unit 732 of basic depth map, and the quick computing unit of cutting apart based on color 735 of depth map has further carried out improving on computational speed to the real-time computing unit 734 of basic depth map again and obtained real-time operation, depth map that quality is higher.

Depth map improves depth map that 740 pairs of depth map computing modules 730 of module calculate to carry out based on the improvement of the depth map of reference frame with based on the improvement of present frame distribution of color and locus.Wherein, the depth map that calculates of depth map computing module 730 comprise the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation.

Depth map improves module 740 and judges whether to carry out improvement based on the reference frame depth map.When judging the improvement to carry out based on the reference frame depth map, the depth map that calculates is asked for, searches, checked and improves.At first, depth map improves module 740 is asked for reference frame according to the depth map that calculates depth map.Only need depth map calculate interface will before reference frame as present present frame, with before present frame as present reference frame, the depth map of reference frame correspondence before can calculating.Then, by certain the pixel P in the present frame _C(i, degree of depth Z j) _C(i j) can find pixel P corresponding in the reference frame _R(i, j), then the corresponding degree of depth of this pixel in the reference frame depth map is Z _R(i, j).

Depth map improves the improvement that module 740 is carried out based on the reference frame depth map.At first, check depth map.If Z _C(i, j) and Z _R(i, j) difference of two depth values is less than or equal to predetermined threshold t, then represents two pixel P _C(i, j) and P _R(i, j) corresponding same object point can judge that then depth calculation is accurate; If Z _C(i, j) and Z _R(i when j) difference of two depth values is greater than predetermined threshold t, represents that then two pixels are not corresponding, can assert that present frame is at pixel P _C(i, the depth calculation mistake of j) locating.Predetermined threshold t can reach needs as the case may be and set.In one embodiment of the invention, predetermined threshold t can be 1.If the depth calculation mistake then can be got Z _C(i, j) and Z _R(i, j) smaller between the two is current frame pixel point P _C(i, degree of depth j), thus reach the purpose of improving the depth map quality.

Depth map improves module 740 and further judges whether to carry out improvement based on present frame distribution of color and locus.Refer to that based on the present frame distribution of color and the improvement of locus the ID value can carry out bilateral filtering by distribution of color in the pixel value place window area of its corresponding present frame and locus and improve.

When improvement based on the reference frame depth map is carried out in judgement, pixel Z (p) for any the expression degree of depth among the ID figure, its respective pixel in present frame is p, is center pixel with the pixel p in the present frame, and window width is that all pixels in the window of N are q.The pixel value difference Gauss weight factor w of all pixel q and center pixel p in the calculation window ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q), and the depth value Z (q) of two weight factors and pixel q multiplied each other, obtain the depth value Z (q) of cum rights repeated factor.Pixel value difference Gauss weight factor w ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q) be respectively:

$w_1(V_p,V_q)=e^{-\frac{{(V_p-V_q)}^2}{2{\mathrm{\σ}}_1^2}}$

$w_2(S_p,S_q)=e^{-\frac{{(p\left(x\right)-q\left(x\right))}^2}{2{\mathrm{\σ}}_2^2}}*e^{-\frac{{(p\left(y\right)-q\left(y\right))}^2}{2{\mathrm{\σ}}_2^2}},$

Wherein,

Be the gaussian filtering function, V remarked pixel value, p (x), p (y), q (x), q (y) is the horizontal ordinate of remarked pixel p, q respectively.σ ₁, σ ₂Expression gaussian filtering variance.Wherein, σ ₁, σ ₂Can reach as the case may be needs and is provided with.In one embodiment of the invention, σ ₁=15, σ ₂=5.

The depth value Z (q) that depth map improves 740 pairs of cum rights repeated factors of module adds up, and weight is carried out normalized, and the degree of depth that obtains is the degree of depth of improving the back pixel p

$\stackrel{\‾}{Z}\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)Z\left(q\right)}{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)}.$

From following formula as can be known, p is a pixel of present frame, and q is for being the center with current pixel point p, and window width is any pixel in the window of N, N _pThe expression window size.In one embodiment of the invention, window width N is 7, and then window size is 7*7.Z (q) is the depth value of pixel q in the present frame,

Be the depth value after the pixel p improvement in the present frame, w ₁(V _p, V _q) be Gauss's weight factor of the pixel value difference of pixel p and pixel q, w ₂(S _p, S _q) be Gauss's weight factor of the pixel space distance of pixel p and pixel q.

Noise or erroneous matching when the depth map after adopting above-mentioned depth map to improve one's methods to improve not only can filter out depth calculation, and keep the boundary information of present frame ID.

The depth map based on reference frame that the embodiment of the invention provides improves and can use separately or use simultaneously based on the improvement of present frame distribution of color and locus.

Depth map that output module 750 will be calculated by depth map computing module 730 and depth map improve the depth map based on reference frame after the improvement of module 740, the plane picture based on the original input of the depth map of present frame distribution of color and locus and plane picture input module 710 after improving is exported.

Depth map according to the embodiment of the invention is asked for device, adopts to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and by utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Improve algorithm by depth map based on gaussian filtering, both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map, and the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

In the description of this specification, concrete feature, structure, material or characteristics that the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means in conjunction with this embodiment or example description are contained at least one embodiment of the present invention or the example.In this manual, the schematic statement to above-mentioned term not necessarily refers to identical embodiment or example.And concrete feature, structure, material or the characteristics of description can be with the suitable manner combination in any one or more embodiment or example.

Although illustrated and described embodiments of the invention, for the ordinary skill in the art, be appreciated that without departing from the principles and spirit of the present invention and can carry out multiple variation, modification, replacement and modification that scope of the present invention is by claims and be equal to and limit to these embodiment.

Claims (20)

1. a depth map acquiring method is characterized in that, comprises the steps:

The input plane image, wherein, described plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Described plane picture is carried out preliminary treatment;

Judge whether pretreated plane picture needs to carry out that color of image is cut apart and whether need to move the moving average algorithm, to described plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates and calculates fast based on the depth map that color is cut apart;

The depth map that calculates is carried out improving and based on the improvement of present frame distribution of color and locus based on the depth map of reference frame, wherein, the described depth map that calculates comprise the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation; And

Export the described depth map that calculates and improve after the depth map based on reference frame, depth map and the described plane picture after improving based on present frame distribution of color and locus.

2. depth map acquiring method as claimed in claim 1, it is characterized in that, when the plane picture of input is two-way video or two-way image sequence, for the synchronization picture frame, with the picture frame in one road video or the image sequence as present frame, with the picture frame in another road video or the image sequence as the reference frame;

When the planar graph of input when being single channel video or single channel image sequence, the current image frame on time shaft is as present frame with described single channel video or single channel image sequence, with the back frame of the current image frame on the time shaft as the default reference frame.

3. depth map acquiring method as claimed in claim 2 is characterized in that, when described plane picture is single channel video or single channel image sequence, described plane picture is carried out preliminary treatment comprise the steps:

Described single channel video or single channel image sequence are carried out the key frame judgement to obtain the reference frame as the depth map calculating of present frame;

When described plane picture is two-way video or two-way image sequence, described plane picture is carried out preliminary treatment comprise the steps: described two-way video or two-way image sequence are carried out the outer level line correction of image.

4. depth map acquiring method as claimed in claim 1 is characterized in that, judges whether pretreated plane picture needs to carry out color of image and cut apart,

When judging that described pretreated plane picture does not need to carry out color of image when cutting apart, need further to judge whether utilization moving average algorithm, described plane picture is carried out the real-time calculating of basic depth map when judging when needing utilization moving average algorithm; Otherwise, described plane picture is carried out the calculating of basic depth map;

When judging that described pretreated plane picture need carry out color of image when cutting apart, need further to judge whether utilization moving average algorithm, when judgement needs utilization moving average algorithm, described plane picture is carried out calculating fast based on the depth map that color is cut apart; Otherwise the depth map that described plane picture is carried out cutting apart based on color calculates.

5. depth map acquiring method as claimed in claim 4 is characterized in that, the described calculating that plane picture is carried out basic depth map comprises the steps:

Window and search volume are set;

According to described window described plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with described search volume, obtain the basic depth map of present frame correspondence, comprise the steps:

Any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described window matching value E should be arranged _d, at described a plurality of window matching value E _dIn choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _s

6. depth map acquiring method as claimed in claim 4 is characterized in that, described plane picture is carried out the real-time calculating of basic depth map, comprises the steps:

Initialization calculating that line slip on line direction is average and the average recursive calculation of line slip, and the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain the real-time window matching value Q of any pixel correspondence of described present frame _{(i, j)}

Any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described real-time window matching value Q should be arranged _{(i, j)}, at described a plurality of real-time window matching value Q _{(i, j)}In choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _S1

Ask for the real-time pixel parallax of each pixel of present frame, obtain the basic real-time deep figure of present frame correspondence.

7. depth map acquiring method as claimed in claim 4 is characterized in that, the described depth map that plane picture is carried out cutting apart based on color calculates and comprises the steps:

Each pixel of described present frame is connected to n bar limit with n pixel around corresponding, weight w (the i on every limit, j) be the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) |, obtain a plurality of limit edge (i, j), wherein, i is a pixel of present frame, and j is on every side n the pixel corresponding with a pixel of described present frame;

Each pixel of described present frame is divided into a class, and setting initial merger threshold value is threshold (i)=c, is sorted from small to large according to weights in described every limit;

The pixel of the pairing present frame in limit after the ordering and on every side n the pixel place class corresponding with the pixel of described present frame are carried out the operation of merger in twos, and the color that obtains after each pixel is sorted out is cut apart figure;

Window and search volume are set, in described window according to the color segmentation result, window area is divided into two parts, a part of and currently wait to ask degree of depth pixel to belong to same color region, another part and currently wait to ask degree of depth pixel not belong to same color region, reduce the latter's calculating weight, described plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with described search volume, obtain the depth map of cutting apart based on color of image of present frame correspondence, comprise the steps:

Any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described window matching value E should be arranged _d, at described a plurality of window matching value E _dIn choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _s

8. depth map acquiring method as claimed in claim 4 is characterized in that, described plane picture is carried out calculating fast based on the depth map that color is cut apart, and comprises the steps:

Described plane picture is carried out based on the line slip of image segmentation average initialization calculating and the average recursive calculation of line slip, and carry out initialization calculating and row moving average recursive calculation based on the row moving average of image segmentation, obtain window matching value G based on image segmentation _{S (i, j)}

According to described real-time window matching value Q _{(i, j)}With described window matching value G based on image segmentation _{S (i, j)}, ask for real-time window matching value M based on image segmentation (i, j);

Any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described real-time window matching value M (i based on image segmentation should be arranged, j), (i chooses the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame in j) at described real-time window matching value M based on image segmentation _S2

Ask for the real-time pixel parallax of each pixel of present frame, obtain the real-time deep figure based on image segmentation of present frame correspondence.

9. depth map acquiring method as claimed in claim 1 is characterized in that, described the depth map that calculates is carried out improving based on the depth map of reference frame, comprises the steps:

The position of reference frame and present frame is exchanged, calculated the depth map of reference frame;

Each pixel in the described present frame is searched the corresponding pixel points in the described reference frame and the depth value of described corresponding pixel points;

Depth value according to corresponding pixel points in the described reference frame is checked the depth value of each pixel in the present frame, comprise: when the difference of the depth value of corresponding pixel points in the depth value of a pixel in the described present frame and the described reference frame is less than or equal to predetermined threshold, judge that then the depth value of this pixel in the described present frame is correct; When the difference of the depth value of corresponding pixel points in the depth value of certain pixel in the described present frame and the described reference frame during, then judge the depth value mistake of this pixel in the described present frame greater than described predetermined threshold;

In judging described present frame during the depth value mistake of a pixel, the depth value of corresponding pixel points in the depth value of this pixel in the described present frame and the described reference frame is compared, comprising:

When the depth value of this pixel in the described present frame less than described reference frame in the depth value of corresponding pixel points, keep the depth value of this pixel in the described present frame;

When the depth value of this pixel in the described present frame greater than described reference frame in the depth value of corresponding pixel points, the depth value of this pixel in the described present frame is the depth value of corresponding pixel points in the described reference frame.

10. depth map acquiring method as claimed in claim 1 is characterized in that, described the depth map that calculates is carried out improvement based on present frame distribution of color and locus, comprises the steps:

Any depth value in the described depth map that calculates is carried out bilateral filtering according to distribution of color and locus in its corresponding pixel points place window in present frame.

11. a depth map is asked for device, it is characterized in that, comprising:

The plane picture input module, described plane picture input module is used for the input plane image, and wherein, described plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Pretreatment module, described pretreatment module are used for described plane picture is carried out preliminary treatment;

The depth map computing module, described depth map computing module is used to judge whether the pretreated plane picture of described pretreatment module needs to carry out that color of image is cut apart and whether need to move the moving average algorithm, to described plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates fast or calculates based on the depth map that color is cut apart;

Depth map improves module, described depth map improves module and is used for the depth map that described depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame distribution of color and locus, wherein, the described depth map that calculates comprises the basic depth map that calculates by basic depth map, by the basic real-time deep figure of calculating in real time of basic depth map, by the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast by the depth map of cutting apart based on color based on image segmentation; And

Output module, described output module be used to export the described depth map that calculates and improve after the depth map based on reference frame, depth map and the described plane picture after improving based on present frame distribution of color and locus.

12. depth map as claimed in claim 11 is asked for device, it is characterized in that, when the plane picture of described plane picture input module input is two-way video or two-way image sequence, for the synchronization picture frame, with the picture frame in one road video or the image sequence as present frame, with the picture frame in another road video or the image sequence as the reference frame;

When the planar graph of described plane picture input module input is single channel video or single channel image sequence, current image frame on time shaft is as present frame with described single channel video or single channel image sequence, with the back frame of the current image frame on the time shaft as the default reference frame.

13. depth map as claimed in claim 11 is asked for device, it is characterized in that, when described plane picture was single channel video or single channel image sequence, described pretreatment module was carried out the key frame judgement to obtain the reference frame as the depth map calculating of present frame to described single channel video or single channel image sequence;

When described plane picture was two-way video or two-way image sequence, described pretreatment module was carried out the outer level line correction of image to described two-way video or two-way image sequence.

14. depth map as claimed in claim 11 is asked for device, it is characterized in that, described depth map computing module comprises judging unit, the basic real-time computing unit of depth map, basic depth map computing unit, the depth map computing unit of cutting apart based on color and the quick computing unit of cutting apart based on color of depth map

Wherein, whether the pretreated plane picture of described judgment unit judges needs to carry out color of image cuts apart, when judging that described pretreated plane picture does not need to carry out color of image when cutting apart, need further to judge whether utilization moving average algorithm, when judgement needed utilization moving average algorithm, the real-time computing unit of described basic depth map carried out the real-time calculating of basic depth map to described plane picture; Otherwise described basic depth map computing unit carries out the calculating of basic depth map to described plane picture;

When the described pretreated plane picture of described judgment unit judges need carry out color of image when cutting apart, need further to judge whether utilization moving average algorithm, when judgement needed utilization moving average algorithm, the described quick computing unit of cutting apart based on color deeply of depth map carried out calculating fast based on the depth map that color is cut apart to described plane picture; Otherwise the described depth map computing unit of cutting apart based on color calculates the depth map that described plane picture carries out cutting apart based on color.

15. depth map as claimed in claim 14 is asked for device, it is characterized in that, described basic depth map computing unit is provided with window and search volume, according to described window described plane picture is carried out calculating based on the pixel coupling of window, obtains window matching value E _d, and according to window matching value E _dObtain each pixel pixel parallax in the present frame with described search volume, obtain the basic depth map of present frame correspondence, be included in any pixel in the present frame corresponding to each relative position in the described search volume to a plurality of described window matching value E should be arranged _d, and at described a plurality of window matching value E _dIn choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _s

16. depth map as claimed in claim 14 is asked for device, it is characterized in that, initialization calculating that the line slip of the real-time computing unit of described basic depth map on line direction is average and the average recursive calculation of line slip, and the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain the real-time window matching value Q of any pixel correspondence of described present frame _{(i, j)}And any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described real-time window matching value Q should be arranged _{(i, j)}, at described a plurality of real-time window matching value Q _{(i, j)}In choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _S1, and ask for the real-time pixel parallax of each pixel of present frame, obtain the basic real-time deep figure of present frame correspondence.

17. depth map as claimed in claim 14 is asked for device, it is characterized in that, the described depth map computing unit of cutting apart based on color comprises the steps:

Each pixel of described present frame is connected to n bar limit with n pixel around corresponding, weight w (the i on every limit, j) be the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) |, obtain a plurality of limit edge (i, j), wherein, i is a pixel of present frame, j is on every side n the pixel corresponding with a pixel of described present frame, each pixel of described present frame is divided into a class, setting initial merger threshold value is threshold (i)=c, sorted from small to large according to weights in described every limit, and to the pixel of the pairing present frame in limit after the ordering and corresponding with the pixel of described present frame around n pixel place class carry out merger in twos and operate, the color that obtains after each pixel classification is cut apart figure;

Window and search volume are set, in described window according to the color segmentation result, window area is divided into two parts, a part of and currently wait to ask degree of depth pixel to belong to same color region, another part and currently wait to ask degree of depth pixel not belong to same color region, reduce the latter's calculating weight, described plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with described search volume, obtain the depth map of cutting apart based on color of image of present frame correspondence, comprise the steps:

Any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described window matching value E should be arranged _d, at described a plurality of window matching value E _dIn choose the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame _s

18. depth map as claimed in claim 14 is asked for device, it is characterized in that, the described quick computing unit of cutting apart based on color of depth map carries out based on the line slip of image segmentation average initialization calculating and the average recursive calculation of line slip described plane picture, and carry out initialization calculating and row moving average recursive calculation based on the row moving average of image segmentation, obtain window matching value G based on image segmentation _{S (i, j)}, according to described real-time window matching value Q _{(i, j)}With described window matching value G based on image segmentation _{S (i, j),}Ask for real-time window matching value M (i based on image segmentation, j), any pixel in present frame corresponding to each relative position in the described search volume to a plurality of described real-time window matching value M (i based on image segmentation should be arranged, j), (i chooses the parallax value d of the parallax value of minimum value correspondence as current pixel point in the described present frame in j) at described real-time window matching value M based on image segmentation _S2, ask for the real-time pixel parallax of each pixel of present frame, obtain the real-time deep figure based on image segmentation of present frame correspondence.

19. depth map as claimed in claim 11 is asked for device, it is characterized in that, described depth map improves module and the position of reference frame and present frame is exchanged the depth map that calculates reference frame, each pixel in the described present frame is searched the corresponding pixel points in the described reference frame and the depth value of described corresponding pixel points, depth value according to corresponding pixel points in the described reference frame is checked the depth value of each pixel in the present frame, comprise: when the difference of the depth value of corresponding pixel points in the depth value of a pixel in the described present frame and the described reference frame is less than or equal to predetermined threshold, judge that then the depth value of this pixel in the described present frame is correct; When the difference of the depth value of corresponding pixel points in the depth value of a pixel in the described present frame and the described reference frame during, then judge the depth value mistake of this pixel in the described present frame greater than described predetermined threshold;

In judging described present frame during the depth value mistake of a pixel, the depth value of corresponding pixel points in the depth value of this pixel in the described present frame and the described reference frame is compared, comprising:

When the depth value of this pixel in the described present frame less than described reference frame in the depth value of corresponding pixel points, keep the depth value of this pixel in the described present frame;

When the depth value of this pixel in the described present frame greater than described reference frame in the depth value of corresponding pixel points, the depth value of this pixel in the described present frame is the depth value of corresponding pixel points in the described reference frame.

20. depth map as claimed in claim 11 is asked for device, it is characterized in that described output module carries out bilateral filtering to any depth value in the described depth map that calculates according to distribution of color and locus in its corresponding pixel points place window in present frame.

Images