CN102098526B - Depth map calculating method and device - Google Patents

Abstract

The invention discloses a depth map calculating method which comprises the steps of: inputting plane image; pretreating; selectively carrying out basic depth map calculation, basic depth map real-time calculation, depth map calculation based on color segmentation and rapid depth map calculation based on color segmentation on the plane image; carrying out depth map improvement based on reference frame and color distribution and space position improvement based on current frame on the depth map obtained by calculation; and outputting the depth map obtained by calculation, the improved depth map based on the reference frame, the improved color distribution and space position depth map based on the current frame and the plane image. The invention can be used for improving the quality of the depth map, meets the needs of depth map calculation with multiple time and quality requirements, and is wider in application scope. The invention also discloses a depth map calculating device.

Description

A kind of depth map acquiring method and device

Technical field

The present invention relates to technical field of computer vision, particularly a kind of depth map acquiring method and device.

Background technology

Existing various video medium generally is that the form with the plane exists, but above-mentioned video medium has lost the depth information of concrete scene when obtaining.

The three-dimensional video-frequency technology is as following Development of Multimedia Technology direction, and being a kind ofly can provide relief novel video technique.Compare with the single channel video, three-dimensional video-frequency generally has two video channels, and data volume will be far longer than the single channel video, so the efficient compression of stereoscopic video is particularly important.Particularly, three-dimensional video-frequency not only comprises the surface information about scene of conventional two-dimensional video, but also comprises the 3 D stereo information relevant with the scene particular location.Compare with the traditional two-dimensional video; Three-dimensional video-frequency be a kind of more effectively, more real expression way; The one-sidedness of two-dimensional video and the shortcoming of passivity have been overcome; Can satisfy visual perception's demand of people more fully, a lot of fields such as, advertisement media live in Interactive Free viewpoint video (FVV), virtual reality, 3DTV, 3D recreation, physical culture have a wide range of applications.

In stereovision technique, the extraction of depth map is then particularly important as the depth information that obtains concrete scene.Though existing various depth extraction method is a lot, all exist computation complexity high, calculate characteristics such as consuming time.Especially, for the depth map acquisition algorithm in the binocular stereo vision, be divided into two kinds of the overall situation and local algorithms.The complexity of above-mentioned algorithm generally is directly proportional with the geometric progression of the number of pixels of image or frame of video, and particularly Global Algorithm has related to the iterative process of global optimization, and complexity is higher.

Special, existing depth map method for distilling is not high for the real-time of the real-time image data processing on ordinary consumption type host computer.

Summary of the invention

The object of the invention is intended to solve at least one of above-mentioned technological deficiency, has proposed a kind of depth map acquiring method and device especially.

For achieving the above object, the embodiment of first aspect present invention has proposed a kind of depth map acquiring method, comprises the steps:

The input plane image, wherein, said plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Said plane picture is carried out preliminary treatment;

Judge whether pretreated plane picture need carry out that color of image is cut apart and whether need move the moving average algorithm, to said plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates and calculates fast based on the depth map that color is cut apart;

The depth map that calculates is carried out improving and based on the improvement of present frame color distribution and locus based on the depth map of reference frame; Wherein, the said depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation; And

Export the said depth map that calculates and improve after the depth map based on reference frame, depth map and the said plane picture after improving based on present frame color distribution and locus.

According to the depth map acquiring method of the embodiment of the invention, adopt and to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and through utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Depth map through based on bilateral filtering improves algorithm; Both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map; And the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

The embodiment of second aspect present invention has proposed a kind of depth map and has asked for device; Comprise the plane picture input module; Said plane picture input module is used for the input plane image, and wherein, said plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence; Pretreatment module, said pretreatment module are used for said plane picture is carried out preliminary treatment; The depth map computing module; Said depth map computing module is used to judge whether the pretreated plane picture of said pretreatment module need carry out that color of image is cut apart and whether need move the moving average algorithm, to said plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates and calculates fast based on the depth map that color is cut apart; Depth map improves module; Said depth map improves module and is used for the depth map that said depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame color distribution and locus; Wherein, the said depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation; And output module, said output module be used to export the said depth map that calculates and improve after the depth map based on reference frame, depth map and the said plane picture after improving based on present frame color distribution and locus.

Depth map according to the embodiment of the invention is asked for device, adopts to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and through utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Depth map through based on bilateral filtering improves algorithm; Both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map; And the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

Aspect that the present invention adds and advantage part in the following description provide, and part will become obviously from the following description, or recognize through practice of the present invention.

Description of drawings

Above-mentioned and/or additional aspect of the present invention and advantage are from obviously with easily understanding becoming the description of embodiment below in conjunction with accompanying drawing, wherein:

Fig. 1 is the FB(flow block) according to the depth map acquiring method of the embodiment of the invention;

Fig. 2 is the flow chart according to the input plane image of the embodiment of the invention;

Fig. 3 is for carrying out pretreated flow chart according to the embodiment of the invention to plane picture;

Fig. 4 is the flow chart according to the depth map calculating of the embodiment of the invention;

Fig. 5 is according to the improved flow chart of the depth map of the embodiment of the invention;

Fig. 6 is the schematic flow sheet according to the depth map output of the embodiment of the invention;

Fig. 7 is for asking for the structural representation of device according to the depth map of the embodiment of the invention; And

Fig. 8 is the structural representation according to the depth map computing module of the embodiment of the invention.

Embodiment

Describe embodiments of the invention below in detail, the example of said embodiment is shown in the drawings, and wherein identical from start to finish or similar label is represented identical or similar elements or the element with identical or similar functions.Be exemplary through the embodiment that is described with reference to the drawings below, only be used to explain the present invention, and can not be interpreted as limitation of the present invention.

Referring to figs. 1 to Fig. 6 the depth map acquiring method according to the embodiment of the invention is described below.

As shown in Figure 1, the depth map acquiring method according to the embodiment of the invention comprises the steps:

S101: input plane image;

As shown in Figure 2, plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence.Wherein, plane picture can change 3D video system, other two-way videos or image sequence output interface or other single channel videos or image sequence output interface from binocular solid collection and three-dimensional Play System, full-automatic 2D video.

For binocular solid collection and three-dimensional Play System, with taking two-way video that end collects or two-way picture frame plane picture as input.Change the 3D video system for full-automatic 2D video, directly with the plane picture of two frames of the front and back on the time shaft in the 2D video as input.

When the plane picture of input is single channel video or two-way video, need carry out decoding processing.Based on different video compression format different video encoding/decoding methods is arranged.Wherein, the decoding to video comprises that reading data flow converts suitable frame of video form again into from known video.If the frame of video form of decoding output and follow-up depth map are asked for interface and be not inconsistent, then need carry out the conversion operations of frame of video form.

When the plane picture of input is two-way video or two-way image sequence; Synchronization picture frame for dual input; With the picture frame of wherein one road video or image sequence as present frame; With the picture frame of another road video or image sequence frame as a reference, the order of two-path video can be set according to application request.Particularly,, the picture frame of a left side (or right) road video or image sequence is called present frame, the picture frame in same moment of right (or left side) road video or image sequence is called reference frame for binocular solid collection and three-dimensional Play System.

When the plane picture of input is single channel video or single channel image sequence; For example change the 3D video system for full-automatic 2D video; Be input as single channel 2D video, then the present frame of single channel video on time shaft be called present frame, the back frame on the time shaft is called the default reference frame.

S102: plane picture is carried out preliminary treatment;

As shown in Figure 3, when the plane picture of input when being single channel video or image sequence, single channel video or image sequence are carried out preliminary treatment comprise the steps:

S1021: judge whether to carry out key frame and detect, then carry out S1022 if desired; Otherwise carry out S103;

Change the 3D video for real-time 2D video, be input as two picture frames of the front and back on the time shaft in the 2D video, need do a key frame to back one frame this moment and judge, with the image of a frame after confirming and the similitude of current frame image.If the diversity of back one two field picture and current frame image surpasses the threshold value under certain special metric, think that then back one two field picture is a key frame, otherwise think that then back one two field picture is not key frame.

S1022: key frame detects;

At first, front and back two frames are carried out horizontal central line simultaneously cut apart with median vertical line and cut apart, thereby two frames are divided into four little image blocks respectively before and after making.In one embodiment of the invention, the length of four little image blocks after cutting apart and wide size are original image frame length and roomy little by 1/2.

Secondly, four little image blocks of front and back two frame correspondence positions are carried out the image block pixel value respectively subtract each other, the absolute value of getting difference to obtain new image block, is asked for pixel average and variance to four new image blocks as new pixel value.

Once more, the pixel average and the variance of four new images pieces are carried out threshold decision respectively.If it is a new key frame that individual number average that average and variance surpass threshold value in four new images pieces, is then judged the back relative present frame of a frame greater than preset value a.In one embodiment of the invention, preset value a can be 3.

In one embodiment of the invention, pixel average and variance threshold values can be the arbitrary value within [20,30].

At last, if one frame relative present frame in back is new key frame, then can not the reference frame of back one frame as the depth map calculating of present frame can be chosen the reference frame of the former frame of present frame as the depth map calculating of present frame this moment.

When the plane picture of input when being two-way video or two-way image sequence, two-way video or two-way image sequence are carried out preliminary treatment comprise the steps:

S1023: judge whether to carry out outer level line and proofread and correct, then carry out S1024 if desired, otherwise carry out S103;

For binocular solid collection and three-dimensional Play System, two-way video or the two-way image sequence of importing carried out preliminary treatment, need proofread and correct the outer level of two-path video frame or image sequence.Outer level line is proofreaied and correct and can be made certain delegation's pixel of sustained height in the corresponding reference frame of present frame pixel, has eliminated the deviation on the vertical direction.Thereby can reduce in the corresponding reference frame of this pixel with the search dimension of the corresponding pixel points in the time chart picture frame, reduce amount of calculation.

S1024: the outer level of image line is proofreaied and correct.

A certain pixel in the identical time chart picture frame in any pixel in a certain time chart picture frame in the road video in the two-way video and another road video is unique corresponding, thus the jobbie point in the space is gathered in common expression.

If but corresponding degree of depth the unknown of some pixels in a certain picture frame in one road video, this pixel possibly appear on the optional position of a certain straight line in one picture frame of back with the unique corresponding pixel points in the time chart picture frame in another road video so.Be on this straight line in the back picture frame all pixels all maybe with same object point in the corresponding space of this pixel.This straight line is the outer level of this pixel correspondence line.Generally speaking, outer level line is not a horizontal linear, but an oblique line.Proofread and correct the corresponding outer level line that can make in corresponding another picture frame of this pixel through outer level line and be transformed to horizontal direction; Thereby eliminated departing from the vertical direction; Can dwindle in corresponding another road video of this pixel search dimension, thereby reduce amount of calculation with the unique corresponding pixel points in the time chart picture frame.

Certainly it will be appreciated by persons skilled in the art that and to adopt other outer level line bearing calibrations the outer level line correction of two-way video or image sequence.

As can be seen from Figure 3, the plane picture of input also can be without preliminary treatment and the input of directly calculating as depth map.

S103: compute depth figure;

As shown in Figure 4; In an embodiment of the present invention; The calculating of depth map is found any pixel of present frame corresponding pixel in reference frame through the window matching algorithm, obtains the depth information of current frame pixel point then through the position relation of two corresponding pixels.

S1031: judge whether to carry out color of image and cut apart, if then carry out S1035; Otherwise carry out S1032;

To from the plane picture of the direct input of step S101 or judge whether to carry out color of image from the plane picture after the improvement of step S102 and cut apart.

S1032: judge whether to use the moving average algorithm, if then carry out S1034; Otherwise carry out S1033;

When judging that plane picture need not carry out color of image when cutting apart, and judges further whether plane picture need use the moving average algorithm.

S1033: calculate basic depth map;

When judging that plane picture need not use the moving average algorithm, calculate the basic depth map of plane picture, comprise the steps:

The first step is provided with window and search volume.

The size of window at first is set, and the window width that window is set is odd number N, in window width is the window of N, includes N*N pixel.With candidate's respective pixel on the region of search in any pixel in the present frame and the reference frame respectively as center pixel.Wherein, center pixel is positioned at the center of window.All pixels that comprise in the window become (comprising center pixel itself) the support pixel of center pixel.In one embodiment of the invention, window width N can be 15.Certainly it will be appreciated by persons skilled in the art that window width N can also be other odd number width.

When the ranks pixel of the pixel at four angles of each image or four peripheries is set up the window of pre-set dimension as center pixel, will run into this pixel around do not have the situation of pixel.At this moment, can adopt one of following dual mode that window is set.

1) do not exist the pixel value of pixel to be set to 0 around this pixel.

2) window size of directly setting up the corresponding window of above-mentioned pixel is set to 1 * 1 pixel (being about to this pixel itself as a window).

Certainly it will be appreciated by persons skilled in the art that above-mentioned additive method also falls into protection scope of the present invention when under the situation that does not have pixel on every side that adopts this pixel of additive method solution the problem of window being set.

Any pixel for present frame is provided with the hunting zone that the region of search is promptly set the candidate corresponding pixel points of this pixel in reference frame.The region of search is represented in the present frame relative position relation of candidate's respective pixel in the pixel and reference frame arbitrarily.In one embodiment of the invention, the region of search can be according to actual concrete condition of attending with flexible processing.

Eliminate later present frame and the reference frame of vertical missing to proofreading and correct through the outer level of image line, the region of search can be set to [d _Min, d _Max]=[0,32], wherein d is the difference of waiting arbitrarily in reference frame candidate corresponding pixel points and the present frame to ask between the horizontal coordinate of degree of depth pixel, d _MinThe coordinate of candidate's respective pixel waits to ask the pixel coordinate of the degree of depth consistent with present frame in=0 expression reference frame, and two pixels are in the same position in reference frame and the present frame respectively.

In second step, plane picture is carried out calculating based on the pixel coupling of window according to window;

Pixel coupling calculating based on window can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈N_p,{\stackrel{\‾}{q}}_d\∈N_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, p representes any pixel in the present frame;

Expression is candidate's corresponding pixel points in the reference frame of d apart from current frame pixel point p relative position; N _pExpression is all pixels that window was comprised (comprising pixel p itself) of center pixel with pixel p;

Expression is with candidate pixel point in the reference frame

For all pixels that window comprised of center pixel (comprise pixel Itself); P with Corresponding window width is N; Q representes in the present frame with p to be any pixel in the window of center pixel,

Represent in the reference frame with the candidate pixel point

Be any pixel in the window of center pixel,

Relative position in window is identical with q,

Remarked pixel point q with

The absolute value of margin of image element.

The 3rd step, obtain each pixel pixel parallax in the present frame according to window matching value and search volume, obtain the corresponding basic depth map of present frame.

At first obtain the pixel parallax of each pixel in the present frame.For any pixel p in the present frame, at the region of search [d _Min, d _Max] in choose the candidate pixel point in one of them d corresponding reference frame, the absolute difference and the summation of calculating all pixels in these two central pixel point place windows obtain E _d, to interval [d _Min, d _Max] interior all corresponding E of any d _d, select all E at last _dThe corresponding parallax value of minimum value as the parallax value d of current pixel point in the present frame _s, simultaneously with parallax value d _sCandidate pixel point in the corresponding reference frame As the final respective pixel of pixel p, wherein,

The employing said method can obtain the time difference of all pixels of present frame, according to the parallax d of all pixels of present frame that obtain _{(i, j)}=d _s, d wherein _{(i, j)}Be corresponding pixel points (i, parallax j).The depth map corresponding according to the present frame that obtains,

$Z_{(i,j)}=\frac{d_{(i,j)}-d_{\min }}{(d_{\max }-d_{\min })}*255$

Z wherein _{(i, j)}Be corresponding pixel points (i, degree of depth j).

S1034: calculate basic real-time deep figure;

When judging that plane picture need use the moving average algorithm, calculate the basic real-time deep figure of plane picture, comprise the steps:

For the region of search [d _Min, d _Max] in each parallax d, for each pixel in the present frame, all need calculate in its place window the pixel value difference that candidate pixel point in each pixel and the reference frame belongs to the respective pixel in the window.

When being input as long and wide the be present frame of W*H and reference frame; Window size is N*N, and search length is L, wherein for each pixel in the present frame and the candidate pixel point in the reference frame in the hunting zone; Every calculating once; Amount of calculation is N*N time a subtraction, N*N-1 time addition, and final amount of calculation is W*H*L* (N*N) * (N*N-1).This amount of calculation is directly proportional with the biquadratic of window width N, because N is variable, when N increased, then amount of calculation will be geometric progression increased.

The first step; Initialization calculating that line slip on line direction is average and the average recursive calculation of line slip; And the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain any pixel corresponding real-time window matching value Q of said present frame _{(i, j)}

At first calculate in the initialization of the up moving average of line direction.First pixel (i, 0) to each row in the present frame is interior to pixel value difference absolute value C in window line width scope [0, N] _{(i, j)}Carry out accumulation calculating.Wherein,

j∈[0，N]，A _(i，0)＝A _(i，0)+C _(i，j)

Wherein,

Any i of delegation, A are got in expression _{(i, 0)}The C of first pixel (i, 0) in window line width scope that representes any i of delegation _{(i, j)}Accumulation calculating result, C _{(i, j)}Support pixel in the expression window line width scope (i, j) with reference frame in the pixel value difference absolute value of respective pixel, N representes window width.

Then line slip is averaged recursive calculation.Particularly, adopt following formula that any capable i is carried out the average recursive calculation of line slip:

$A_{(i,j)}=A_{(i,j-1)}+C_{(i,j+\frac{N}{2})}-C_{(i,j-\frac{N}{2}-1)},$

Wherein, for the center pixel of the capable j of i row (i, j), the sliding average A on its corresponding row direction _{(i, j)}Line slip mean value A by previous pixel _{(i, j-1)}Add the current window right endpoint

Result of calculation

With, deduct the left end point of previous window again

Result of calculation

Recursive calculation obtains.

Again, the initialization of the above-listed moving average of column direction is calculated.Particularly, to first pixel of any row j in the present frame (0, j) adopt following formula in the window column wide region, line slip mean value to be carried out accumulation calculating:

i∈[0，N]，Q _(0,j)＝Q _(0,j)+A _(i，j)，

Wherein

Represent to appoint and get a j, Q _{(0, j)}Represent that (0, j) A is calculated in the accumulation to line slip mean value to any first pixel of row j in the window column wide region _{(i, j)}(N representes window width to pixel for i, line slip mean value j) in the expression present frame.

Be listed as the average recursive calculation of slip at last.Particularly, adopt following formula to carry out column direction row moving average recursive calculation for any row j:

$Q_{(i,j)}=Q_{(i-1,j)}+A_{(i+\frac{N}{2},j)}-A_{(i-\frac{N}{2}-1,j)},$

Q wherein _{(i, j)}Represent any one row j the capable pixel of i (i, j) the recursive calculation result of row sliding average in the window column wide region is by the row sliding average Q of a last pixel _{(i-1, j)}With the current window lower extreme point

Line slip mean value with, deduct a window upper extreme point again

Line slip mean value recursive calculation obtain.

Second the step, any pixel in present frame (i, j) corresponding to each relative position among the d of particular search space to a plurality of said real-time window matching value Q should be arranged _{(i, j)}, for the region of search [d _Min, d _Max] in a plurality of real-time window matching value Q _{(i, j)}In choose the corresponding parallax value of minimum value as current pixel point in the present frame (i, parallax value d j) _S1Thereby, can obtain the depth value of respective pixel.

$d_{s1}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{Q_{(i,j)}{|}_d\right\}.$

In the 3rd step, ask for the real-time pixel parallax of each pixel of present frame, thereby can obtain the corresponding basic real-time deep figure of present frame.

S1035: judge whether to use the moving average algorithm, if then carry out S1037; Otherwise carry out S1036;

Because in the ordinary course of things, the corresponding same object of the image-region of same color, depth value is same or similar; And the corresponding different objects of the image-region of different colours, the degree of depth is inconsistent.Therefore can utilize color of image to cut apart the classified information that obtains; Based on the window calculation depth map time; Increase belongs to the shared weight of of a sort other pixels with the window center pixel, reduces not belong to the shared weight of of a sort other pixels with the window center pixel.

S1036: calculate depth map based on image segmentation.

The first step, (i j), is connected to 8 limits with each pixel (i) of importing present frame and its 8 pixels (j) on every side to set up graph model V.Wherein, two end points on every limit be two pixels (i, j), the weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) |, obtain thus a series of limit edge (i, j).

Second step under initial situation, was one type with each pixel (i), and to set this moment type initial merger threshold value be threshold (i)=c, was sorted according to weights from small to large in every limit in the above-mentioned graph model then.

In the 3rd step, (i j) belongs to class and carries out the operation of merger in twos, and wherein, merger comprises the steps: to pairing two pixels in limit after the ordering

As a limit edge (i; J) (i is j) than two end points on this limit (i, j) the merger threshold value threshold (i) at place type for weight w; Threshold (j) hour; Then with this limit edge (i, two types of merger at two pixels place j) are one type, upgrade new type the merger threshold value that obtains simultaneously to be:

$\mathrm{threshold}(i,j)=w(i,j)+\frac{c}{\mathrm{num}(i,j)},$

Wherein, num (i, j) remarked pixel point i and the j number of pixels addition of type of place originally.

The 4th step; Class to the number of pixels that obtains after the merger is fewer is forced merger operation, travel through all limit edge (i, j); If exist two pixels on any limit not at same time-like; And the number of pixels of the class at two pixel places all is less than a specified value minSize, so just two classes at two pixels place carried out the merger operation, and the color that finally obtains after each pixel classification is cut apart figure.

The 5th goes on foot, and the plane picture after according to window color being cut apart carries out calculating based on the pixel coupling of window;

Pixel coupling calculating based on color is cut apart can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈{\mathrm{NS}}_p,{\stackrel{\‾}{q}}_d\∈{\mathrm{NS}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d)+\mathrm{\λ}\×{\mathrm{\Σ}}_{q\∈{N\stackrel{\‾}{S}}_p,{\stackrel{\‾}{q}}_d\∈{N\stackrel{\‾}{S}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, NS _pExpression with pixel p be center pixel window was comprised and pixel p belongs to the pixel (comprising pixel p itself) of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that belongs to same color block (comprises pixel

Itself); Expression with pixel p be center pixel window was comprised and pixel p does not belong to the pixel of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that does not belong to same color block; λ representes not belong to center pixel the accumulation calculating weight of the area pixel of same block.In one embodiment of the invention, λ can be 0.01.

The 6th step, obtain each pixel pixel parallax in the present frame according to window matching value and search volume, obtain the corresponding basic depth map of present frame.

$d_s=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{E_d\right\}.$

S1037: calculate real-time deep figure based on image segmentation;

The first step is calculated based on the initialization of the line slip of the carve information of image segmentation.

For each row of present frame, make the line slip of all blocks on average be initially zero, i.e. T _S=0, wherein s is the block under the pixel.Then each row is carried out calculating based on the average initialization of the line slip of carve information, promptly in window line width scope to pixel value difference according to the place block, carry out accumulation calculating.

j∈[0，N]，T _S(i，j)＝T _S(i，j)+C _(i，j)

In the following formula, T _{S (i, j)}(i, j) the affiliated sliding average of block on line direction is by belonging to block S (i, C j) for remarked pixel _{(i, j)}Adding up obtains, C _{(i, j)}Pixel in the expression present frame (i, j) with reference frame in the pixel value difference absolute value of respective pixel.Calculate through following formula, can obtain any delegation based on the average initial calculation result of the line slip of carve information.

Second step is based on the average recursive calculation of line slip of the carve information of image segmentation

Adopt following formula that any delegation is carried out based on the average recursive calculation of the line slip of carve information:

$T_{S(i,j+\frac{N}{2})}=T_{S(i,j+\frac{N}{2})}+C_{S(i,j+\frac{N}{2})}$

$T_{S(i,j-\frac{N}{2}-1)}=T_{S(i,j-\frac{N}{2}-1)}-C_{S(i,j-\frac{N}{2}-1)},$

In following formula, (i, j), the result of calculation of the right endpoint of its corresponding window does the center pixel that is listed as for the capable j of i

The result of calculation of the left end point of the previous window that it is corresponding does

Respectively according to the block at its place Carry out addition and subtraction, upgrade block T _SThe average recursive calculation result of line slip.Finally for each center pixel (i, j), obtain based on the line slip average computation result of carve information do

$A_{(i,j)}^r=T_{S(i,j)}.$

In the 3rd step, calculate based on the initialization of the row moving average of the carve information of image segmentation

For each row of present frame, make the row moving average of all blocks be initially zero, i.e. G _S=0, wherein s is the block under the pixel, and G is a row moving average result of calculation.Then each row is carried out calculating based on the row moving average initialization of carve information, promptly in the window column wide region to based on the line slip average computation result of carve information according to the place block, carry out accumulation calculating.

i∈[0，N]， $G_{S(i,j)}=G_{S(i,j)}+A_{(i,j)}^r$

In following formula, G _{S (i, j)}Remarked pixel (i, j) the window moving average on the affiliated block column direction,

Pixel (i, j) the line slip average computation result on line direction in the expression present frame based on carve information.Calculate through following formula, can obtain for the initial calculation result of any rows of directions based on the row moving average of carve information.

The 4th step is based on the row moving average recursive calculation of the carve information of image segmentation.

Adopt following formula to carry out row moving average recursive calculation to any row based on carve information:

$G_{S(i+\frac{N}{2},j)}=G_{S(i+\frac{N}{2},j)}+A_{S(i+\frac{N}{2},j)}^r$

$G_{S(i-\frac{N}{2}-1,j)}=G_{S(i-\frac{N}{2}-1,j)}-A_{S(i-\frac{N}{2}-1,j)}^r,$

In following formula, (i, j), the line slip average computation result of its corresponding window lower extreme point does to be listed as the capable center pixel of i for j

The line slip average computation result of the upper extreme point of a window does on its respective column direction

Respectively according to the block at its place

Carry out addition and subtraction, recurrence is upgraded block G _SRow moving average result.(i, j) the window accumulation result of calculation based on carve information is G finally to obtain any pixel _{S (i, j)}

In the 5th step, ask for parallax and depth map based on the carve information of image segmentation.

Through obtaining any pixel (i, basic window accumulation result of calculation Q j) _{(i, j)}With window accumulation result of calculation G based on the color carve information _{S (i, j)}, utilize The above results to adopt the basic depth map of following improvement to ask for algorithm in real time:

M(i，j)| _d＝λ(Q _(i，j)| _d-G _S(i，j)| _d)+G _S(i，j)| _d

In following formula | _dExpression is for d specific in the hunting zone, and (i, j) the final window of expression is accumulated result of calculation, Q to M _{(i, j)}-G _{S (i, j)}When expression window accumulation is calculated and center pixel (i j) does not belong to the accumulation calculating section of same block, G _{S (i, j)}(i j) belongs to the accumulation calculating section of same block, and λ representes that the accumulation that does not belong to the area pixel of same block with center pixel calculates weight for expression and center pixel.In one embodiment of the invention, λ can be 0.01.(i, the corresponding parallax value of minimum value j) is the parallax d of current frame pixel finally to get M _S2

$d_{s2}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{M(i,j){|}_d\right\}.$

In one embodiment of the invention; Based on the calculating of the algorithm of four kinds of different characteristics to the depth map of present frame; Be that the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculate and calculate fast based on the depth map that color is cut apart, four kinds of algorithms are separate.Each algorithm can be as independently depth map calculating.Certainly the independence or the comprehensive use that it will be appreciated by persons skilled in the art that above-mentioned four kinds of algorithms all belong to protection scope of the present invention.

The real-time calculating of the basic depth map of the embodiment of the invention improves on computation complexity the calculating of basic depth map, has reached the real-time requirement.The depth map calculating of cutting apart based on color of the embodiment of the invention further improves on the calculating accuracy the calculating of basic depth map; The quick calculating of cutting apart based on color of depth map of the embodiment of the invention all improves on calculating accuracy and real-time, has obtained the higher depth map of quality.

S104: improve depth map;

Because the quick calculating of the calculating of the depth map that utilizes the real-time calculating of the calculating of above-mentioned basic depth map, basic depth map, cuts apart based on color and the depth map cut apart based on color is extracted the depth map of present frame; There is the pixel matching error; Perhaps there is not corresponding matched pixel point in some pixel of present frame in reference frame, therefore need the depth map that calculate among the step S103 be improved.

As shown in Figure 5, the depth map that calculates among the step S103 is carried out based on the improvement of the depth map of reference frame with based on the improvement of present frame color distribution and locus.Wherein, the depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation.

S1041: judge whether to carry out improvement based on the reference frame depth map; If then carry out S1042; Otherwise carry out S1044;

S1042: calculate the reference frame depth map;

When judgement need be carried out the improvement based on the reference frame depth map, the depth map that calculates is asked for, searches, checked and improves.Particularly, comprise the steps:

The first step is asked for the depth map of reference frame according to the depth map that calculates.Only need depth map calculate interface will before reference frame as present present frame, with before present frame as present reference frame, the corresponding depth map of reference frame before can calculating.

Second step is by certain the pixel P in the present frame _C(i, degree of depth Z j) _C(i j) can find pixel P corresponding in the reference frame _R(i, j), then the corresponding degree of depth of this pixel in the reference frame depth map is Z _R(i, j).

S1043: based on the improvement of reference frame depth map;

The first step, the inspection depth map.If Z _C(i, j) and Z _R(i, j) difference of two depth values is less than or equal to predetermined threshold t, then representes two pixel P _C(i, j) and P _R(i, j) corresponding same object point can judge that then depth calculation is accurate; If Z _C(i, j) and Z _R(i when j) difference of two depth values is greater than predetermined threshold t, representes that then two pixels are not corresponding, can assert that present frame is at pixel P _C(i, the depth calculation mistake of j) locating.

Predetermined threshold t can reach needs as the case may be and set.In one embodiment of the invention, predetermined threshold t can be 1.

In second step, depth map improves.If the depth calculation mistake then can be got Z _C(i, j) and Z _R(i, j) between the two smaller is current frame pixel point P _C(i, degree of depth j), thus reach the purpose of improving the depth map quality.

S1044: judge whether to carry out improvement based on present frame color distribution and locus; If then carry out S1045; Otherwise carry out S105;

S1045: based on the improvement of reference frame depth map.

Refer to that based on the present frame color distribution and the improvement of locus the ID value can carry out the bilateral filtering improvement by color distribution in the window area of the pixel value of its corresponding present frame place and locus.

For the pixel Z (p) of any the expression degree of depth among the ID figure, its respective pixel in present frame is p, is center pixel with the pixel p in the present frame, and window width is that all pixels in the window of N are q.The pixel value difference Gauss weight factor w of all pixel q and center pixel p in the calculation window ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q), and the depth value Z (q) of two weight factors and pixel q multiplied each other, obtain the depth value Z (q) of cum rights repeated factor.Pixel value difference Gauss weight factor w ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q) be respectively:

$w_1(V_p,V_q)=e^{-\frac{{(V_p-V_q)}^2}{2{\mathrm{\σ}}_1^2}}$

$w_2(S_p,S_q)=e^{-\frac{{(p\left(x\right)-q\left(x\right))}^2}{2{\mathrm{\σ}}_2^2}}*e^{-\frac{{(p\left(y\right)-q\left(y\right))}^2}{2{\mathrm{\σ}}_2^2}},$

Wherein,

is the gaussian filtering function; V remarked pixel value, p (x), p (y); Q (x), q (y) is the horizontal ordinate of remarked pixel p, q respectively.σ ₁, σ ₂Expression gaussian filtering variance.Wherein, σ ₁, σ ₂Can reach as the case may be needs and is provided with.In one embodiment of the invention, σ ₁=15, σ ₂=5.

Depth value Z (q) to the cum rights repeated factor adds up; And weight is carried out normalization handle, the degree of depth that obtains is the degree of depth

of improving the back pixel p

$\stackrel{\‾}{Z}\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)Z\left(q\right)}{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)}.$

Can know that from following formula p is a pixel of present frame, q is for being the center with current pixel point p, and window width is any pixel in the window of N, N _pThe expression window size.In one embodiment of the invention, window width N is 7, and then window size is 7*7.Z (q) is the depth value of pixel q in the present frame,

Be the depth value after the pixel p improvement in the present frame, w ₁(V _p, V _q) be Gauss's weight factor of the pixel value difference of pixel p and pixel q, w ₂(S _p, S _q) be Gauss's weight factor of the pixel space distance of pixel p and pixel q.

Noise or erroneous matching when the depth map after adopting above-mentioned depth map to improve one's methods to improve not only can filter out depth calculation, and keep the boundary information of present frame ID.

The depth map based on reference frame that the embodiment of the invention provides improves and can use separately or use simultaneously based on the improvement of present frame color distribution and locus.

S105: output depth map.

As shown in Figure 6, exporting after the depth map after the improvement that will export by the depth map that calculates and the step S104 of step S103 output, the improvement based on reference frame based on the depth map of present frame color distribution and locus and the plane picture of original input.

According to the depth map acquiring method of the embodiment of the invention, adopt and to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and through utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Depth map through based on bilateral filtering improves algorithm; Both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map; And the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

As shown in Figure 7; The embodiment of the invention provides a kind of depth map to ask for device 700; Comprise plane picture input module 710, wherein, plane picture input module 710 is used for the input plane image; Wherein, plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence; Pretreatment module 720, wherein pretreatment module 720 is used for plane picture is carried out preliminary treatment; Depth map computing module 730; Wherein said depth map computing module 730 depth map computing modules are used to judge whether the pretreated plane picture of said pretreatment module need carry out that color of image is cut apart and whether need move the moving average algorithm, to said plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates fast or calculates based on the depth map that color is cut apart; Depth map improves module 740; Wherein said depth map improves module 740 and is used for the depth map that said depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame color distribution and locus; Wherein, the said depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation; With output module 750, wherein said output module 750 be used to export the said depth map that calculates and improve after the depth map based on reference frame, depth map and the said plane picture after improving based on present frame color distribution and locus.

In one embodiment of the invention, plane picture can change 3D video system, other two-way videos or image sequence output interface or other single channel videos or image sequence output interface from binocular solid collection and three-dimensional Play System, full-automatic 2D video.For binocular solid collection and three-dimensional Play System, with taking two-way video that end collects or two-way picture frame plane picture as input.Change the 3D video system for full-automatic 2D video, directly with the plane picture of two frames of the front and back on the time shaft in the 2D video as input.

When the plane picture of plane picture input module 710 inputs is single channel video or two-way video, need carry out decoding processing.Based on different video compression format different video encoding/decoding methods is arranged.Wherein, the decoding to video comprises that reading data flow converts suitable frame of video form again into from known video.If the frame of video form of decoding output and follow-up depth map are asked for interface and be not inconsistent, then need carry out the conversion operations of frame of video form.

When the plane picture of plane picture input module 710 inputs is two-way video or two-way image sequence; Synchronization picture frame for dual input; With the picture frame of wherein one road video or image sequence as present frame; With the picture frame of another road video or image sequence frame as a reference, the order of two-path video can be set according to application request.Particularly,, the picture frame of a left side (or right) road video or image sequence is called present frame, the picture frame in same moment of right (or left side) road video or image sequence is called reference frame for binocular solid collection and three-dimensional Play System.

When the plane picture of plane picture input module 710 inputs is single channel video or single channel image sequence; For example change the 3D video system for full-automatic 2D video; Be input as single channel 2D video; Then the present frame of single channel video on time shaft is called present frame, the back frame on the time shaft is called the default reference frame.

When the plane picture of plane picture input module 710 inputs was single channel video or image sequence, pretreatment module 720 judged whether to carry out key frame and detects.Change the 3D video for real-time 2D video, be input as two picture frames of the front and back on the time shaft in the 2D video, need do a key frame to back one frame this moment and judge, with the image of a frame after confirming and the similitude of current frame image.If the diversity of back one two field picture and current frame image surpasses the threshold value under certain special metric, think that then back one two field picture is a key frame, otherwise think that then back one two field picture is not key frame.

When judgement need be carried out key frame when detecting, at first 720 pairs of front and back two frames of pretreatment module carry out horizontal central line simultaneously and cut apart with median vertical line and cut apart, thereby two frames are divided into four little image blocks respectively before and after making.In one embodiment of the invention, the length of four little image blocks after cutting apart and wide size are original image frame length and roomy little by 1/2.Secondly, pretreatment module 720 is carried out the image block pixel value respectively with four little image blocks of front and back two frame correspondence positions and is subtracted each other, and the absolute value of getting difference to obtain new image block, is asked for pixel average and variance to four new image blocks as new pixel value.Once more, the pixel average of 720 pairs of four new images pieces of pretreatment module and variance are carried out threshold decision respectively.If it is a new key frame that individual number average that average and variance surpass threshold value in four new images pieces, is then judged the back relative present frame of a frame greater than preset value a.In one embodiment of the invention, preset value a can be 3.In one embodiment of the invention, pixel average and variance threshold values can be the arbitrary value within [20,30].At last, if one frame relative present frame in back is new key frame, then can not be with the reference frame of back one frame as the depth map calculating of present frame, this moment, pretreatment module 720 can be chosen the reference frame of the former frame of present frame as the depth map calculating of present frame.

When the plane picture of plane picture input module 710 inputs was two-way video or two-way image sequence, pretreatment module 720 judged whether to carry out outer level line and proofreaies and correct.For binocular solid collection and three-dimensional Play System, two-way video or the two-way image sequence of importing carried out preliminary treatment, need proofread and correct the outer level of two-path video frame or image sequence.Outer level line is proofreaied and correct and can be made certain delegation's pixel of sustained height in the corresponding reference frame of present frame pixel, has eliminated the deviation on the vertical direction.Thereby can reduce in the corresponding reference frame of this pixel with the search dimension of the corresponding pixel points in the time chart picture frame, reduce amount of calculation.

Level line timing outside judgement need be carried out; Pretreatment module 720 is unique corresponding with a certain pixel in the identical time chart picture frame in any pixel in a certain time chart picture frame in the road video in the two-way video and another road video, thereby the jobbie point in the space is gathered in common expression.If but corresponding degree of depth the unknown of some pixels in a certain picture frame in one road video, this pixel possibly appear on the optional position of a certain straight line in one picture frame of back with the unique corresponding pixel points in the time chart picture frame in another road video so.Be on this straight line in the back picture frame all pixels all maybe with same object point in the corresponding space of this pixel.This straight line is the outer level of this pixel correspondence line.Generally speaking, outer level line is not a horizontal linear, but an oblique line.Proofread and correct the corresponding outer level line that can make in corresponding another picture frame of this pixel through outer level line and be transformed to horizontal direction; Thereby eliminated departing from the vertical direction; Can dwindle in corresponding another road video of this pixel search dimension, thereby reduce amount of calculation with the unique corresponding pixel points in the time chart picture frame.Certainly it will be appreciated by persons skilled in the art that and to adopt other outer level line bearing calibrations the outer level line correction of two-way video or image sequence.

In one embodiment of the invention, the plane picture of plane picture input module 710 input also can be without the preliminary treatment of pretreatment module 720 and directly as the input of depth map computing module 730.

As shown in Figure 8, depth map computing module 730 comprises the real-time computing unit of judging unit 731, basic depth map 732, basic depth map computing unit 733, the depth map computing unit of cutting apart based on color 734 and the quick computing unit of cutting apart based on color 735 of depth map.

At first, 731 pairs of plane pictures from plane picture input module 710 or pretreatment module 720 of judging unit judge whether to carry out color of image and cut apart.When judging that plane picture need not carry out color of image when cutting apart, and judges further whether plane picture need use the moving average algorithm.

When judging unit 731 judged that plane picture need not use the moving average algorithm, basic depth map computing unit 733 calculated the basic depth map of plane picture.

At first basic depth map computing unit 733 is provided with the size of window, and the window width that window is set is odd number N, in window width is the window of N, includes N*N pixel.With candidate's respective pixel on the region of search in any pixel in the present frame and the reference frame respectively as center pixel.Wherein, center pixel is positioned at the center of window.All pixels that comprise in the window become (comprising center pixel itself) the support pixel of center pixel.In one embodiment of the invention, window width N can be 15.Certainly it will be appreciated by persons skilled in the art that window width N can also be other odd number width.

Basic depth map computing unit 733 will run into the situation that does not have pixel on every side of this pixel when the ranks pixel of the pixel at four angles of each image or four peripheries is set up the window of pre-set dimension as center pixel.At this moment, can adopt one of following dual mode that window is set.

1) do not exist the pixel value of pixel to be set to 0 around this pixel.

2) window size of directly setting up the corresponding window of above-mentioned pixel is set to 1 * 1 pixel (being about to this pixel itself as a window).

Certainly it will be appreciated by persons skilled in the art that above-mentioned additive method also falls into protection scope of the present invention when under the situation that does not have pixel on every side that adopts this pixel of additive method solution the problem of window being set.

For any pixel of present frame, basic depth map computing unit 733 is provided with the hunting zone that the region of search is promptly set the candidate corresponding pixel points of this pixel in reference frame.The region of search is represented in the present frame relative position relation of candidate's respective pixel in the pixel and reference frame arbitrarily.In one embodiment of the invention, the region of search can be according to actual concrete condition of attending with flexible processing.

The outer level of basic depth map computing unit 733 pairs of processes image line is proofreaied and correct and is eliminated later present frame and the reference frame of vertical missing, and the region of search can be set to [d _Min, d _Max]=[0,32], wherein d is the difference of waiting arbitrarily in reference frame candidate corresponding pixel points and the present frame to ask between the horizontal coordinate of degree of depth pixel, d _MinThe coordinate of candidate's respective pixel waits to ask the pixel coordinate of the degree of depth consistent with present frame in=0 expression reference frame, and two pixels are in the same position in reference frame and the present frame respectively.

Basic depth map computing unit 733 calculates based on the pixel coupling of window and can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈N_p,{\stackrel{\‾}{q}}_d\∈N_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, p representes any pixel in the present frame;

Expression is candidate's corresponding pixel points in the reference frame of d apart from current frame pixel point p relative position; N _pExpression is all pixels that window was comprised (comprising pixel p itself) of center pixel with pixel p; Expression is with candidate pixel point in the reference frame

For all pixels that window comprised of center pixel (comprise pixel Itself); P with

Corresponding window width is N; Q representes in the present frame with p to be any pixel in the window of center pixel,

Represent in the reference frame with the candidate pixel point

Be any pixel in the window of center pixel,

Relative position in window is identical with q,

Remarked pixel point q with

The absolute value of margin of image element.

Basic depth map computing unit 733 at first obtains the pixel parallax of each pixel in the present frame.For any pixel p in the present frame, at the region of search [d _Min, d _Max] in choose the candidate pixel point in one of them d corresponding reference frame, the absolute difference and the summation of calculating all pixels in these two central pixel point place windows obtain E _d, to interval [d _Min, d _Max] interior all corresponding E of any d _d, select all E at last _dThe corresponding parallax value of minimum value as the parallax value d of current pixel point in the present frame _s, simultaneously with parallax value d _sCandidate pixel point in the corresponding reference frame

As the final respective pixel of pixel p, wherein,

The employing said method can obtain the time difference of all pixels of present frame, according to the parallax d of all pixels of present frame that obtain _{(i, j)}=d _s, d wherein _{(i, j)}Be corresponding pixel points (i, parallax j).The depth map corresponding according to the present frame that obtains,

$Z_{(i,j)}=\frac{d_{(i,j)}-d_{\min }}{(d_{\max }-d_{\min })}*255$

Z wherein _{(i, j)}Be corresponding pixel points (i, degree of depth j).

When judging unit 731 judged that plane picture need use the moving average algorithm, the real-time computing unit 732 of basic depth map calculated the basic real-time deep figure of plane picture.For the region of search [d _Min, d _Max] in each parallax d, for each pixel in the present frame, all need calculate in its place window the pixel value difference that candidate pixel point in each pixel and the reference frame belongs to the respective pixel in the window.

When being input as long and wide the be present frame of W*H and reference frame; Window size is N*N, and search length is L, wherein for each pixel in the present frame and the candidate pixel point in the reference frame in the hunting zone; Every calculating once; Amount of calculation is N*N time a subtraction, N*N-1 time addition, and final amount of calculation is W*H*L* (N*N) * (N*N-1).This amount of calculation is directly proportional with the biquadratic of window width N, because N is variable, when N increased, then amount of calculation will be geometric progression increased.

Basic average initialization calculating and the average recursive calculation of line slip of the line slip of the real-time computing unit 732 of depth map on line direction; And the initialization calculating of the row moving average on column direction and row moving average recursive calculation, obtain any pixel corresponding real-time window matching value Q of said present frame _{(i, j)}

At first calculate in the initialization of the up moving average of line direction.First pixel (i, 0) to each row in the present frame is interior to pixel value difference absolute value C in window line width scope [0, N] _{(i, j)}Carry out accumulation calculating.Wherein,

j∈[0，N]，A _(i，0)＝A _(i，0)+C _(i，j)

Wherein, Any i of delegation, A are got in expression _{(i, 0)}The C of first pixel (i, 0) in window line width scope that representes any i of delegation _{(i, j)}Accumulation calculating result, C _{(i, j)}Support pixel in the expression window line width scope (i, j) with reference frame in the pixel value difference absolute value of respective pixel, N representes window width.

The basic real-time computing unit 732 of depth map averages recursive calculation to line slip then.Particularly, adopt following formula that any capable i is carried out the average recursive calculation of line slip:

$A_{(i,j)}=A_{(i,j-1)}+C_{(i,j+\frac{N}{2})}-C_{(i,j-\frac{N}{2}-1)},$

Wherein, for the center pixel of the capable j of i row (i, j), the sliding average A on its corresponding row direction _{(i, j)}Line slip mean value A by previous pixel _{(i, j-1)}Add the current window right endpoint

Result of calculation

With, deduct the left end point of previous window again

Result of calculation

Recursive calculation obtains.

Again, the initialization of basic 732 pairs of above-listed moving averages of column direction of the real-time computing unit of depth map is calculated.Particularly, to first pixel of any row j in the present frame (0, j) adopt following formula in the window column wide region, line slip mean value to be carried out accumulation calculating:

i∈[0，N]，Q _(0，j)＝Q _(0，j)+A _(i，j)，

Wherein Represent to appoint and get a j, Q _{(0, j)}Represent that (0, j) A is calculated in the accumulation to line slip mean value to any first pixel of row j in the window column wide region _{(i, j)}(N representes window width to pixel for i, line slip mean value j) in the expression present frame.

The last basic real-time computing unit 732 of depth map is listed as the average recursive calculation of slip.Particularly, adopt following formula to carry out column direction row moving average recursive calculation for any row j:

$Q_{(i,j)}=Q_{(i-1,j)}+A_{(i+\frac{N}{2},j)}-A_{(i-\frac{N}{2}-1,j)},$

Q wherein _{(i, j)}Represent any one row j the capable pixel of i (i, j) the recursive calculation result of row sliding average in the window column wide region is by the row sliding average Q of a last pixel _{(i-1, j)}With the current window lower extreme point

Line slip mean value with, deduct a window upper extreme point again

Line slip mean value recursive calculation obtain.

Any pixel in present frame of the basic real-time computing unit 732 of depth map (i, j) corresponding to each relative position among the d of particular search space to a plurality of said real-time window matching value Q should be arranged _{(i, j)}, for the region of search [d _Min, d _Max] in a plurality of real-time window matching value Q _{(i, j)}In choose the corresponding parallax value of minimum value as current pixel point in the present frame (i, parallax value d j) _S1Thereby, can obtain the depth value of respective pixel.

$d_{s1}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{Q_{(i,j)}{|}_d\right\}.$

The basic real-time computing unit 732 of depth map is asked for the real-time pixel parallax of each pixel of present frame, thereby can obtain the corresponding basic real-time deep figure of present frame.

Because in the ordinary course of things, the corresponding same object of the image-region of same color, depth value is same or similar; And the corresponding different objects of the image-region of different colours, the degree of depth is inconsistent.Therefore can utilize color of image to cut apart the classified information that obtains; Based on the window calculation depth map time; Increase belongs to the shared weight of of a sort other pixels with the window center pixel, reduces not belong to the shared weight of of a sort other pixels with the window center pixel.

The depth map that the depth map computing unit of cutting apart based on color 734 calculates based on image segmentation.

At first, the depth map computing unit of cutting apart based on color 734 is set up graph model V, and (i j), is connected to 8 limits with each pixel (i) of importing present frame and its 8 pixels (j) on every side.Wherein, two end points on every limit be two pixels (i, j), the weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (i) |, obtain thus a series of limit edge (i, j).

Under initial situation; The depth map computing unit of cutting apart based on color 734 is one type with each pixel (i); And setting the initial merger threshold value of class this moment is threshold (i)=c, is sorted according to weights from small to large in every limit in the above-mentioned graph model then.

Secondly, and pairing two pixels in limit after 734 pairs of orderings of depth map computing unit of cutting apart based on color (i, j) the place class is carried out the operation of merger in twos, and wherein, merger comprises the steps:

As a limit edge (i; J) (i is j) than two end points on this limit (i, j) the merger threshold value threshold (i) at place type for weight w; Threshold (j) hour; Then with this limit edge (i, two types of merger at two pixels place j) are one type, upgrade new type the merger threshold value that obtains simultaneously to be:

$\mathrm{threshold}(i,j)=w(i,j)+\frac{c}{\mathrm{num}(i,j)},$

Wherein, num (i, j) remarked pixel point i and the j number of pixels addition of type of place originally.

Once more; The fewer class of the number of pixels that obtains after 734 pairs of merger of depth map computing unit of cutting apart based on color is forced the merger operation, travel through all limit edge (i, j); If exist two pixels on any limit not at same time-like; And the number of pixels of the class at two pixel places all is less than a specified value minSize, so just two classes at two pixels place carried out the merger operation, and the color that finally obtains after each pixel classification is cut apart figure.

Then, the plane picture after the depth map computing unit of cutting apart based on color 734 is cut apart color according to window carries out calculating based on the pixel coupling of window;

Pixel coupling calculating based on color is cut apart can adopt following formula to calculate,

$E_d={\mathrm{\Σ}}_{q\∈{\mathrm{NS}}_p,{\stackrel{\‾}{q}}_d\∈{\mathrm{NS}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d)+\mathrm{\λ}\×{\mathrm{\Σ}}_{q\∈{N\stackrel{\‾}{S}}_p,{\stackrel{\‾}{q}}_d\∈{N\stackrel{\‾}{S}}_{{\stackrel{\‾}{p}}_d}}e(q,{\stackrel{\‾}{q}}_d),$

Wherein, NS _pExpression with pixel p be center pixel window was comprised and pixel p belongs to the pixel (comprising pixel p itself) of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel

The pixel that belongs to same color block (comprises pixel

Itself);

Expression with pixel p be center pixel window was comprised and pixel p does not belong to the pixel of same color block;

Expression is with candidate pixel point in the reference frame

For center pixel window comprised and pixel The pixel that does not belong to same color block; λ representes not belong to center pixel the accumulation calculating weight of the area pixel of same block.In one embodiment of the invention, λ can be 0.01.

At last, obtain each pixel pixel parallax in the present frame, obtain the corresponding basic depth map of present frame according to window matching value and search volume.

$d_s=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{E_d\right\}.$

The real-time deep figure that the quick computing unit of cutting apart based on color 735 of depth map calculates based on image segmentation.For each row of present frame, the quick computing unit of cutting apart based on color 735 of depth map makes the line slip of all blocks on average be initially zero, i.e. T _S=0, wherein s is the block under the pixel.Then each row is carried out calculating based on the average initialization of the line slip of carve information, promptly in window line width scope to pixel value difference according to the place block, carry out accumulation calculating.

j∈[0，N]，T _S(i，j)＝T _S(i，j)+C _(i，j)

In the following formula, T _{S (i, j)}(i, j) the affiliated sliding average of block on line direction is by belonging to block S (i, C j) for remarked pixel _{(i, j)}Adding up obtains, C _{(i, j)}Pixel in the expression present frame (i, j) with reference frame in the pixel value difference absolute value of respective pixel.Calculate through following formula, can obtain any delegation based on the average initial calculation result of the line slip of carve information.

The quick computing unit of cutting apart based on color 735 of depth map is based on the average recursive calculation of the line slip of the carve information of image segmentation.Adopt following formula that any delegation is carried out based on the average recursive calculation of the line slip of carve information:

$T_{S(i,j+\frac{N}{2})}=T_{S(i,j+\frac{N}{2})}+C_{S(i,j+\frac{N}{2})}$

$T_{S(i,j-\frac{N}{2}-1)}=T_{S(i,j-\frac{N}{2}-1)}-C_{S(i,j-\frac{N}{2}-1)},$

In following formula, (i, j), the result of calculation of the right endpoint of its corresponding window does the center pixel that is listed as for the capable j of i

The result of calculation of the left end point of the previous window that it is corresponding does

Respectively according to the block at its place

Carry out addition and subtraction, upgrade block T _SThe average recursive calculation result of line slip.Finally for each center pixel (i, j), obtain based on the line slip average computation result of carve information do

$A_{(i,j)}^r=T_{S(i,j)}.$

The quick computing unit of cutting apart based on color 735 of depth map calculates based on the initialization of the row moving average of the carve information of image segmentation.For each row of present frame, make the row moving average of all blocks be initially zero, i.e. G _S=0, wherein s is the block under the pixel, and G is a row moving average result of calculation.Then each row is carried out calculating based on the row moving average initialization of carve information, promptly in the window column wide region to based on the line slip average computation result of carve information according to the place block, carry out accumulation calculating.

i∈[0，N]， $G_{S(i,j)}=G_{S(i,j)}+A_{(i,j)}^r$

In following formula, G _{S (i, j)}Remarked pixel (i, j) the window moving average on the affiliated block column direction, Pixel (i, j) the line slip average computation result on line direction in the expression present frame based on carve information.Calculate through following formula, can obtain for the initial calculation result of any rows of directions based on the row moving average of carve information.

The quick computing unit of cutting apart based on color 735 of depth map is based on the row moving average recursive calculation of the carve information of image segmentation.Adopt following formula to carry out row moving average recursive calculation to any row based on carve information:

$G_{S(i+\frac{N}{2},j)}=G_{S(i+\frac{N}{2},j)}+A_{S(i+\frac{N}{2},j)}^r$

$G_{S(i-\frac{N}{2}-1,j)}=G_{S(i-\frac{N}{2}-1,j)}-A_{S(i-\frac{N}{2}-1,j)}^r,$

In following formula, (i, j), the line slip average computation result of its corresponding window lower extreme point does to be listed as the capable center pixel of i for j

The line slip average computation result of the upper extreme point of a window does on its respective column direction

Respectively according to the block at its place

Carry out addition and subtraction, recurrence is upgraded block G _SRow moving average result.(i, j) the window accumulation result of calculation based on carve information is G finally to obtain any pixel _{S (i, j)}

The quick computing unit 735 of depth map based on color is cut apart is asked for parallax and depth map based on the carve information of image segmentation.Through obtaining any pixel (i, basic window accumulation result of calculation Q j) _{(i, j)}With window accumulation result of calculation G based on the color carve information _{S (i, j)}, utilize The above results to adopt the basic depth map of following improvement to ask for algorithm in real time:

M(i，j)| _d＝λ(Q _(i，j)| _d-G _S(i，j)| _d)+G _S(i，j)| _d

In following formula | _dExpression is for d specific in the hunting zone, and (i, j) the final window of expression is accumulated result of calculation, Q to M _{(i, j)}-G _{S (i, j)}When expression window accumulation is calculated and center pixel (i j) does not belong to the accumulation calculating section of same block, G _{S (i, j)}(i j) belongs to the accumulation calculating section of same block, and λ representes that the accumulation that does not belong to the area pixel of same block with center pixel calculates weight for expression and center pixel.In one embodiment of the invention, λ can be 0.01.(i, the corresponding parallax value of minimum value j) is the parallax d of current frame pixel finally to get M _S2

$d_{s2}=\underset{d\∈[d_{\min },d_{\max }]}{\min }\left\{M(i,j){|}_d\right\}.$

In one embodiment of the invention; Based on the calculating of the algorithm of four kinds of different characteristics to the depth map of present frame; The quick calculating of the calculating of the depth map that is the real-time calculating of the calculating of basic depth map, basic depth map, cuts apart based on color and the depth map cut apart based on color; Four kinds of algorithms are separate, and the real-time computing unit of promptly basic depth map 732, basic depth map computing unit 733, the depth map computing unit of cutting apart based on color 734 and the quick computing unit of cutting apart based on color 735 of depth map can independent operating compute depth figure.Certainly depth map computing unit 734 that it will be appreciated by persons skilled in the art that the real-time computing unit of basic depth map 732, basic depth map computing unit 733, cuts apart based on color and the quick computing unit 735 of depth map cut apart based on color can independent operating compute depth figure independence or comprehensive use all belong to protection scope of the present invention.

732 pairs of basic depth map computing units 733 of the real-time computing unit of the basic depth map of the embodiment of the invention improve on computation complexity, have reached the real-time requirement.The depth map computing unit of cutting apart based on color 734 of the embodiment of the invention further improves calculating on the accuracy the real-time computing unit 732 of basic depth map, and the quick computing unit of cutting apart based on color 735 of depth map has further carried out improving on computational speed to the real-time computing unit 734 of basic depth map again and obtained real-time operation, depth map that quality is higher.

Depth map improves depth map that 740 pairs of depth map computing modules 730 of module calculate to carry out based on the improvement of the depth map of reference frame with based on the improvement of present frame color distribution and locus.Wherein, the depth map that calculates of depth map computing module 730 comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation.

Depth map improves module 740 and judges whether to carry out the improvement based on the reference frame depth map.When judgement need be carried out the improvement based on the reference frame depth map, the depth map that calculates is asked for, searches, checked and improves.At first, depth map improves module 740 is asked for reference frame according to the depth map that calculates depth map.Only need depth map calculate interface will before reference frame as present present frame, with before present frame as present reference frame, the corresponding depth map of reference frame before can calculating.Then, by certain the pixel P in the present frame _C(i, degree of depth Z j) _C(i j) can find pixel P corresponding in the reference frame _R(i, j), then the corresponding degree of depth of this pixel in the reference frame depth map is Z _R(i, j).

Depth map improves the improvement that module 740 is carried out based on the reference frame depth map.At first, inspection depth map.If Z _C(i, j) and Z _R(i, j) difference of two depth values is less than or equal to predetermined threshold t, then representes two pixel P _C(i, j) and P _R(i, j) corresponding same object point can judge that then depth calculation is accurate; If Z _C(i, j) and Z _R(i when j) difference of two depth values is greater than predetermined threshold t, representes that then two pixels are not corresponding, can assert that present frame is at pixel P _C(i, the depth calculation mistake of j) locating.Predetermined threshold t can reach needs as the case may be and set.In one embodiment of the invention, predetermined threshold t can be 1.If the depth calculation mistake then can be got Z _C(i, j) and Z _R(i, j) between the two smaller is current frame pixel point P _C(i, degree of depth j), thus reach the purpose of improving the depth map quality.

Depth map improves module 740 and further judges whether to carry out the improvement based on present frame color distribution and locus.Refer to that based on the present frame color distribution and the improvement of locus the ID value can carry out the bilateral filtering improvement by color distribution in the window area of the pixel value of its corresponding present frame place and locus.

When the improvement based on the reference frame depth map is carried out in judgement; Pixel Z (p) for any the expression degree of depth among the ID figure; Its respective pixel in present frame is p, is center pixel with the pixel p in the present frame, and window width is that all pixels in the window of N are q.The pixel value difference Gauss weight factor w of all pixel q and center pixel p in the calculation window ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q), and the depth value Z (q) of two weight factors and pixel q multiplied each other, obtain the depth value Z (q) of cum rights repeated factor.Pixel value difference Gauss weight factor w ₁(V _p, V _q), pixel space is apart from Gauss's weight factor w ₂(S _p, S _q) be respectively:

$w_1(V_p,V_q)=e^{-\frac{{(V_p-V_q)}^2}{2{\mathrm{\σ}}_1^2}}$

$w_2(S_p,S_q)=e^{-\frac{{(p\left(x\right)-q\left(x\right))}^2}{2{\mathrm{\σ}}_2^2}}*e^{-\frac{{(p\left(y\right)-q\left(y\right))}^2}{2{\mathrm{\σ}}_2^2}},$

Wherein,

is the gaussian filtering function; V remarked pixel value, p (x), p (y); Q (x), q (y) is the horizontal ordinate of remarked pixel p, q respectively.σ ₁, σ ₂Expression gaussian filtering variance.Wherein, σ ₁, σ ₂Can reach as the case may be needs and is provided with.In one embodiment of the invention, σ ₁=15, σ ₂=5.

The depth value Z (q) that depth map improves 740 pairs of cum rights repeated factors of module adds up; And weight is carried out normalization handle, the degree of depth that obtains is the degree of depth of improving the back pixel p

$\stackrel{\‾}{Z}\left(p\right)=\frac{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)Z\left(q\right)}{{\mathrm{\Σ}}_{q\∈N_p}{w}_1(V_p,V_q)w_2(S_p,S_q)}.$

Can know that from following formula p is a pixel of present frame, q is for being the center with current pixel point p, and window width is any pixel in the window of N, N _pThe expression window size.In one embodiment of the invention, window width N is 7, and then window size is 7*7.Z (q) is the depth value of pixel q in the present frame,

Be the depth value after the pixel p improvement in the present frame, w ₁(V _p, V _q) be Gauss's weight factor of the pixel value difference of pixel p and pixel q, w ₂(S _p, S _q) be Gauss's weight factor of the pixel space distance of pixel p and pixel q.

Noise or erroneous matching when the depth map after adopting above-mentioned depth map to improve one's methods to improve not only can filter out depth calculation, and keep the boundary information of present frame ID.

The depth map based on reference frame that the embodiment of the invention provides improves and can use separately or use simultaneously based on the improvement of present frame color distribution and locus.

Depth map that output module 750 will be calculated by depth map computing module 730 and depth map improve the depth map based on reference frame after the improvement of module 740, the plane picture based on the original input of the depth map of present frame color distribution and locus and plane picture input module 710 after improving is exported.

Depth map according to the embodiment of the invention is asked for device, adopts to ask for algorithm in real time based on the depth map of moving average method and can realize that real-time deep figure extracts, thereby can be applied in the stereo visual system that various real-times have relatively high expectations; The degree of depth that employing is cut apart based on color of image is asked for algorithm fast, has made full use of the classified information of color of image, has improved basic depth map acquiring method, uses the moving average method simultaneously and has realized that depth map quasi real time asks for; Employing improves algorithm based on the depth map of reference frame depth map, detects the occlusion area and the erroneous matching of image effectively, and through utilizing the depth information of reference frame, can obviously improve the wrong depth information of occlusion area; Depth map through based on gaussian filtering improves algorithm; Both can keep the depth map boundary information, the noise in again can the filtering depth map has further improved the quality of depth map; And the depth map that can satisfy multiple time requirement, quality requirement is asked for, and range of application is more extensive.

In the description of this specification, the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means the concrete characteristic, structure, material or the characteristics that combine this embodiment or example to describe and is contained at least one embodiment of the present invention or the example.In this manual, the schematic statement to above-mentioned term not necessarily refers to identical embodiment or example.And concrete characteristic, structure, material or the characteristics of description can combine with suitable manner in any one or more embodiment or example.

Although illustrated and described embodiments of the invention; For those of ordinary skill in the art; Be appreciated that under the situation that does not break away from principle of the present invention and spirit and can carry out multiple variation, modification, replacement and modification that scope of the present invention is accompanying claims and be equal to and limit to these embodiment.

Claims (18)

1. a depth map acquiring method is characterized in that, comprises the steps:

The input plane image, wherein, said plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Said plane picture is carried out preliminary treatment;

Judge whether pretreated plane picture need carry out that color of image is cut apart and whether need move the moving average algorithm; To said plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates or calculates fast based on the depth map that color is cut apart; Wherein, When judging that said pretreated plane picture need not carry out color of image when cutting apart; Need further to judge whether utilization moving average algorithm, when judgement needs utilization moving average algorithm, said plane picture is carried out the real-time calculating of basic depth map; Otherwise, said plane picture is carried out the calculating of basic depth map;

When judging that said pretreated plane picture need carry out color of image when cutting apart; Need further to judge whether utilization moving average algorithm; When judgement needs utilization moving average algorithm, said plane picture is carried out calculating fast based on the depth map that color is cut apart; Otherwise the depth map that said plane picture is carried out cutting apart based on color calculates;

The depth map that calculates is carried out improving and based on the improvement of present frame color distribution and locus based on the depth map of reference frame; Wherein, the said depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation; And

Export the said depth map that calculates and improve after the depth map based on reference frame, depth map and the said plane picture after improving based on present frame color distribution and locus.

2. depth map acquiring method as claimed in claim 1; It is characterized in that; When the plane picture of input is two-way video or two-way image sequence; For the synchronization picture frame, with the picture frame in one road video or the image sequence as present frame, with the frame as a reference of the picture frame in another road video or the image sequence;

When the plane picture of input when being single channel video or single channel image sequence, with said single channel video or the current image frame of single channel image sequence on time shaft as present frame, with the back frame of the current image frame on the time shaft as the default reference frame.

3. depth map acquiring method as claimed in claim 2 is characterized in that, when said plane picture is single channel video or single channel image sequence, said plane picture is carried out preliminary treatment comprise the steps:

Said single channel video or single channel image sequence are carried out the key frame judgement to obtain the reference frame as the depth map calculating of present frame;

When said plane picture is two-way video or two-way image sequence, said plane picture is carried out preliminary treatment comprise the steps: said two-way video or two-way image sequence are carried out the outer level line correction of image.

4. depth map acquiring method as claimed in claim 1 is characterized in that, the said calculating that plane picture is carried out basic depth map comprises the steps:

Window and search volume are set;

According to said window said plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with said search volume, obtain the corresponding basic depth map of present frame, comprise the steps:

Any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said window matching value E should be arranged _d, at said a plurality of window matching value E _dIn choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _s

5. depth map acquiring method as claimed in claim 4 is characterized in that, said plane picture is carried out the real-time calculating of basic depth map, comprises the steps:

Initialization calculating and the average recursive calculation of line slip in the enterprising every trade moving average of line direction; And, obtain any pixel corresponding real-time window matching value Q of said present frame in the initialization calculating and the row moving average recursive calculation of the enterprising ranks moving average of column direction _{(i, j)}

Any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said real-time window matching value Q should be arranged _{(i, j)}, at said a plurality of real-time window matching value Q _{(i, j)}In choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _S1

Ask for the real-time pixel parallax of each pixel of present frame, obtain the corresponding basic real-time deep figure of present frame.

6. depth map acquiring method as claimed in claim 1 is characterized in that, the said depth map that plane picture is carried out cutting apart based on color calculates and comprises the steps:

Each pixel of said present frame is connected to n bar limit with corresponding n pixel on every side; The weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) | and, obtain a plurality of limit edge (i; J); Wherein, i is a pixel of present frame, and j is on every side n the pixel corresponding with a pixel of said present frame;

Each pixel of said present frame is divided into one type, and setting initial merger threshold value is threshold (i)=c, is sorted according to weights from small to large in said every limit;

The pixel of the pairing present frame in limit after the ordering and on every side n the pixel place class corresponding with the pixel of said present frame are carried out the operation of merger in twos, and the color that obtains after each pixel is sorted out is cut apart figure;

Window and search volume are set; In said window,, window area is divided into two parts, a part of and currently waits to ask degree of depth pixel to belong to same color region according to the color segmentation result; Another part and currently wait to ask degree of depth pixel not belong to same color region; Reduce the latter's calculating weight, said plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with said search volume, obtain the corresponding depth map of present frame, comprise the steps: based on image segmentation

Any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said window matching value E should be arranged _d, at said a plurality of window matching value E _dIn choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _s

7. depth map acquiring method as claimed in claim 5 is characterized in that, said plane picture is carried out calculating fast based on the depth map that color is cut apart, and comprises the steps:

Said plane picture is carried out based on the line slip of image segmentation average initialization calculating and the average recursive calculation of line slip; And carry out initialization calculating and row moving average recursive calculation based on the row moving average of image segmentation, obtain window matching value G based on image segmentation _{S (i, j)}

According to said real-time window matching value Q _{(i, j)}With said window matching value G based on image segmentation _{S (i, j)}, ask for real-time window matching value M based on image segmentation (i, j);

Any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said real-time window matching value M (i based on image segmentation should be arranged; J); (i chooses the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame in j) at said real-time window matching value M based on image segmentation _S2

Ask for the real-time pixel parallax of each pixel of present frame, obtain the corresponding real-time deep figure of present frame based on image segmentation.

8. depth map acquiring method as claimed in claim 1 is characterized in that, said the depth map that calculates is carried out improving based on the depth map of reference frame, comprises the steps:

The position of reference frame and present frame is exchanged, calculated the depth map of reference frame;

Each pixel in the said present frame is searched corresponding pixel points and the depth value of said corresponding pixel points in the said reference frame;

Depth value according to corresponding pixel points in the said reference frame is checked the depth value of each pixel in the present frame; Comprise: when the difference of the depth value of corresponding pixel points in the depth value of a pixel in the said present frame and the said reference frame is less than or equal to predetermined threshold, judge that then the depth value of this pixel in the said present frame is correct; When the difference of the depth value of corresponding pixel points in the depth value of certain pixel in the said present frame and the said reference frame during, then judge the depth value mistake of this pixel in the said present frame greater than said predetermined threshold;

In judging said present frame during the depth value mistake of a pixel, the depth value of corresponding pixel points in the depth value of this pixel in the said present frame and the said reference frame is compared, comprising:

When the depth value of this pixel in the said present frame less than said reference frame in the depth value of corresponding pixel points, keep the depth value of this pixel in the said present frame;

When the depth value of this pixel in the said present frame greater than said reference frame in the depth value of corresponding pixel points, the depth value of this pixel in the said present frame is the depth value of corresponding pixel points in the said reference frame.

9. depth map acquiring method as claimed in claim 1 is characterized in that, said the depth map that calculates is carried out the improvement based on present frame color distribution and locus, comprises the steps:

Any depth value in the said depth map that calculates is carried out bilateral filtering according to color distribution and locus in its corresponding pixel points place window in present frame.

10. a depth map is asked for device, it is characterized in that, comprising:

The plane picture input module, said plane picture input module is used for the input plane image, and wherein, said plane picture comprises single channel video, two-way video, single channel image sequence or two-way image sequence;

Pretreatment module, said pretreatment module are used for said plane picture is carried out preliminary treatment;

The depth map computing module; Said depth map computing module is used to judge whether the pretreated plane picture of said pretreatment module need carry out that color of image is cut apart and whether need move the moving average algorithm; To said plane picture optionally carry out the real-time calculating of the calculating of basic depth map, basic depth map, the depth map of cutting apart based on color calculates fast or calculates based on the depth map that color is cut apart; Wherein, Said depth map computing module comprises judging unit, the basic real-time computing unit of depth map, basic depth map computing unit, the depth map computing unit of cutting apart based on color and the quick computing unit of cutting apart based on color of depth map, wherein

Whether the pretreated plane picture of said judgment unit judges need carry out color of image cuts apart; When judging that said pretreated plane picture need not carry out color of image when cutting apart; Need further to judge whether utilization moving average algorithm; When judgement needed utilization moving average algorithm, the real-time computing unit of said basic depth map carried out the real-time calculating of basic depth map to said plane picture; Otherwise said basic depth map computing unit carries out the calculating of basic depth map to said plane picture;

When the said pretreated plane picture of said judgment unit judges need carry out color of image when cutting apart; Need further to judge whether utilization moving average algorithm; When judgement needed utilization moving average algorithm, the said quick computing unit of cutting apart based on color of depth map carried out calculating fast based on the depth map that color is cut apart to said plane picture; Otherwise the said depth map computing unit of cutting apart based on color calculates the depth map that said plane picture carries out cutting apart based on color;

Depth map improves module; Said depth map improves module and is used for the depth map that said depth map computing module calculates is carried out that depth map based on reference frame improves and based on the improvement of present frame color distribution and locus; Wherein, the said depth map that calculates comprise the basic depth map that calculates through basic depth map, through the basic real-time deep figure of calculating in real time of basic depth map, through the depth map of cutting apart based on color calculate based on the depth map of image segmentation and the real-time deep figure that calculates fast through the depth map of cutting apart based on color based on image segmentation; And

Output module, said output module be used to export the said depth map that calculates and improve after the depth map based on reference frame, depth map and the said plane picture after improving based on present frame color distribution and locus.

11. depth map as claimed in claim 10 is asked for device; It is characterized in that; When the plane picture of said plane picture input module input is two-way video or two-way image sequence; For the synchronization picture frame, with the picture frame in one road video or the image sequence as present frame, with the frame as a reference of the picture frame in another road video or the image sequence;

When the plane picture of said plane picture input module input is single channel video or single channel image sequence; With said single channel video or the current image frame of single channel image sequence on time shaft as present frame, with the back frame of the current image frame on the time shaft as the default reference frame.

12. depth map as claimed in claim 10 is asked for device; It is characterized in that; When said plane picture was single channel video or single channel image sequence, said pretreatment module was carried out the key frame judgement to obtain the reference frame as the depth map calculating of present frame to said single channel video or single channel image sequence;

When said plane picture was two-way video or two-way image sequence, said pretreatment module was carried out the outer level line correction of image to said two-way video or two-way image sequence.

13. depth map as claimed in claim 10 is asked for device, it is characterized in that, said basic depth map computing unit is provided with window and search volume, according to said window said plane picture is carried out calculating based on the pixel coupling of window, obtains window matching value E _d, and according to window matching value E _dObtain each pixel pixel parallax in the present frame with said search volume, obtain the corresponding basic depth map of present frame, be included in any pixel in the present frame corresponding to each relative position in the said search volume to a plurality of said window matching value E should be arranged _d, and at said a plurality of window matching value E _dIn choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _s

14. depth map as claimed in claim 13 is asked for device; It is characterized in that; The real-time computing unit of said basic depth map is in the initialization calculating and the average recursive calculation of line slip of the enterprising every trade moving average of line direction; And, obtain any pixel corresponding real-time window matching value Q of said present frame in the initialization calculating and the row moving average recursive calculation of the enterprising ranks moving average of column direction _{(i, j)}, and any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said real-time window matching value Q should be arranged _{(i, j)}, at said a plurality of real-time window matching value Q _{(i, j)}In choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _S1, and ask for the real-time pixel parallax of each pixel of present frame, obtain the corresponding basic real-time deep figure of present frame.

15. depth map as claimed in claim 10 is asked for device, it is characterized in that, the said depth map computing unit of cutting apart based on color is carried out following steps:

Each pixel of said present frame is connected to n bar limit with corresponding n pixel on every side, the weight w on every limit (i j) is the absolute value of the pixel value difference of corresponding two end points | value (i)-value (j) |; Obtain a plurality of limit eage (i; J), wherein, i is a pixel of present frame; J is on every side n the pixel corresponding with a pixel of said present frame; Each pixel of said present frame is divided into one type, and setting initial merger threshold value is threshold (i)=c, is sorted according to weights from small to large in said every limit; And to the pixel of the pairing present frame in limit after the ordering and corresponding with the pixel of said present frame around n pixel place type carry out merger in twos and operate, the color that obtains after each pixel classification is cut apart figure;

Window and search volume are set; In said window,, window area is divided into two parts, a part of and currently waits to ask degree of depth pixel to belong to same color region according to the color segmentation result; Another part and currently wait to ask degree of depth pixel not belong to same color region; Reduce the latter's calculating weight, said plane picture is carried out calculating based on the pixel coupling of window, obtain window matching value E _d

According to window matching value E _dObtain each pixel pixel parallax in the present frame with said search volume, obtain the corresponding depth map of present frame, comprise the steps: based on image segmentation

Any pixel in present frame corresponding to each relative position in the said search volume to a plurality of said window matching value E should be arranged _d, at said a plurality of window matching value E _dIn choose the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame _s

16. depth map as claimed in claim 14 is asked for device; It is characterized in that; The said quick computing unit of cutting apart based on color of depth map carries out based on the line slip of image segmentation average initialization calculating and the average recursive calculation of line slip said plane picture; And carry out initialization calculating and row moving average recursive calculation based on the row moving average of image segmentation, obtain window matching value G based on image segmentation _{S (i, j)}, according to said real-time window matching value Q _{(i, j)}With said window matching value G based on image segmentation _{S (i, j)}Ask for real-time window matching value M (i based on image segmentation; J), any pixel in present frame corresponding to each relative position in the said search volume to should have a plurality of said real-time window matching value M based on image segmentation (i, j); (i chooses the parallax value d of the corresponding parallax value of minimum value as current pixel point in the said present frame in j) at said real-time window matching value M based on image segmentation _S2, ask for the real-time pixel parallax of each pixel of present frame, obtain the corresponding real-time deep figure of present frame based on image segmentation.

17. depth map as claimed in claim 10 is asked for device; It is characterized in that; Said depth map improves module and the position of reference frame and present frame is exchanged the depth map that calculates reference frame; Each pixel in the said present frame is searched corresponding pixel points and the depth value of said corresponding pixel points in the said reference frame; Depth value according to corresponding pixel points in the said reference frame is checked the depth value of each pixel in the present frame; Comprise: when the difference of the depth value of corresponding pixel points in the depth value of a pixel in the said present frame and the said reference frame is less than or equal to predetermined threshold, judge that then the depth value of this pixel in the said present frame is correct; When the difference of the depth value of corresponding pixel points in the depth value of a pixel in the said present frame and the said reference frame during, then judge the depth value mistake of this pixel in the said present frame greater than said predetermined threshold;

In judging said present frame during the depth value mistake of a pixel, the depth value of corresponding pixel points in the depth value of this pixel in the said present frame and the said reference frame is compared, comprising:

When the depth value of this pixel in the said present frame less than said reference frame in the depth value of corresponding pixel points, keep the depth value of this pixel in the said present frame;

When the depth value of this pixel in the said present frame greater than said reference frame in the depth value of corresponding pixel points, the depth value of this pixel in the said present frame is the depth value of corresponding pixel points in the said reference frame.

18. depth map as claimed in claim 10 is asked for device; It is characterized in that said output module carries out bilateral filtering to any depth value in the said depth map that calculates according to color distribution and locus in its corresponding pixel points place window in present frame.

Images