Background technology
From the eighties of eighties of last century, abroad just begin to have the report of view synthetic technology, domestic relevant scholar is in recent years also for the synthetic a few thing of having done of image.The more than ten years in past, synthetic this direction of view has received widely to be paid close attention to, because it has related to the technology of computer vision, computer graphics and Digital Image Processing all directions, this makes it not only be difficult to solve, and method also is tending towards variation.
The synthetic of virtual view is one of difficult point of three-dimensional video-frequency for many years always.The view that the mode of traditional composograph is based on 3-D geometric model synthesizes.The method based on model for traditional need be paid huge workload owing to generate complicated geometrical model, and needs a lot of skills; In addition, in order to draw out lighting effect preferably, often need very expensive hardware spending, amount of calculation is huge, be difficult on common computing platform, handle in real time, and the sense of reality of image does not ensure.So another kind of view synthetic technology based on image obtains increasing concern under this situation.Based on the view synthetic technology of the image product that to be computer vision cooperate with figure educational circles.Just proposed the imagination that combines with graphics as far back as vision worker in 1984, the trial of cooperating had several times also been arranged, but never find an appropriate direction, disputed on very big for ultimate aim especially.Free discussion by initiations such as R C Jain in 1991 afterwards all once promoted both cooperations, such as having proposed image modeling based on CAD etc.
It is the view distortion that Seitz and Dyer have proposed a kind of image deformation method; The image of the new viewpoint that this method generated can keep the appearance profile characteristic of original reference image; This method has solved original image distortion method in the process that two width of cloth images to the same scene different points of view are out of shape, and intermediate image is difficult to guarantee the problem of the inner shape characteristic in the original image.Synthetic method is looked in the virtual centre that people such as Ott propose, and it is synthetic that first utilizes stereo-picture to carry out view, and they have considered the problem of the virtual contact ac of participant's eyes in teleconference.Given two video cameras that are positioned over teleconference display two ends; Utilize two known width of cloth images to synthesize virtual medial view; But do not consider occlusion area in the literary composition, ignored in the disparity map, used gray-level interpolation in the hollow sectors of synthetic view owing to block the cavity of causing.People such as McMillan provide in hypothesis under the situation of disparity map, have obtained the method for in real time synthetic new viewpoint image in the helmet-type display system, and have provided a kind of straightforward procedure that solves occlusion issue based on forward direction mapping (Forwarding Mapping).When Chen and Williams effectively carry out problem that image draws in solving computer graphics, introduced image interpolation method, only pointed out that under the baseline condition parallel with the plane of delineation, the image interpolation method of linearity could synthesize correct view.People such as Lawrence at first utilize belief propagation algorithm (BP), over-segmentation technology under Hidden Markov (MRF) framework, obtain dense disparity map, utilize interpolation technique to synthesize medial view then.
It is external to compare, domestic less to the synthetic research of view, after obtaining reliable dense disparity map, generates disparity map through the correspondence position of pixel in the image of the left and right sides of seeking in the medial view like people such as Zhang Zhaoyang, Anping, Lv Chaohui, Luo Yan; People such as Hu Zhiping propose the method for the synthetic new viewpoint of a kind of image based on the horizontal and vertical mobile collection of video camera, and this method preferential profile that guarantees in composograph is accurately corresponding, thereby makes the composograph clear-cut; People such as Wu Qiongyu have then proposed a kind of view synthesizing method as if the image sequence of learning based on image under the demarcation condition, and this method has been avoided coupling and three-dimensional reconstruction, directly generates medial view; People such as Chai Deng-feng have proposed a kind of two minutes figure and have cut the estimation that algorithm carries out parallax, and foreground area and background area are then distinguished in the detection of blocking then, find out the match point of medial view in input picture at last and obtain the medial view picture.
Summary of the invention
Synthetic technology is looked in the centre based on disparity estimation that the purpose of this invention is to provide under a kind of parallel double lens camera system, comprises the application of disparity estimation and two aspects of view interpolation.
The present invention adopts following technical scheme for realizing above-mentioned purpose:
The present invention is based on the centre of improving the BP algorithm looks synthetic method and comprises the steps:
Step 1) adopts the method based on the iteration of improving BP, obtains the difference vector information of each pixel;
Step 2); Detect occlusion area; And the parallax of occlusion area compensated, and then carry out the interpolation calculation of medial view: at first utilize the consistency constraint condition that image is carried out dividing region, be divided into occlusion area, consistent zone and fuzzy region to image; Meanwhile carry out the compensation of occlusion area, take different interpolation to obtain medial view according to the characteristics in each zone itself then.
Preferably, the described method based on the iteration of improving BP of step 1) is following:
The first step, the initial matching cost when calculating the 0th grade, the data item during then to each rank is carried out initialization; The color distance of the node when calculating each rank and its neighbours' domain node, and the color distance value between the adjacent node during to each rank is carried out initialization;
Second step, total L level, l=0,1 ..., L, the iterations of each grade are t=0,1 ..., T, L and T are the natural number greater than 1;
In the 3rd step, utilize formula
To iterations is that 0 o'clock message is carried out initialization; Then according to formula
Calculate the message m of each node
L, t(d
q), and be the minimum value of information minimum of each node selection;
In the 4th step, the message of calculating gained is handled as follows:
for?d
q?from?1?tok-1:
m
l,t(d
q)←min(m
l,t(d
q),m
l,t(d
q-1)+c)
for?d
q?from?k-2to?0:
m
l,t(d
q)←min(m
l,t(dq),m
l,t(d
q+1)+c)
Add minimum the ρ (I) of
formula, and save as MIN;
The 5th step, compare value in the message vector and MIN, if less than MIN, then the value of correspondence position substitutes with MIN in the message vector, at last the message vector is carried out normalization, gets message values to the end;
In the 6th step, iterations adds 1, repeats for the 3rd to the 5th step, is T up to iterations;
In the 7th step, progression subtracts 1, repeats for the 3rd to the 6th step, knows that progression becomes 0;
In the 8th step, obtaining each node according to formula
at last is that the optimum label of pixel is a parallax.
Preferably, step 2) the described consistency constraint condition of utilizing is carried out dividing region to image, is divided into occlusion area, consistent zone and fuzzy region to image and is meant: at first, and for left disparity map d
LRUtilize formula d
LR(x
L, y)-d
RL(x
L-d
LR(x
L, y), y)>T detects right half occlusion area, then for right disparity map d
RLUtilize formula d
RL(x
R, y)-d
LR(x
R+ d
RL(x
R, y), y)>T detects left half occlusion area; And respectively a left side half occlusion area in half occlusion area of the right side in the left disparity map and the right disparity map is carried out mark; In addition, the parallax value of the rightmost partial pixel point of the Far Left partial pixel of left image and right image all composes 0; Then, to satisfying formula d in the left disparity map
LR(x
L, y)=d
RL(x
L-d
LR(x
L, y), pixel region y) is labeled as right consistent zone, for satisfying formula d in the right disparity map
RL(x
R, y)=d
LR(x
R+ d
RL(x
R, y), pixel region y) is labeled as the first from left and causes the zone; At last, the zone that is not labeled in the left disparity map, being labeled as right fuzzy region, is the zone marker that is not labeled in the right disparity map left fuzzy region.
Preferably, step 2) described interpolation method step is following:
At first, be reference with left image, utilize formula:
Look (I in the middle of generating left transition
LM), the pixel value of its right-of-center in political views's half occlusion area directly replaces with the pixel value of the correspondence position of left image, and right uniformity zone and right fuzzy region are utilized interpolation technique;
Then, be reference with right image, utilize formula:
Generate right transition medial view picture (I
RM);
At last, left and right sides transition medial view is looked like to utilize formula:
carries out an interpolation again and obtained medial view.
Preferably, the medial view that obtains synthesizing is as I
MIn still have the zone do not shone upon, take the method for reverse mapping, in the image of the left and right sides, seek its immediate point respectively,
Choose respectively and satisfy formula
And formula
Left and right sides image in optimal match point, wherein the hunting zone of the independent variable of two formulas is respectively formula x
M≤x
L≤x
M+ (1-a) d
MaxWith formula x
M-ad
Max≤x
R≤x
MUtilize formula aI then
LM(x
L ', y)+(1-a) I
RM(x
R ', y) interpolation is carried out in the zone of not shone upon, obtained final medial view picture.
Advantage of the present invention and effect are:
1. improve the BP algorithm, removed excessive level and smooth ill effect, further improved the accuracy that parallax calculates.
2. design new interpolation algorithm, considered the characteristics of stereo-picture various piece better, improved the visual effect and the quality of virtual view.
Other advantages of the present invention and effect will continue to describe below.
Embodiment
Below in conjunction with accompanying drawing and embodiment, technical scheme of the present invention is done further to set forth.
The present invention is based on the centre of improving the BP algorithm and look synthetic method, may further comprise the steps:
From disparity estimation and two big aspects of view interpolation middle view synthetic technology is analyzed and studied.The disparity estimation purpose here also is to obtain dense reliable disparity map; But we will adopt a kind of method based on the iteration of improving BP here; Obtain the difference vector information of each pixel; All be that difference between the simple parallax label that is endowed according to neighbor is provided with level and smooth cutoff value when utilizing linear model to calculate level and smooth, cause the excessively level and smooth of disparity map possibly in view of former BP algorithm.We have proposed a kind of color distance that utilizes and come self adaptation that the method for level and smooth cutoff value is set in this part; After having obtained disparity map, we will carry out view interpolation, but, because the existence of occlusion area has had dense disparity map, might not obtain good medial view picture, because not there is the matched pixel point in occlusion area itself.Therefore, before view interpolation, need detect occlusion area; And the parallax of occlusion area compensated; And then carry out the interpolation calculation of medial view, and considering that here each zone of image possibly take on a different character, we have proposed a kind of new interpolation strategies; At first utilize the consistency constraint condition that image is carried out dividing region; Be divided into occlusion area, consistent zone and fuzzy region to image, meanwhile carry out the compensation of occlusion area, take different interpolation to obtain medial view according to the characteristics in each zone itself then.
Look in the synthetic technology based on the centre of improving the BP algorithm described, described following based on the parallax calculation procedure of improving the BP algorithm:
The first step, the initial matching cost when calculating the 0th grade, the data item during then to each rank is carried out initialization; The color distance of the node when calculating each rank and its neighbours' domain node, and the color distance value between the adjacent node during to each rank is carried out initialization.
Second step, suppose that we have the L level, l=0,1 ..., L, the iterations of each grade are t=0,1 ..., T.
In the 3rd step, utilizing formula
is that 0 o'clock message is carried out initialization to iterations.Then according to formula
Calculate the message m of each node
L, t(d
q), and be the minimum value of information minimum of each node selection.
In the 4th step, the message of calculating gained is handled as follows:
for?d
q?from?1?to?k-1:
m
l,t(d
q)←min(m
l,t(d
q),m
l,t(d
q-1)+c)
for?d
q?fromk-2to?0:
m
l,t(d
q)←min(m
l,t(d
q),m
l,t(d
q+1)+c)
Add minimum the ρ (I) of
formula, and save as MIN.
The 5th step, compare value in the message vector and MIN, if less than MIN, then the value of correspondence position substitutes with MIN in the message vector, at last the message vector is carried out normalization, gets message values to the end.
In the 6th step, iterations adds 1, repeats for the 3rd to the 5th step, is T up to iterations.
In the 7th step, progression subtracts 1, repeats for the 3rd to the 6th step, knows that progression becomes 0;
The 8th goes on foot, and obtains the optimum label (parallax) of each node (pixel) at last according to formula
.
Look in the synthetic technology based on the centre of improving the BP algorithm described, the described consistency constraint condition of utilizing is carried out dividing region to image, is divided into occlusion area, consistent zone and fuzzy region to image and is meant at first, for left disparity map d
LRUtilize formula d
LR(x
L, y)-d
RL(x
L-d
LR(x
L, y), y)>T detects right half occlusion area, then for right disparity map d
RLUtilize formula d
RL(x
R, y)-d
LR(x
R+ d
RL(x
R, y), y)>T detects left half occlusion area.And respectively a left side half occlusion area in half occlusion area of the right side in the left disparity map and the right disparity map is carried out mark.In addition; The rightmost partial pixel of Far Left partial pixel and the right image of left side image is because the restriction of left and right cameras coverage, though in disparity estimation, also obtained parallax value; In fact there is not corresponding matched pixel point in it; Therefore according to the definition of occlusion area, also should to its block mark (for compensate make things convenient for mark the time, the parallax value of these pixels all composes 0).
Then, to satisfying formula d in the left disparity map
LR(x
L, y)=d
RL(x
L-d
LR(x
L, y), pixel region y), we are labeled as right consistent zone, for satisfying formula d in the right disparity map
RL(x
R, y)=d
LR(x
R+ d
RL(x
R, y), pixel region y), we are labeled as the first from left and cause the zone.At last, we are labeled as right fuzzy region to the zone that is not labeled in the left disparity map, are the zone marker that is not labeled in the right disparity map left fuzzy region.Left and right sides image has been divided into half occlusion area, consistent zone and fuzzy region three parts respectively like this.
Look in the synthetic technology based on the centre of improving the BP algorithm described, described interpolation strategies step is following:
The process of view interpolation was divided into for three steps, and at first, we are reference with left image, utilize formula
Look (I in the middle of generating left transition
LM), the pixel value of its right-of-center in political views's half occlusion area directly replaces with the pixel value of the correspondence position of left image, and right uniformity zone and right fuzzy region are utilized interpolation technique.Then, we are reference with right image, utilize formula
Generate right transition medial view picture (I
RM); When generating right transition medial view picture, we have only handled left fuzzy region and left half occlusion area, the first from left is caused the zone do not handle; This be because the first from left cause the zone with right consistent zone in pixel be one to one; Therefore the abscissa value of middle view image therefrom also equates, so after interpolation is carried out in consistent zone to the right side, there is no need again the first from left to be caused the zone and carry out interpolation.So just, obtained left and right sides transition medial view picture.At last, we look like to utilize formula
to carry out interpolation again one time to left and right sides transition medial view.So just, obtained medial view.
The medial view that the above step of process is synthesized is as I
MIn still have the zone do not shone upon, therefore, need further handle it.In order to address this problem, take the method for reverse mapping, in the image of the left and right sides, seek its immediate point respectively,
We choose respectively and satisfy formula for this reason
And formula
Left and right sides image in optimal match point, wherein the hunting zone of the independent variable of two formulas is respectively formula x
M≤x
L≤x
M+ (1-a) d
MaxWith formula x
M-ad
Max≤x
R≤x
MUtilize formula aI then
LM(x
L ', y)+(1-a) I
RM(x
R ', y) interpolation is carried out in the zone of not shone upon, obtained final medial view picture.
In the 3 D video display system, the number of video camera is limited, can only describe three-dimensional scenic from specific visual angle.If seek out the picture of free-viewing angle, realize the effect of " looking around ", the method for solution is exactly the synthetic of medial view.The synthetic of medial view is exactly according to known source images, synthesizes the image of the virtual view between the source images viewpoint, and the simplest situation is a parallel double purpose camera chain; At this moment, virtually look synthetic, generate the medial view that is positioned on the straight line that left and right sides reference view determined promptly according to the image of the Same Scene of left and right cameras picked-up; As shown in Figure 1; It mainly was divided into for two steps: at first, to about two width of cloth reference pictures carry out solid coupling, obtain the difference vector of each pixel; Then according to the match point in the reference picture of the left and right sides to carry out interpolation obtain in the middle of the image of new viewpoint.
As shown in Figure 2, we obtain with the parallel camera chain of binocular and comprise the scene of A to the F object, obtain I
LAnd I
RLeft and right sides stereo-picture is right.As can be seen from the figure, right image I
RIn can only see the projection of A, B, C and F object, and can only see the projection of A, C, E and F object among the left image I L.We see being merely able to as B project objects zone at right image, and are called left half occlusion area at the pixel region that left image can't see; As E project objects zone, seeing and call right half occlusion area at left image at the pixel region that right image can't see; And be defined as full occlusion area to the pixel region of as D, in the image of the left and right sides, all can't see.
When medial view synthetic; Full occlusion area as D might be half occlusion area for certain the virtual viewpoint on the baseline, yet considers that from left and right sides image pair can not get any information of object D; Simultaneously also for the detection of simplifying occlusion area and the difficulty of compensation; We do not consider the occlusion area as D, and the arbitrary objects on the image that hypothesis intermediate-view place video camera is obtained, and that sees in all at least can the arbitrary image in the image of the left and right sides arrives.
Experiment selects for use classical stereo-picture that Tsukuba, Venus, Teddy, Cones are tested.Experiment is to move gained on the PC of Pentium43.4GHz at CPU, and operating system is WindowsXP SP2, and programming tool is VC++6.0.When the realization of algorithm, we are provided with 5 grades BP algorithm, the iterations during each rank be respectively 10,10,5,5,5}.The parameter value of algorithm is as shown in the table.
Fig. 3 has compared level and smooth and has adopted linear model to calculate; The performance of the disparity map that linear model is obtained when getting different cutoff value is when level and smooth item utilizes formula V (x)=min (c|x|, ρ (I)) to calculate; We get 1.0,3.0,5.0,7.0,9.0 respectively by cutoff value, obtain the first five point.Adopt algorithm of the present invention to obtain last point according to the adaptive level and smooth item of cutoff value calculating that is provided with of color distance.
From figure, can see; Calculate the disparity map that level and smooth item is obtained according to the adaptive method that the linear model cutoff value is set of color distance; No matter be whole mistake matching rate (all); Still at the mistake matching rate in parallax discontinuity zone (disc.) and unshielding zone (nocc.), basically all be in minimum position, especially to Venus, Teddy and Cones image.This shows; What the present invention proposed is provided with the algorithm of linear model cutoff value according to the color distance self adaptation; Can a dynamic scope be set for the maximin of level and smooth cost, not only can avoid the excessively level and smooth of disparity map, and help the raising of three-dimensional coupling accuracy.
Level and smooth disparity map that is obtained of method calculating that the linear model cutoff value is set according to this self adaptation is as shown in Figure 4.
The algorithm (representing with Proposed in the experimental result) that we here propose the present invention and a kind of current current paper by Changming Jin, the virtual view composition algorithm (representing with Other in the experimental result) of Hong Jeong proposition compares.Respectively from subjectivity watch, residual plot and three aspects of Y-PSNR come the evaluation algorithms result.
During experiment; Use Venus, Teddy, Cones image sequence; Left and right sides reference picture is respectively second in image sequence and the 6th two field picture; Because the distance of obtaining between the viewpoint of every two field picture equates, we make a be respectively 0.25,0.5,0.75 synthesizing the 3rd, the 4th, the 5th two field picture here.
We at first adopt improvement BP algorithm parallax estimation method to obtain the right horizontal parallax figure of stereo-picture.As shown in Figure 5, we have listed the image that the Venus image sequence adopted invention to propose the virtual view that algorithm synthesizes in algorithm and the document at 0.25,0.5,0.75 o'clock in a difference value.
The earlier simple virtual synthetic view that obtains, the virtual visual point image that composition algorithm can obtain having photo feel is looked in centre of the present invention, more approaches true picture than the algorithm in the document.
In order more to be clear that the difference of algorithms of different performance, Fig. 6 has listed corresponding residual image.White dotted line in the residual plot is exactly the discrepancy of virtual composograph and true picture.Can find out that result's white dotted line that algorithm draws in the document is more clear, that is to say that error rate is higher.In addition, the erroneous point major part occurs in the fringe region of object, so the problem that the processing at edge still will further be optimized in the future.
Algorithm of the present invention is at unshielding, and the error rate of three aspects of error rate of discontinuous and Zone Full all is lower than the algorithm in the document.Than complex image such as Venus, error rate can be reduced to 30% of document algorithm for background; Than complex image such as Teddy and Cones, error rate can be reduced to 80% of document algorithm for background.
At last; We have compared the Y-PSNR that different virtual is looked composition algorithm; We can draw, and for the fairly simple image of background (Venus), algorithm of the present invention has improved 0.1 to 0.3dB than algorithm in the document; And for the image (Teddy and Cones) of more complicated, algorithm of the present invention has improved 0.4 to 1dB than algorithm in the document.