Technical background
Planar video transfers the three-dimensional video-frequency technology to more and more to be paid attention to, but because it realizes bigger, the also comparatively imperfect so far solution of difficulty, is a problem that still is in the exploratory stage.
Seen the people of 3D film all can stay deep impression to its sense of reality, the sort of sense assignee on the spot in person is difficult to forget, and now, three-dimensional video-frequency is just more and more promoted in worldwide and in a plurality of industries and is favored, such as, the large-scale shopping place and the public place of entertainment of some key cities of China have begun to arrange three-dimensional player, advertisement or other promotional videos made in advance with these three-dimensional player plays, give very strong visual impact, can attract very much spectators' eyeball.
Three-dimensional video-frequency must comprise two video sequences, is respectively left view sequence and right view sequence, and two sequences are seen with right eye for respectively spectators' left eye and seen.The people watches why the world has third dimension, be to have a little different and have a parallax (disparity) because left eye and right eye are seen the visual angle in the world, and each frame of stereopsis two sequence correspondences frequently also has parallax, and this also is to make spectators watch three-dimensional video-frequency can produce relief reason as if on the spot in person.
Above-mentioned parallax (disparity) is meant the horizontal displacement corresponding to two picture elements of same world point of left view and right view, and the parallax value of all pixels is formed disparity map (disparity map).The theorem of computer vision field is pointed out, the degree of depth (depth) of the parallax of certain point and its pairing world point is inversely proportional to, and that is to say, and is more little from the parallax value of watching point a little far away more, the parallax of infinite point is 0, and the depth value of being had a few has been formed depth map (depth map).If given left view (right view) and disparity map just can obtain right view (left view).
Optical flow method is the important method of present movement image analysis, and it is meant pattern movement velocity in the time varying image.The variation of image has been expressed in light stream, because it has comprised the information of target travel, but therefore the observed person is used for determining the motion conditions of target.By the definition of the light stream optical flow field of can extending out, optical flow field is meant a kind of two dimension (2D) instantaneous velocity field that all picture elements constitute in the image.Optical flow field is by the calculating to adjacent two frames (M frame and M+1 frame), obtains the motion conditions from the M frame to the M+1 frame of the every bit of M frame, it is emphasized that the motion here refers to the motion in the image, is meant the offset of same object point at consecutive frame.When calculating optical flow field, require the difference of adjacent frame can not be too big, otherwise can't calculate.About the calculating of optical flow field, very ripe and simple algorithm has been arranged at present.
Present existing converting plane video into stereoscopic video method is a kind of method that is called frame time-delay (Frame Delay), and its step is as follows:
A) obtain the sports ground (motion field) of each frame;
B) analyze the image motion situation of this frame by the sports ground of each frame;
C) motion conditions according to this frame directly finds a frame and this frame to form left and right sides view in former sequence;
D) obtain the left and right sides view of all frames, thereby realize converting plane video into stereoscopic video.
There are following two problems in above-mentioned existing method:
(1) problem of vertical parallax: such as the upwards emission of a rocket, no matter how to choose with existing method, also can only choose two figure that vertical parallax is arranged, promptly have only the skew on the vertical direction and do not have skew on the horizontal direction, two figure like this show at three-dimensional display, not only do not have any third dimension, also can allow the people dizzy and tired.
(2) the do more physical exercises situation of object: if the A object is to left movement in the video, the B object moves right, and no matter how chooses with existing method, and the parallax that also can only be one of them object is right and parallax another object is wrong.
Summary of the invention
The objective of the invention is for overcoming the existing methods weak point, a kind of method of the converting plane video into stereoscopic video based on optical flow field is provided, this method is simple, and effect is better, obtains good compromise between method complexity and effect.
The method based on the converting plane video into stereoscopic video of optical flow field that the present invention proposes is characterized in that, may further comprise the steps:
1) optical flow field of given each frame of planar video sequence of calculating:
(11) to the calculating of the first frame optical flow field: first frame and second frame calculate the optical flow field vector of each picture element of the optical flow field of first frame; Utilize the improper optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering;
(12) to the calculating of second frame to the optical flow field vector of frame second from the bottom: a back frame of present frame and present frame calculates the optical flow field vector of all picture elements of optical flow field of present frame; Obtain the ratio that all optical flow field vectors of present frame surpass the optical flow field vector of this threshold value according to first threshold, again this ratio and predefined second threshold value are compared, to judge whether present frame is boundary frame, if not, then utilize the improper optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering; If then abandon described optical flow field, and recomputate optical flow field, and utilize the improper optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering by present frame and former frame;
(13) to the calculating of last frame optical flow field: last frame and former frame calculate optical flow field, and utilize the optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering;
2) be proportional to the absolute value of optical flow field vector according to parallax value, by the parallax value that is not obtained corresponding point in the optical flow field of present frame by the optical flow field vector of filtering; Utilized interpolation to obtain by the parallax value of the pairing point of optical flow field vector of filtering in this present frame, the parallax value that present frame is had a few is formed complete disparity map;
3) obtain new view with present frame, present frame and described disparity map are synthetic, as the left and right view of present frame;
4) with the synthetic stereo video frame that obtains of the left and right view of each frame;
5) all stereo video frame are formed stereoscopic video sequence.
Characteristics effect of the present invention:
Method of the present invention is the disparity map that obtains each frame by the optical flow field of former each frame of sequence (optical flow), can obtain new right view sequence by each frame and the disparity map thereof of former sequence.This method has solved the problem that prior art exists, and simple, obtains good compromise between method complexity and effect.
Though this method can not perfectly be applicable to various situations, can obtain average effect preferably.
Embodiment
The method based on the converting plane video into stereoscopic video of optical flow field that the present invention proposes is described in detail as follows in conjunction with the embodiments:
The inventive method may further comprise the steps:
1) optical flow field of given each frame of planar video sequence of calculating specifically comprises:
11) to the calculating of the first frame optical flow field: first frame and second frame calculate the optical flow field vector of each picture element of the optical flow field of first frame
(vector that optical flow field is every is represented the moving displacement vector of corresponding picture element from first frame to next frame, and its absolute value is unit with the number of pixels, and the concrete computational methods of optical flow field are very ripe, are not described in detail in this);
Utilize first threshold T
1The optical flow field vector that surpasses first threshold in this optical flow field of filtering, the error that several problems cause below avoiding may occurring in the calculating of optical flow field:
The problem of mistake in computation: the optical flow field vector value of certain point should be 0 or near 0 originally, but because mistake in computation may draw a paranormal very big value, the optical flow field vector value of these points is improper optical flow field vector.
Occlusion issue a: point that in the M frame, exists, because the variation at visual angle may disappear in the M+1 frame, such point can't calculate light stream, being reflected on the result is exactly that the value calculated is very big, and the optical flow field vector value of such point is improper optical flow field vector.
First threshold T
1Be used to remove the improper optical flow field vector that above-mentioned situation occurs, concrete value is: determine (needing the abundant different video of test) by abundant test in advance, by the size of continuous adjustment T1, choosing it has the value of filter effect preferably, and the present invention obtains T by experiment
1Be 80-120.Embodiment can choose T
1Be 100, promptly absolute value can be by filtering above the optical flow field vector of 100 length in pixels.
(12) to the calculating of second frame to the optical flow field vector of frame second from the bottom: a back frame of present frame and present frame calculates the optical flow field vector of all picture elements of optical flow field of present frame; According to first threshold T
1Obtain the ratio that all optical flow field vectors of present frame surpass the optical flow field vector of this threshold value, again with this ratio and the predefined second threshold value T
2Whether compare, be boundary frame to judge present frame, if not, then utilize the improper optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering; If then abandon described optical flow field, and recomputate optical flow field, and utilize the improper optical flow field vector that surpasses first threshold in this optical flow field of first threshold filtering by present frame and former frame;
Above-mentioned boundary frame is meant the frame of the position that is in scene sudden change.Because can take the photograph camera lens is transferred to another scene suddenly by a scene situation, to suppose whole video is divided into some fragments, each fragment is corresponding to a scene.(be made as the A section) in same video segment, the variation between the different frame is less, is reflected in optical flow field, and the optical flow field vector value that is exactly the great majority point is also little; And the frame between fragment and the fragment, such as the last frame M of Segment A and the first frame M+1 of next fragment (being made as the B section), between difference should be very big, be reflected in the optical flow field, the vector value that is exactly almost every bit is all very big.Therefore for two adjacent frame M and M+1, also can judge whether to be in the boundary of A section and B section by optical flow field; The present invention judges that the method for boundary frame is as follows:
The number that adds up to capable number and row of the pixel of frame M multiplies each other, that is: Width * Hight, and the two-dimensional array Array that then sets up a Width * Hight preserves the optical flow field vector absolute value of each pixel of frame M, adds up among the Array absolute value then greater than T
1Optical flow field vector number Number, obtain the ratio P that this number accounts for the total flow field vector number of frame again:
If P 〉=T
2(T
2Be given proportion threshold value, i.e. second threshold value), think that then M is the last frame of certain video clips, promptly be boundary frame.If M is a boundary frame, then the optical flow field that calculates of itself and M+1 frame is abandoned, recomputate optical flow field with M frame and M-1 frame, do the motion conditions from M to the M-1 frame (opposite) that is of the optical flow field reflection that obtains although it is so, only pay close attention to the motion size but the present invention does not pay close attention to the direction of motion with the real motion direction; The optical flow field that obtains has like this solved the problem of boundary frame calculating optical flow field.
The above-mentioned second threshold value T
2Also be to determine (needing the abundant different video of test) by abundant test in advance, constantly adjust T
2Size judge in the hope of boundary frame being made accurately, the threshold value of choosing is reached does not judge the result who does not also fail to judge by accident.In the present invention by test T
2Can get 0.4-0.6, embodiment desirable 0.5.
2) be proportional to the absolute value of optical flow field vector according to parallax value, by the parallax value that is not obtained corresponding point in the optical flow field of present frame by the optical flow field vector of filtering; Utilized linear interpolation to obtain by the parallax value of the pairing point of optical flow field vector of filtering in this present frame, the parallax value that present frame is had a few is formed complete disparity map;
The absolute value that parallax value of the present invention is proportional to the optical flow field vector (promptly
) obtain according to following analysis:
Motion in the frame of video is that the relative motion by video camera and real scene causes, can be divided into following four kinds of situations:
First kind of situation is: video camera is static, does not have moving object in the scene, and the optical flow field of frame is zero vector entirely.The value that makes disparity map have a few is 0, and promptly left view and right view are just the same.Satisfy
Second kind of situation is: video camera is static, and moving object is arranged in the scene, and the image section of the motion in the optical flow field of frame has non-zero vector.
For second kind of situation, Yun Dong object often is in from the closer prospect of video camera in general, that is to say that depth value is smaller, and the speed of often in image, moving big more (no matter what direction of motion) from the near more object of video camera, therefore can be similar to and think that depth value is inversely proportional to the absolute value of light stream vector, promptly
Be inversely proportional to depth value by parallax value again, just obtain
The third is a camera motion, does not have moving object in the scene, and in this case, the optical flow field of frame major part is non-zero vector.
For this situation, the video camera of motion is taken static scene fully, will strictness satisfy relational expression:
(this is from intuitively being readily appreciated that, when people's eyes were looked about the environment of on every side static, the translational speed of near more place in people's the visual field was fast more) is inversely proportional to depth value by parallax value again, just obtains
The 4th kind is camera motion, and moving object is arranged in the scene.
This is the most complicated a kind of situation, and camera motion not only, the object of scene the inside also have motion, in this case for static background parts
Hypothesis still set up (with the third situation 3), only the prospect for motion partly is false.
In sum, the present invention proposes
In very large range be reasonably, can realize effect preferably.
According to the above-mentioned criterion of being calculated disparity map by optical flow field: the present invention exists
Concrete enforcement on, determine that the rational proportion coefficient is as follows: disparity map is a gray-scale map, the scope that every bit can assignment is the maximum parallax of 0 to 255,255 expression, 0 expression does not have parallax.If setting the maximum disparity value is D
Max(concrete numerical value can be set according to the stereoeffect of wanting, i.e. D
MaxThe big more stereoeffect that obtains at last is just obvious more, but too big words will allow the people come over dizzy, and the present invention can get 200-230 by test, for example gets 220), and the maximum value of the vector of the optical flow field of certain frame is
Then proportionality coefficient just is
Even the maximum of the vector of optical flow field corresponds to the maximum disparity value of wanting.
Therefore, according to above-mentioned proportionality coefficient, do not obtained the parallax value of corresponding point in the optical flow field of present frame by the optical flow field vector of filtering.
Utilized interpolation to obtain by the parallax value of the pairing point of optical flow field vector of filtering in this present frame, concrete interpolation method is: the parallax value of the parallax value Linear Estimation A of the point by the known parallax value adjacent with an A; All try to achieve as the parallax value of 8 points around the A among the embodiment, ask this parallax value of 8 arithmetic average to obtain the parallax value of A so.
The parallax value that present frame is had a few is formed complete disparity map.
3) with the synthetic new view that obtains of present frame, present frame and described disparity map, as the left and right view of present frame; The value of disparity map every bit is represented the horizontal displacement of this picture element between the view of the left and right sides, disparity map is added on the present frame, in other words each picture element of present frame is done corresponding displacement, just can obtain new view (during concrete operations, for example can be the frame of former sequence as left view, point on the left view is done corresponding displacement left according to disparity map and is just obtained right view, just newly schemes, and left and right sides view has so just been arranged).
4) with the synthetic stereo video frame that obtains of the left and right view of each frame, specific practice is to make that the value of pixel of the odd column of stereo video frame is the value of pixel of the odd column of left view correspondence, and the value of the pixel of the even column of stereo video frame is the value of pixel of the even column of right view correspondence.
All repeat above-mentioned institute in steps for each frame of sequence, until last frame.
5) all stereo video frame are formed stereoscopic video sequence.
Above-mentioned steps 3), 4), 5) concrete grammar all can adopt prior art to realize.