CN103761725B - A kind of video plane detection method based on innovatory algorithm - Google Patents

Abstract

The present invention relates to a kind of video plane extracting method based on innovatory algorithm, the method, first to given video sequence, carries out the tracking of characteristic point, the feature point trajectory that record obtains by standard K LT method；Judge whether track belongs to the plane corresponding to current homography matrix by calculating the re-projection error between consecutive frame, carry out tracking and the extraction of the plane to whole video sequence；The areal model obtained and they corresponding lists are utilized to gather, it is possible to video to be split obtain final plane and extracts result.The present invention directly can extract plane from video sequence；And select track as operation object, it is ensured that plane extracts the global coherency of result, and is greatly accelerated speed and the accuracy of test convergence.

Description

A kind of video plane detection method based on innovatory algorithm

Technical field

The present invention relates to a kind of video plane detection method based on innovatory algorithm, be specifically related to the video sequence plane extracting method that a kind of distinguished point based is followed the tracks of, belong to image processing field.

Background technology

Plane is extracted and is always an important research direction in computer vision and graphics field.Owing to planar structure is one of class basis geometric properties most commonly seen in reality scene, wherein contain a large amount of two-dimensional structure information of object, and the point being pointed in plane has the strongest geometrical constraint.In a lot of actual application, as built scene space, barrier or target object model etc., the planar structure being required for extracting in scene in advance carries out processing or alternately.nullThe work of this respect has had a lot，Such as based on J-Linkage cluster space plane extracts (Toldo R,Fusiello A.Robust multiple structures estimation with j-linkage.Computer Vision–ECCV2008.Springer Berlin Heidelberg,2008:537-547) and plane based on lines matching extract (Kim H,Lee S.Multiple planar region extraction based on the coplanar line pairs.Robotics and Automation(ICRA),2011IEEE International Conference on.IEEE,2011:2059-2064).Above plane extraction work can generate more attractive result, but due to these methods or need to recover in advance three-dimensional information and the camera parameters of scene, this process amount of calculation is very big, iff the planar structure needed in extraction image or video, this kind of algorithm cost prohibitive；Use affine transformation set up image between transformation relation model, and have ignored nonlinear effect that may be present in conversion process.Therefore it is difficult to from video sequence, extract planar structure rapidly and accurately by these traditional plane extraction algorithms.

Summary of the invention

The problem being difficult to extract from video sequence rapidly and accurately planar structure in order to solve conventional planar extracting method, the present invention proposes a kind of video plane extracting method based on innovatory algorithm, using the teaching of the invention it is possible to provide the preferably plane extraction effect of video sequence.The method needn't recover video camera three-dimensional information, directly by setting up the projective transformation model under epipolar-line constraint between video consecutive frame, calculates the list induced by plane and answers, and each list should correspond to a plane in scene；Then split according to each frame of the list reply video calculated, extract the planar structure in video.

Concrete implementation process of the present invention is as follows:

A kind of video plane extracting method based on innovatory algorithm, the method comprises the steps:

Step one, video read；

Read input video, become picture format to record by frame solution video；

Step 2, feature point tracking；

The characteristic point in KLT algorithm frame each to video is used to extract and follow the tracks of, the feature point trajectory crossing over whole video sequence that record obtains；

Step 3, plane track and extract；

Video sequence for the N frame that step one reads, it is assumed that step 2 obtains the collection of M feature point trajectory and is combined intoFor each track T_j, make p_jAnd q_j(1≤p_j<q_j≤ N) represent its starting and ending frame number, track T respectively_jIt is expressed asForm, whereinRepresent jth characteristic point homogeneous coordinates in the i-th frame；T is share for crossing over the collection of all tracks of a frame and b frame^ab={T_j∈T:p_j≤a,q_j>=b} represents；For a tracing pointIf it is positioned in a plane, then its coordinate on the i-th frame and kth frame just can singly be answered H by what this plane induced_ikAssociate, haveIn order to weigh the areal model fitness to a track, to a trackDefine it to kth (p_j≤k≤q_j) re-projection error of frame is as follows:

$e=\underset{i=p_j,i\&NotEqual;k}{\overset{q_j}{\mathrm{\&Sigma;}}}\left(\right||x_j^k-H_{\mathrm{ki}}{x}_j^i{\left|\right|}^2+||x_j^i-{H_{\mathrm{ki}}}^{-1}{x}_j^k{\left|\right|}^2)$

According to re-projection error, all of track being divided into different set, each set correspond to a plane, specifically can be divided into two steps:

(1) stochastical sampling: for the video sequence read in step oneConstruct N-1 to adjacent frames to C={ (F₁,F₂),(F₂,F₃),...,(F_N-1,F_N)}；First from C, any frame is chosen to (F_i-1,F_i), take the track T crossing over this two frame^(i-1)iAt F_i-1Interior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a Delaunay triangle_tri, then at T_triInside take any one triangle t, use three trajectory calculation corresponding to t to obtain F_i-1And F_iBetween a homography matrix H_(i-1)i, computing formula is as follows:

Known basis matrix F and three groups of picture point are correspondingWhat then 3D point place plane was induced singly should be:

H=A-e'(M^-1b)^T

Wherein A=[e']_×F, e' are the antipodal points on the second width image, have $\left\{\begin{array}{c}\mathrm{Fe}=0\\ F^{T}e\&prime;=0\end{array}\right.;$ [e']_×Represent the antisymmetric matrix of e', it is assumed that e'=(e₁,e₂,e₃)^T, have:

$[e\&prime;{]}_{\&times;}=\left[\begin{array}{ccc}0& {-e}_3& e_2\\ e_2& 0& {-e}_1\\ -e_2& e_1& 0\end{array}\right]$

B is a 3-dimensional vector, and its element is:

b_i=(x_i'×(Ax_i))^T(x_i'×e')/x_i'×e'²

And M is a row vector is3 × 3 matrixes；

A given threshold epsilon, by judging re-projection error e=x_i-H_(i-1)ix_i-1 ²+x_i-1-H_(i-1)i ^-1x_i ²Whether less than ε, by T^(i-1)iInterior track is divided into T_inAnd T_out；

(2) concordance calculates: select and (F_i-1,F_i) adjacent frame is to (F_i,F_i+1), take Tⁱ⁽ⁱ⁺¹⁾In have been labeled as T_inTrack, i.e. T_inITⁱ⁽ⁱ⁺¹⁾At F_iInterior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a new Delaunay triangle_tri；Then, same to previous step, use three trajectory calculation corresponding to t to obtain F_iAnd F_i+1Between homography matrix H_i(i+1), by judging whether re-projection error e is less than threshold epsilon, by Tⁱ⁽ⁱ⁺¹⁾Interior track is also classified into T_inAnd T_out, constantly repeat this step until all frames are the most through processing or T_in∩Tⁱ⁽ⁱ⁺¹⁾Interior trace number is less than 3；

Step (1) and (2) are repeated several times, select T_inPlane that size is maximum and it corresponding toSet, the result output extracted as this secondary flat；

Step 4, plane are split；

Corresponding with them according to the areal model obtained by step 3After set, according toVideo is split to obtain final result, and the present invention selects Graph Cuts algorithm to split video:

The purpose that the N frame video sequence comprising P plane to is split is for each pixel p ∈ F={F₁,F₂,...,F_NSpecify a label f_p∈ 0,1 ..., P-1}, obtain each two field picture belongs to conplane collection of pixels f={p:f_p=i,i∈{0,1,...,P-1}}；Tag set { f_pObtained by one energy function of optimization, first input picture is represented with figure G=(N, E), the node set during wherein N represents figure, each node n_i∈ N is corresponding to a pixel in input picture, and E represents the set on the limit in figure between node, and each limit is corresponding to having the node ＜ n of particular kind of relationship for a pair_p,n_q〉；The optimization objective function of Graph Cuts algorithm is as follows:

$E\left(f\right)=\underset{n\&Element;N}{\mathrm{\&Sigma;}}{D}_n\left(f_n\right)+\underset{n_p<n_q\&Element;N}{\mathrm{\&Sigma;}}{V}_{n_p,n_q}(f_{n_p},f_{n_q})$

Wherein, D_n(f_n) represent the label of node n is set to f_nTime bring data item punishment；n_p＜ n_qRepresenting the unidirectional combination between neighborhood of nodes in figure, " adjacent " here is that four neighborhoods are adjacent or eight neighborhood is adjacent；Represent two neighborhood of nodes ＜ n_p,n_qThe label of ＞ is respectively set toWithTime bring based on spatial continuity smooth item punishment；Label { the f of each node_nObtain by minimizing above-mentioned object function；

Here, data item punishment is defined as follows:

D_n(f_n)=((R(H[f_n]n)-R(n))²

+(G(H[f_n]n)-G(n))²

+(B(H[f_n]n)-B(n))²)^1/2

Wherein, H [f_n] it is f_nThe list that individual plane is induced is answered, and R (), B (), G () represent the RGB component of current pixel point respectively, and smooth item is punished for node n_pIt is adjacent node n_qConnection weights, be defined as follows:

$V_{n_p,n_q}(f_{n_p},f_{n_q})=\mathrm{\&lambda;}{(f_{n_p}-f_{n_q})}^2$

Wherein n_p,n_q∈ N, λ are constant, for the weight between equilibrium criterion penalty term and smooth penalty term, typically take 80～100.

Beneficial effect:

(1) traditional plane extracting method one class needs to recover in advance three-dimensional information and the camera parameters of scene, and this process amount of calculation is very big, iff needing to extract the planar structure in image or video, and this kind of method cost prohibitive；And another kind of algorithm generally use affine transformation to set up image between transformation relation model, and have ignored nonlinear effect that may be present in conversion process, this is often not accurate enough.In addition, current existing Equations of The Second Kind plane extraction algorithm is substantially and carries out for image, the plane of image is extracted and is typically to carry out between two width views, and the plane of video is extracted except to extract plane between consecutive frame, also to ensure this plane global coherency in whole video, the plane scene structure that i.e. each to video frame finally extracts should be consistent.And the inventive method is by setting up the projective transformation model under epipolar-line constraint between video consecutive frame, video camera three-dimensional information needn't be recovered, directly can extract plane from video sequence；And select track as operation object, it is ensured that plane extracts the global coherency of result.

(2) present invention employs the RANSAC plane extracting method of improvement, take four characteristic points from traditional RANSAC algorithm the most at random and carry out testing different, the present invention utilizes not collinear three points i.e. to can determine that this axiom of plane, the characteristic point traced into carries out Delaunay trigonometric ratio, and each test takes three points on a triangle at random and combines fundamental matrix and calculate.And three points on a triangle belong to conplane probability and are substantially higher in four characteristic points taken at random probability at grade, therefore, inventive algorithm is not only greatly accelerated the speed of test convergence, and accuracy is higher.

Detailed description of the invention

Below the embodiment of the inventive method is elaborated.

A kind of video plane extracting method based on innovatory algorithm, the method, first to given video sequence, carries out the tracking of characteristic point, the feature point trajectory that record obtains by standard K LT method；Judge whether track belongs to the plane corresponding to current homography matrix by calculating the re-projection error between consecutive frame, carry out tracking and the extraction of the plane to whole video sequence；The areal model obtained and they corresponding lists are utilized to gather, it is possible to video to be split obtain final plane and extracts result.

The inventive method to implement process as follows:

Step one, video read；

Read input video, become picture format to record by frame solution video；

Step 2, feature point tracking；

Use standard K LT algorithm (Shi J, Tomasi C.Good features to track.Computer Vision and Pattern Recognition, 1994.Proceedings CVPR'94., 1994) characteristic point in frame each to video is extracted and is followed the tracks of, the feature point trajectory crossing over whole video sequence that record obtains；

Step 3, plane track and extract；

Video sequence for the N frame that step one reads, it is assumed that step 2 obtains the collection of M feature point trajectory and is combined intoFor each track T_j, make p_jAnd q_j(1≤p_j<q_j≤ N) represent its starting and ending frame number respectively, then track T_jJust can be expressed asForm, whereinRepresent jth characteristic point homogeneous coordinates in the i-th frame.T can be used for crossing over the set of all tracks of a frame and b frame^ab={T_j∈T:p_j≤a,q_j>=b} represents.For a tracing pointIf it is positioned in a plane, then its coordinate on the i-th frame and kth frame just can singly be answered H by what this plane induced_ikAssociate, haveThe present invention uses re-projection error to weigh the areal model fitness to a track.To a trackDefine it to kth (p_j≤k≤q_j) re-projection error of frame is as follows:

$e=\underset{i=p_j,i\&NotEqual;k}{\overset{q_j}{\mathrm{\&Sigma;}}}\left(\right||x_j^k-H_{\mathrm{ki}}{x}_j^i{\left|\right|}^2+||x_j^i-{H_{\mathrm{ki}}}^{-1}{x}_j^k{\left|\right|}^2)$

The final purpose of inventive algorithm is exactly, according to re-projection error, all of track is divided into different set, and each set correspond to a plane.Specifically can be divided into two steps:

1. stochastical sampling: for the video sequence read in step oneN-1 can be constructed to adjacent frames to C={ (F₁,F₂),(F₂,F₃),...,(F_N-1,F_N)}.First from C, any frame is chosen to (F_i-1,F_i), we typically take (F here₀,F₁), take the track T crossing over this two frame^(i-1)iAt F_i-1Interior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a Delaunay triangle_tri.Then, at T_triInside take any one triangle t, use three tracks corresponding to t just can be calculated F_i-1And F_iBetween a homography matrix H_(i-1)i.Computing formula is as follows:

Known basis matrix F and three groups of picture point are correspondingWhat then 3D point place plane was induced singly should be:

H=A-e'(M^-1b)^T

Wherein A=[e']_×F, e' are the antipodal points on the second width image, have $\left\{\begin{array}{c}\mathrm{Fe}=0\\ F^{T}e\&prime;=0\end{array}\right..$ [e']_×Represent the antisymmetric matrix of e', it is assumed that e'=(e₁,e₂,e₃)^T, have:

$[e\&prime;{]}_{\&times;}=\left[\begin{array}{ccc}0& {-e}_3& e_2\\ e_2& 0& {-e}_1\\ -e_2& e_1& 0\end{array}\right]$

B is a 3-dimensional vector, and its element is:

b_i=(x_i'×(Ax_i))^T(x_i'×e')/x_i'×e'²

And M is a row vector is3 × 3 matrixes.

A given threshold epsilon, we take ε=4 here, by judging re-projection error e=x_i-H_(i-1)ix_i-1 ²+x_i-1-H(_i-1)i ^-1x_i ²Whether less than ε, can be by T^(i-1)iInterior track is divided into T_inAnd T_out。

2. concordance calculates: select and (F_i-1,F_i) adjacent frame is to (F_i,F_i+1), take Tⁱ⁽ⁱ⁺¹⁾In have been labeled as T_inTrack, i.e. T_in∩Tⁱ⁽ⁱ⁺¹⁾At F_iInterior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a new Delaunay triangle_tri.Then, same to previous step, use three trajectory calculation corresponding to t to obtain F_iAnd F_i+1Between homography matrix H_i(i+1), by judging that re-projection error e, can be by T whether less than threshold epsilonⁱ⁽ⁱ⁺¹⁾Interior track is also classified into T_inAnd T_out.Constantly repeat this step until all frames are the most through processing or T_in∩Tⁱ⁽ⁱ⁺¹⁾Interior trace number is less than 3.

Step 1 and 2 is repeated several times, selects T_inPlane that size is maximum and it corresponding toSet, the result output extracted as this secondary flat.

Step 4, plane are split；

Corresponding with them according to the areal model obtained by step 3After set, it is possible to according toSplit to obtain final result to video.Here have selected Graph Cuts algorithm (Narayanan P J, Vineet V, Stich T.Fast Graph Cuts for Computer.GPU Computing Gems Emerald Edition, 2011:439) video is split.

The purpose that the N frame video sequence comprising P plane to is split is for each pixel p ∈ F={F₁,F₂,...,F_NSpecify a label f_p∈ 0,1 ..., P-1}, obtain each two field picture belongs to conplane collection of pixels f={p:f_p=i,i∈{0,1,...,P-1}}.Tag set { f_pObtain typically by one energy function of optimization.First, input picture is schemed G=(N with one, E) represent, node set during wherein N represents figure, each node n i ∈ N is corresponding to a pixel in input picture, E represents the set on the limit in figure between node, and each limit is corresponding to having the node ＜ n of particular kind of relationship for a pair_p,n_q〉.The optimization objective function of Graph Cuts algorithm is as follows:

$E\left(f\right)=\underset{n\&Element;N}{\mathrm{\&Sigma;}}{D}_n\left(f_n\right)+\underset{n_p<n_q\&Element;N}{\mathrm{\&Sigma;}}{V}_{n_p,n_q}(f_{n_p},f_{n_q})$

Wherein, D_n(f_n) represent the label of node n is set to f_nTime bring data item punishment.n_p＜ n_qRepresenting the unidirectional combination between neighborhood of nodes in figure, " adjacent " here can be that four neighborhoods are adjacent or eight neighborhood is adjacent；Represent two neighborhood of nodes ＜ n_p,n_qThe label of ＞ is respectively set toWithTime bring based on spatial continuity smooth item punishment.Label { the f of each node_nObtain by minimizing above-mentioned object function.

Here, data item punishment is defined as follows:

D_n(f_n)=((R(H[f_n]n)-R(n))²

+(G(H[f_n]n)-G(n))²

+(B(H[f_n]n)-B(n))²)^1/2

Wherein, H [f_n] it is f_nThe list that individual plane is induced is answered, and R (), B (), G () represent the RGB component of current pixel point respectively.Smooth item is punished for node n_pIt is adjacent node n_qConnection weights, be defined as follows:

$V_{n_p,n_q}(f_{n_p},f_{n_q})=\mathrm{\&lambda;}{(f_{n_p}-f_{n_q})}^2$

Wherein n_p,n_q∈ N, λ are constant, and for the weight between equilibrium criterion penalty term and smooth penalty term, we take λ=80 here.

Obviously, those skilled in the art can carry out various change and modification without departing from the spirit and scope of the present invention to the present invention.So, if these amendments of the present invention and modification belong within the scope of the claims in the present invention and equivalent technologies thereof, then the present invention is also intended to comprise these change and modification.

Claims (3)

1. a video plane extracting method based on innovatory algorithm, the method comprises the steps:

Step one, video read；

Read input video, video is resolved into picture format by frame and records；

Step 2, feature point tracking；

The characteristic point in KLT algorithm frame each to video is used to extract and follow the tracks of, the feature point trajectory crossing over whole video sequence that record obtains；

Step 3, plane track and extract；

Video sequence for the N frame that step one reads, it is assumed that step 2 obtains the collection of M feature point trajectory and is combined intoFor each track T_j, make p_jAnd q_j, 1≤p_j<q_j≤ N, represents its starting and ending frame number, track T respectively_jIt is expressed asForm, whereinRepresent jth characteristic point homogeneous coordinates in the i-th frame；For crossing over a frame andThe collection of all tracks of frame shareRepresent；For a tracing pointIf it is positioned in a plane, then its coordinate on the i-th frame and kth frame just can singly be answered H by what this plane induced_ikAssociate, haveIn order to weigh the areal model fitness to a track, to a trackDefine it as follows to the re-projection error of kth frame, wherein p_j≤k≤q_j:

According to re-projection error, all of track being divided into different set, each set correspond to a plane, specifically can be divided into two steps:

(1) stochastical sampling: for the video sequence read in step oneConstruct N-1 to adjacent frames to C={ (F₁,F₂),(F₂,F₃),...,(F_N-1,F_N)}；First from C, any frame is chosen to (F_i-1,F_i), take the track T crossing over this two frame^(i-1)iAt F_i-1Interior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a Delaunay triangle_tri, then at T_triInside take any one triangle t, use three trajectory calculation corresponding to t to obtain F_i-1And F_iBetween a homography matrix H_(i-1)i, computing formula is as follows:

Known basis matrix F and three groups of picture point are correspondingα i=1,2,3, then what 3D point place plane was induced singly should be:

H=A-e'(M^-1b)^T

Wherein A=[e']_×F, e' are the antipodal points on the second width image, have[e']_×Represent the antisymmetric matrix of e', it is assumed that e'=(e₁,e₂,e₃)^T, have:

B is a 3-dimensional vector, and its element is:

b_i=(x_i'×(Ax_i))^T(x_i'×e')/||x_i'×e'||²

And M is a row vector is3 × 3 matrixes；

A given threshold epsilon, by judging re-projection errorWhether less than ε, by T^(i-1)iInterior track is divided into T_inAnd T_out；

(2) concordance calculates: select and (F_i-1,F_i) adjacent frame is to (F_i,F_i+1), take Tⁱ⁽ⁱ⁺¹⁾In have been labeled as T_inTrack, i.e. T_in∩Tⁱ⁽ⁱ⁺¹⁾At F_iInterior corresponding point carry out Delaunay trigonometric ratio, obtain the set T of a new Delaunay triangle_tri；Then, same to previous step, use three trajectory calculation corresponding to t to obtain F_iAnd F_i+1Between homography matrix H_i(i+1), by judging whether re-projection error e is less than threshold epsilon, by Tⁱ⁽ⁱ⁺¹⁾Interior track is also classified into T_inAnd T_out, constantly repeat this step until all frames are the most through processing or T_in∩Tⁱ⁽ⁱ⁺¹⁾Interior trace number is less than 3；

Repeat step (1) and (2), select T_inPlane that size is maximum and it corresponding toSet, the result output extracted as this secondary flat；

Step 4, plane are split；

Corresponding with them according to the areal model obtained by step 3After set, according toVideo is split to obtain final result, selects Graph Cuts algorithm that video is split:

The purpose that the N frame video sequence comprising P plane to is split is for each pixel q ∈ F={F₁,F₂,...,F_NSpecify a label f_p∈ 0,1 ..., P-1}, obtain each two field picture belongs to conplane collection of pixels f={q:f_p∈{0,1,...,P-1}}；Tag set { f_pObtained by one energy function of optimization, first input picture is represented with figure G=(R, E), the node set during wherein R represents figure, each node n_{β i}∈ R is corresponding to a pixel in input picture, and E represents the set on the limit in figure between node, and each limit is corresponding to having the node < n of particular kind of relationship for a pair_p,n_q>；The optimization objective function of Graph Cuts algorithm is as follows:

Wherein, D_n(f_n) represent the label of node n is set to f_nTime bring data item punishment；n_p<n_qRepresenting the unidirectional combination between neighborhood of nodes in figure, " adjacent " here is that four neighborhoods are adjacent or eight neighborhood is adjacent；Represent two neighborhood of nodes < n_p,n_q> label be respectively set toWithTime bring based on spatial continuity smooth item punishment；Label { the f of each node_nObtain by minimizing above-mentioned object function；

Here, data item punishment is defined as follows:

D_n(f_n)=((R (H [f_n]n)-R(n))²

+(G(H[f_n]n)-G(n))²

+(B(H[f_n]n)-B(n))²)^1/2

Wherein, H [f_n] it is f_nThe list that individual plane is induced is answered, and R (), B (), G () represent the RGB component of current pixel point respectively, and smooth item is punished for node n_pIt is adjacent node n_qConnection weights, be defined as follows:

Wherein n_p,n_q∈ R, λ are constant, for the weight between equilibrium criterion penalty term and smooth penalty term.

A kind of video plane extracting method based on innovatory algorithm, it is characterised in that: in step 3, threshold epsilon=4.

A kind of video plane extracting method based on innovatory algorithm, it is characterised in that: in step 4, weights λ=80.