Summary of the invention
The purpose of the present invention is to provide the mobile camera motion object detection method under a kind of strong parallax, this method is combined
The grayscale information and depth information of image, establish depth constraints equation, whether meet depth constraints by the point judged on image
Equation carries out moving object detection, is able to solve under vehicle-mounted hand-held equal mobile platforms, three-dimensional scenic can generate asking for parallax
Topic carries out the detection of real-time high-efficiency to moving target.
In order to solve the above-mentioned technical problem, the present invention proposes the mobile camera motion target detection side under a kind of strong parallax
Method, comprising the following steps:
Step 1: two field pictures are matched, and extract the depth information of two field pictures;Camera is demarcated, is obtained
The inner parameter of camera;
Step 2: position when using camera shooting first frame image converses on first frame image as world coordinate system
The corresponding world coordinate system coordinate of point coordinate;
Step 3: the camera motion between two frames is described with spin matrix R and translation matrix t, is obtained on the second frame image
Point and world coordinate system relationship, to obtain depth constraints equation;
Step 4: the coordinate of all match points on the second frame image and corresponding depth information are substituted into depth constraints side
Journey obtains the optimal solution of the Rt matrix of depth constraints equation;
Step 5: it is multiplied using the picture point coordinate of depth constraints equation calculation with corresponding depth information;By the product
With the result that is estimated using depth constraints equation respectively as two three-dimensional coordinate points, calculate two three-dimensional coordinate points it
Between distance if normalized cumulant is greater than the threshold value of setting, judge that this apart from corresponding point is to move and by range normalization
Target, otherwise, this is background apart from corresponding point.
Further, in step 3, shown in the depth constraints equation such as formula (1):
In formula (1), (u2,v2) indicate front and back two field pictures match point image coordinate, u2、v2Pixel is respectively indicated to exist
Abscissa and ordinate on a later frame image;Zc2The match point for respectively indicating front and back two field pictures is believed relative to the depth of camera
Breath;fx、fy, s, m, n be parameter in camera internal parameter K respectively, wherein fx, fyRespectively with x, the pixel dimension in the direction y
The camera of expression is along x, the focal length in the direction y;S is warp parameters;(m, n) is center point coordinate in camera imaging plane;Xw、Yw、Zw
To put the homogeneous coordinates in corresponding world coordinate system on first frame image;r1,r2...r9And t1,t2,t3It is spin matrix respectively
With the parameter of translation matrix, and spin matrixTranslation matrix
Further, in step 4, the optimal solution for meeting the Rt matrix of depth constraints equation is solved using least square method.
Further, between two three-dimensional coordinate points described in step 5 shown in the calculation such as formula (2) of distance,
In formula (2), Distance (i) is distance between two three-dimensional coordinate points,For using deeply
The product of angular coordinate and corresponding depth information on the second frame image that degree constraint equation calculates;(Zc2u2,Zc2v2,Zc2)
For the product of angular coordinate and corresponding depth information on the second frame image.
Further, normalized threshold is chosen between 0 to 1.
Compared with prior art, the present invention its remarkable advantage is, the present invention utilizes two dimensional image coordinate system to three-dimensional generation
Relationship between boundary's coordinate system according to image depth information and photography geometrical principle, and combines in front and back two field pictures and camera
Portion's parameter proposes depth constraints equation, and carries out moving object detection using depth constraints equation, compared to traditional movement
Camera motion object detection method has the advantage that the moving target detecting method under (1) this strong parallax, due to combining
The grayscale information and depth information of image can be eliminated due to the mobile influence generated to moving object detection of camera, reject view
Difference reduces false alarm rate, is applicable not only to the take photo by plane Deng negligible scene of parallaxes, and in the vehicle for being frequently used for strong parallax environment
It carries, also have good effect in handheld device;(2) realization of object detection method under the new moving camera of one kind is proposed
Depth constraints equation is not only applied in the frame of target detection by journey, this method, but also can be in a variety of depth detectors
It is realized on platform;(3) algorithm is simple, fast speed, lower to hardware platform requirements, and moving object detection rate is high.
Specific embodiment
In conjunction with Fig. 1, mobile camera motion object detection method under the strong parallax of the present invention, comprising the following specific steps
Step 1: original image pretreatment: firstly, matched to the pixel of front and back two field pictures, mature KLT with
Track algorithm is due to having many advantages, such as that speed is fast, precision is high, anti-noise ability is strong, using KLT track algorithm to two frame figure of front and back
As pixel is matched, the image coordinate (u of two field pictures match point is obtained1,v1), (u2,v2), wherein u1、v1It respectively indicates
Abscissa and ordinate of the pixel on previous frame image, u2、v2Respectively indicate abscissa of the pixel on a later frame image
With ordinate;Secondly, opposite using such as Kinect camera, laser radar even depth detector, the match point for obtaining two field pictures
In the depth information Z of camerac1, Zc2, i.e. camera range-to-go;Finally, carrying out camera mark using Zhang Zhengyou camera calibration method
It is fixed, obtain the inner parameter of cameraWherein, fx, fyRespectively indicated with x, the pixel dimension in the direction y
Camera is along x, the focal length in the direction y;S is warp parameters;(m, n) is center point coordinate in camera imaging plane.
Step 2: the point coordinate under world coordinate system solves: using the corresponding camera coordinates system of first frame image as world coordinates
System, according to photography geometrical principle, obtain the point on first frame image to world coordinate system corresponding relationship, as shown in formula (1):
Wherein, (u1,v1, 1) and it is the homogeneous coordinates put on first frame image;(Xw,Yw,Zw, 1) and it is that the corresponding world of the point is sat
Homogeneous coordinates in mark system;Zc1For depth information of this under camera coordinates system;I is unit matrix.Is obtained using formula (1)
Point (u on one frame image1,v1) corresponding world coordinate system coordinate (Xw,Yw,Zw, 1), as shown in formula (2):
Step 3: the foundation of depth constraints equation: since camera itself is moving, when shooting the second frame image, the position of camera
It sets and is changed relative to the position of shooting first frame, described between two field pictures with spin matrix R and translation matrix t
Camera motion, therefore, shown in the relationship such as formula (3) of point and world coordinate system on the second frame image:
Wherein, point (u2,v2) it is point (u1,v1) match point coordinate on the second frame image;Zc2It is the point in Current camera
Depth information under coordinate system, willSubstitution formula (3) (wherein r1,r2...r9And t1,t2,t3Point
It is not the parameter of spin matrix and translation matrix), and it is organized into the form of AX=B, obtain depth constraints equation:
The solution of step 4:Rt matrix: the coordinate of all match points on the second frame image and corresponding depth information are substituted into
Depth constraints equation solves the optimal solution for meeting the Rt matrix of depth constraints equation using least square method.
Step 5: the moving object detection based on depth constraints equation: firstly, using step 4 calculate Rt matrix it is optimal
Solution calculates the value of depth constraints equation right side of the equal sign, i.e., angle point on the second frame image come out using depth constraints equation calculation
The product of coordinate and corresponding depth informationSecondly, by angular coordinate on the second frame image with it is corresponding
Product (the Z of depth informationc2u2,Zc2v2,Zc2) and using depth constraints equation calculation come out productIt sees
Two three-dimensional coordinate points are done, the two three-dimensional coordinate points are substituted into formula (5), calculate the distance between two o'clock:
Finally, Distance (i) normalization is obtained Distance_normal (i) and is judged, if normalized cumulant
Greater than the threshold value of setting, then judge this apart from corresponding point for moving target, conversely, then judge this apart from corresponding point for back
Scape, to realize moving object detection.Normalized threshold is chosen between 0 to 1, according to theory deduction and experiment experience, is chosen
The size of normalized threshold should be positively correlated with camera motion speed, negative moves speed about target about camera frame frequency, negative
Degree.If normalized threshold is chosen excessive, it may appear that the missing inspection of moving target;If normalized threshold is chosen too small, it may appear that false-alarm,
General normalized threshold takes 0.7 or so.
Beneficial effects of the present invention can be further illustrated by following experiment:
For the embodiment of the present invention using Matlab2012b as experiment porch, experiment is 640 × 480 using image size, and frame frequency is
The Xtion three-dimension sensor of 30Hz carries out video capture and extraction of depth information.Make using apart from the closer small vehicle model of camera
For the source of strong parallax, in the case where camera is to left and rotates counterclockwise simultaneously, tank model moves from left to right.Such as
Shown in Fig. 1, specific steps are as follows:
(1), according to the present invention described in step 1, the original image obtained to Xtion three-dimension sensor is pre-processed, false
If moving target can be by Based on Feature Points, firstly, extracting the angular coordinate of two field pictures with Harris algorithm;Secondly, using
KLT track algorithm matches front and back two field pictures angle point, obtains the image coordinate (u of two field pictures match point1,v1), (u2,
v2), in Fig. 2 (a), (d), in (g) shown in white point;Again, the matching of two field pictures is obtained using Xtion three-dimension sensor
Depth information Z of the point relative to camerac1, Zc2, in Fig. 2 (b), (e), shown in (h), color shows that distance is closer more deeply feeling;Most
Afterwards, camera calibration is carried out using Zhang Zhengyou camera calibration method, obtains the inner parameter K of camera.
(2), according to the present invention described in step 2, the match point coordinate under world coordinate system is solved: according to formula (1):
Obtain the corresponding world coordinate system coordinate (X of all matching angle points on first frame imagew,Yw,Zw, 1), such as formula (2)
It is shown:
(3), according to the present invention described in step 3, establish depth constraints equation: angle point and the world on the second frame image are sat
Shown in the relationship such as formula (3) for marking system:
It willSubstitution formula (3), and it is organized into the form of AX=B, obtain depth constraints side
Journey:
(4), according to the present invention described in step 4, the Rt matrix in depth constraints equation is solved: by institute on the second frame image
The coordinate and corresponding depth information for having matching angle point substitute into depth constraints equation, are solved using least square method and meet depth about
The optimal solution of the Rt matrix of Shu Fangcheng.
(5), according to the present invention described in step 5, moving object detection is carried out using depth constraints equation: firstly, utilizing formula
(4) angular coordinate that obtains using depth constraints equation and the product of corresponding depth information are calculatedIts
It is secondary, by the product (Z of angular coordinate and corresponding depth information on the second frame imagec2u2,Zc2v2,Zc2) and utilization depth constraints side
What journey estimatedRegard two three-dimensional coordinate points as, the two three-dimensional coordinate points is substituted into formula (5), meter
Calculate the distance between two o'clock:
Finally, Distance (i) normalization is obtained Distance_normal (i) and is judged, if normalized cumulant
Greater than the threshold value of setting, then judge this apart from corresponding point for moving target, conversely, then judge this apart from corresponding point for back
Scape, to realize moving object detection, moving target is as shown in black color dots in (c) in Fig. 2, (f), (i).Wherein, threshold is normalized
Value is chosen for Th=0.9.(c) in Fig. 2, (f), (i) are respectively the testing result of three frame images, what the point of black indicated to detect
Motion corner point., it is evident that the motion corner point in three frame images all is correctly detected out by the present invention, and strong parallactic angle
Point is not detected among out.
20 frame images are randomly selected, hand labeled goes out moving target angle point, is denoted asWherein t indicates t frame;To selection
20 frame images out carry out moving object detection, and the angle point for the moving target that every frame calculates is denoted asIt defines D (t)
Indicate the verification and measurement ratio of t frame:
Defining P (t) indicates the accurate rate of t frame:
Wherein, N indicates the number at set midpoint.The verification and measurement ratio that algorithm is described with formula (6), that is, what is detected is movement mesh
Target angle point accounts for the ratio of real motion Corner;The accurate rate that algorithm is described with formula (7), that is, what is detected is movement mesh
Target angle point accounts for the ratio (being inversely proportional to false alarm rate) of all angle points that detected.The higher expression of the value of verification and measurement ratio and accurate rate should
The performance of algorithm is better.The method of the present invention and epipolar-line constraint algorithm and homography bounding algorithm are compared, by 20 frame images
Algorithm verification and measurement ratio and algorithm accurate rate are depicted as line chart, as shown in Figure 3, Figure 4.From figure 3, it can be seen that the method for the present invention
Verification and measurement ratio is apparently higher than epipolar-line constraint algorithm, slightly above homography bounding algorithm, and apparent missing inspection occurs in epipolar-line constraint algorithm;
Figure 4, it is seen that the accurate rate of the method for the present invention is apparently higher than homography bounding algorithm, slightly above epipolar-line constraint algorithm,
There is apparent false-alarm in homography algorithm.The method of the present invention has good performance on verification and measurement ratio and accurate rate.
In conclusion the grayscale information and depth information of present invention combination image, establish depth constraints equation, pass through judgement
Whether the point on image meets depth constraints equation to carry out moving object detection.Experiment show this method not only verification and measurement ratio compared with
Height, and the depth constraints equation proposed can effectively remove the influence because of the mobile parallax generated of camera to target detection,
It can be not only used for that parallax is negligible to take photo by plane, monitor, and also have for the equipment such as vehicle-mounted, hand-held very strong practical
Property.