CN105711501A - Car look-around camera-based car monitoring method and system in dead zone - Google Patents

Abstract

The invention is applicable to the field of car monitoring, and provides a car look-around camera-based car monitoring method and system in a dead zone. The system comprises a look-around fisheye camera, a processor and a display, wherein the look-around camera is connected with the processor, and the processor is connected with the display in a bi-direction communication manner. By adopting the method and system, the safety potential hazard at the driving links such as lane changing and doubling of a car due to the insufficient of view field of a car rearview mirror and the dead zones of the visual observation always presenting at two sides of the car can be solved. The view angle difference caused by the imaging of the car at different positions in the dead zone can realize a relatively good detection effect; and meanwhile, the method and the system also have the advantages that a large area of dead zone can be detected, the detection result is more reliable, and the detection speed is relatively high.

Description

Vehicle monitoring method and system in a kind of blind area looking around camera based on automobile

Technical field

The invention belongs to automobile monitoring field, particularly relate to vehicle monitoring method and system in a kind of blind area looking around camera based on automobile.

Background technology

In the process that vehicle travels, driver observes the traffic conditions of vehicle side and rear flank side by the rearview mirror of front window both sides.Owing to the visual field of vehicle mirrors is inadequate, can there is the blind area of visualization in vehicle both sides all the time, brings potential safety hazard to the driving link such as vehicle lane change, doubling.In order to eliminate the vehicle blind zone impact on vehicle security drive, currently mainly there is following several blind areas monitoring method.

The one blind area monitoring system being based on ultrasound probe, the echo-signal of barrier in search coverage is delivered to controller by ultrasound probe, and controller calculates the distance of obstacle distance vehicle and sends alarm signal.This product structure is simple, and cost is low, less expensive material benefit.But the usual angle changing rate of signal that ultrasound probe is launched is narrow, it is necessary to expand supervision scope by installing multiple ultrasound probe；The air line distance that can detect is nearer, it is impossible to the region of monitoring certain distance.Owing to being based on echo-signal judges whether it is barrier, therefore cannot be distinguished by isolation strip, limit, the road such as guardrail and trees structure, be also highly susceptible in air the impact of floating body, it is impossible to accurately accomplish only vehicle in blind area to be made warning.

The two blind area monitoring systems being based on microwave or laser ranging technique, similar with the blind area monitoring system based on ultrasound probe, it is also based on echo-signal and calculates the distance with barrier and send alarm signal.Compared with the blind area monitoring system popped one's head in based on overshot ripple, the advantage of this product is to have bigger straight line detection range, but its price also can be high a lot.Owing to being also based on echo-signal judges whether it is barrier, the impact of floating body in limit structure and air can be subject to equally, it is impossible to accurately accomplish only vehicle in blind area to be made warning.

The three blind area monitoring systems being based on photographic head, the fade chart picture that photographic head shooting obtains is sent to controller, and controller passes through pattern recognition scheduling algorithm identification vehicle, and sends alarm signal.Compared with first two product, the blind area monitoring system based on photographic head has bigger motility, it is possible to identify different objects as required, and certainly corresponding algorithm also can be more complicated.What be currently known is in that owing to blind area is generally relatively larger based on the shortcoming of vehicle monitoring method in the blind area of photographic head, and in the image captured, vehicle diverse location in blind area can present bigger visual angle change, makes the difficulty of identification be greatly increased.Additionally, various types of automobile, such as car, MPV (Multi-PurposeVehicles), SUV (SportUtilityVehicle), bus, there is bigger difference due to its size in truck etc., the factors such as aspect ratio change is very big, also can increase the difficulty of identification.

Summary of the invention

It is an object of the invention to provide vehicle monitoring method and system in a kind of blind area looking around camera based on automobile, aim to solve the problem that the vehicle visual angle difference that diverse location imaging causes in blind area, realize good Detection results, the visual field of vehicle mirrors is inadequate, all the time can there is the blind area of visualization in vehicle both sides, to the problem that the driving link such as vehicle lane change, doubling brings potential safety hazard.

The present invention is achieved in that and includes looking around fisheye camera, processor and display unit, said method comprising the steps of:

A, look around fisheye camera catch original fish eye images, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, it is designated as Y-direction vertically upward, passes through equation $\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$ $\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2(r^2+2r^{\′2})\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1(r^2+2y^{\′2})\end{array}\right.$ It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model；

B, carry out rotation and obtain virtual camera looking around fisheye camera, pass through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame；

C, according to perspective projection ultimate principle, obtained the Intrinsic Matrix of virtual camera and outer parameter matrix by equation s ' p=K (RP+t), only consider ground, make Y=0, pass through equationGenerate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, S represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction；

D, according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then makes, make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection；

E, the monitored area image ROI ' that will represent with tire centerline point, pass through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object；

F, to the element in the array A of monitoring point, (u, the detection window represented by v) is (u, v, M, N), input detection window and side elevation image, adopts known technology to detect, and whether finally export is target object.

The further technical scheme of the present invention is: further comprise the steps of: in described step A

A1, obtain looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.

The further technical scheme of the present invention is: further comprise the steps of: in described step B

B1, rotate and look around fisheye camera and obtain an optical axis and detect the vertical virtual camera of plane, can be eliminated the image after perspective distortion by virtual camera；

B2, in the normal driving process of vehicle, virtual video camera is pointed on the left of Current vehicle or right side, the undistorted tire image of side back car can be obtained.

The further technical scheme of the present invention is: further comprise the steps of: in described step F

F1, detection window is carried out feature extraction and calculating, then calculated eigenvalue is inputed to neutral net and carry out the judgement of grader.

The further technical scheme of the present invention is: the feature extraction in described step F1 includes Haar rectangular characteristic or LBP feature or HOG feature, and the grader in described step F1 is cascade Weak Classifier or SVM classifier.

Another object of the present invention is to provide vehicle monitoring system in a kind of blind area looking around camera based on automobile, described system includes looking around fisheye camera, processor and display unit, described camera of looking around is connected with described processor, described processor is connected with described display unit both-way communication, it is characterized in that

Described fluoroscopy images module, original fish eye images is caught for looking around fisheye camera, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, is designated as Y-direction vertically upward, passes through equation $\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$ $\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2(r^2+2r^{\′2})\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1(r^2+2y^{\′2})\end{array}\right.$ It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model；

Described side elevation image module, for carrying out rotation and obtain virtual camera looking around fisheye camera, passes through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame；

Described distance matrix module, for according to perspective projection ultimate principle, obtaining the Intrinsic Matrix of virtual camera and outer parameter matrix, only consider ground, make Y=0, pass through equation by equation s ' p=K (RP+t)Generate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, s represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction；

Described monitored area image ROI ' module, for according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection；

Described monitoring point array module, for the monitored area image ROI ' that will represent with tire centerline point, passes through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object；

Described detection output module, for the element in the array A of monitoring point, (u, the detection window represented by v) is (u, v, M, N), inputs detection window and side elevation image, adopts known technology to detect, and whether finally export is target object.

The further technical scheme of the present invention is: also include demarcating unit；Described demarcating module, for obtaining looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.

The further technical scheme of the present invention is: also include the elementary area after eliminating perspective distortion and undistorted tire image unit；Elementary area after described elimination perspective distortion, looks around fisheye camera and obtains an optical axis for rotating and detect the vertical virtual camera of plane, can be eliminated the image after perspective distortion by virtual camera；Described undistorted tire image unit, for, in the normal driving process of vehicle, pointing on the left of Current vehicle or right side, can obtain the undistorted tire image of side back car by virtual video camera.

The further technical scheme of the present invention is: also include judging unit；Described judging unit, for detection window is carried out feature extraction and calculating, then inputs to neutral net by calculated eigenvalue and carries out the judgement of grader.

The further technical scheme of the present invention is: described in look around fisheye camera expansible.

The invention has the beneficial effects as follows: solve the vehicle visual angle difference that diverse location imaging causes in blind area, realize good Detection results, also there is the larger range of blind area of detection simultaneously, more reliable detection result, and the advantage of detection speed, bring huge use to the driving link such as vehicle lane change, doubling simultaneously.

Accompanying drawing explanation

Fig. 1 is vehicle monitoring method flow block diagram in a kind of blind area looking around camera based on automobile that the embodiment of the present invention provides；

Fig. 2 is that the embodiment of the present invention provides vehicle monitoring system block diagram in a kind of blind area looking around camera based on automobile.

Detailed description of the invention

Accompanying drawing labelling:

Fig. 1 illustrates vehicle monitoring method in a kind of blind area looking around camera based on automobile provided by the invention, aim to solve the problem that the vehicle visual angle difference that diverse location imaging causes in blind area, realize good Detection results, the visual field of vehicle mirrors is inadequate, all the time can there is the blind area of visualization in vehicle both sides, to the problem that the driving link such as vehicle lane change, doubling brings potential safety hazard.

The present invention is achieved in that a kind of vehicle monitoring method in blind area looking around camera based on automobile, including looking around fisheye camera, processor and display unit, comprises the following steps according to method flow diagram:

Step S1, look around fisheye camera and catch original fish eye images, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, is designated as Y-direction vertically upward, obtains looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.Pass through equation $\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$ $\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2(r^2+2r^{\′2})\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1(r^2+2y^{\′2})\end{array}\right.$ It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model.

Step S2, carries out rotation obtain virtual camera to looking around fisheye camera, rotates and looks around fisheye camera and obtain the virtual camera that an optical axis is vertical with detecting plane, can be eliminated the image after perspective distortion by virtual camera.In the normal driving process of vehicle, virtual video camera is pointed on the left of Current vehicle or right side, the undistorted tire image of side back car can be obtained.Pass through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame.

Step S3, according to perspective projection ultimate principle, is obtained the Intrinsic Matrix of virtual camera and outer parameter matrix, only considers ground, make Y=0, pass through equation by equation s ' p=K (RP+t)Generate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, s represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction.

Step S4, according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection.

Step S5, the monitored area image ROI ' that will represent with tire centerline point, pass through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object.

Step S6, to the element (u in the array A of monitoring point, v) detection window represented by is (u, v, M, N), input detection window and side elevation image, detection window is carried out feature extraction and calculating, then calculated eigenvalue is inputed to neutral net and carry out the judgement of grader.Feature extraction includes Haar rectangular characteristic or LBP feature or HOG feature.Grader is cascade Weak Classifier or SVM classifier.Whether adopt known technology to detect, finally exporting is target object.

Vehicle monitoring system in a kind of blind area looking around camera based on automobile as shown in Figure 2, described system includes looking around fisheye camera, processor and display unit, described camera of looking around is connected with described processor, and described processor is connected with described display unit both-way communication.

Described fluoroscopy images module, original fish eye images is caught for looking around fisheye camera, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, is designated as Y-direction vertically upward, passes through equation $\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$ $\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2(r^2+2r^{\′2})\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1(r^2+2y^{\′2})\end{array}\right.$ It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model.

Described demarcation unit, for obtaining looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.Also include the image module after eliminating perspective distortion and undistorted tire image module.

Described side elevation image module, for carrying out rotation and obtain virtual camera looking around fisheye camera, passes through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame.

Elementary area after described elimination perspective distortion, looks around fisheye camera and obtains an optical axis for rotating and detect the vertical virtual camera of plane, can be eliminated the image after perspective distortion by virtual camera.

Described undistorted tire image unit, for, in the normal driving process of vehicle, pointing on the left of Current vehicle or right side, can obtain the undistorted tire image of side back car by virtual video camera.

Described distance matrix module, for according to perspective projection ultimate principle, obtaining the Intrinsic Matrix of virtual camera and outer parameter matrix, only consider ground, make Y=0, pass through equation by equation s ' p=K (RP+t)Generate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, S represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction.

Described monitored area image ROI ' module, for according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection.

Described monitoring point array module, for the monitored area image ROI ' that will represent with tire centerline point, passes through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object.

Described detection output module, for the element in the array A of monitoring point, (u, the detection window represented by v) is (u, v, M, N), inputs detection window and side elevation image, adopts known technology to detect, and whether finally export is target object.

Described judging unit, for detection window is carried out feature extraction and calculating, then inputs to neutral net by calculated eigenvalue and carries out the judgement of grader.

It is described that to look around fisheye camera expansible.

Solve the vehicle visual angle difference that diverse location imaging causes in blind area, realize good Detection results, also there is the larger range of blind area of detection, more reliable detection result simultaneously, and the advantage of detection speed, bring huge use to the driving link such as vehicle lane change, doubling simultaneously.

By automobile being looked around the side elevation image that the image of fisheye camera is converted to the perspective camera of horizontally toward side, vehicle detection is converted to the detection of tire, thus eliminating the imaging difference of the different visual angles that object to be detected diverse location in blind area causes, it is achieved detection accuracy and reliability preferably.Native system realizes necessarily depending on 3 cameras, and description below illustrates with the realization of 3 the most comprehensive, monitoring area is maximum cameras, gives the example based on 1 camera or two cameras in embodiment equally.

The original fish eye images respectively F of fisheye camera output is looked around on automobile left side, right side and rear side 3 tunnel₁, F₂And F₃, definition bodywork reference frame is: with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, being vertically designated as forward X-direction, level is designated as to the right Z-direction, is designated as Y-direction vertically upward.The Intrinsic Matrix K of the fisheye camera of left side, right side and rear side is obtained by known Zhang Zhengyou standardizition₁, K₂And K₃, radial distortion parameter and tangential distortion parameter, and camera is relative to the outer ginseng matrix [R of bodywork reference frame₁, t₁], [R₂, t₂] and [R₃, t₃]。

$\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.---\left(1\right)$

$\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2(r^2+2r^{\′2})\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1(r^2+2y^{\′2})\end{array}\right.---\left(2\right)$

$\left\{\begin{array}{c}u=f_x{x}^{\′\′}+u_0\\ v=f_y{y}^{\′\′}+v_0\end{array}\right.---\left(3\right)$

Original fish eye images is corrected to (3) according to equation (1), the fluoroscopy images after being corrected.Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅And k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model.The fish eye images F of left side, right side and rear side₁, F₂And F₃Fluoroscopy images after correction respectively I '₁, I '₂With I '₃；

Rotate looking around flake machine, obtain an optical axis virtual video camera vertical with plane interested.By this virtual video camera, it is possible to the image after the perspective distortion that is eliminated.In the normal driving process of vehicle, side back car is parallel with the travel direction of Current vehicle, and virtual video camera points on the left of Current vehicle or right side, can obtain the undistorted tire image of side back car.When real camera does not have relative translation with virtual camera, real camera and virtual camera can be mapped according to equation (4).

${\mathrm{sp}}_0=R_0{R}_V^{-1}{K}_V^{-1}{p}_V=K_0{R}_{0V}{K}_V^{-1}{p}_V---\left(4\right)$

Wherein p_VAnd p₀The respectively point on virtual camera and real camera image, s is a zoom factor, K_VAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, R_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame.

According to equation (4), by the left-side images I ' after correction₁With image right I '₂It is respectively converted into the side elevation image I in left side₁Side elevation image I with right side₂, the monitored area on corresponding vehicle left side and right side respectively；By the rear image I ' after correction₃Be converted to two side elevation image, wherein left side be designated as I₃, right side be designated as I₄, the monitored area of corresponding vehicle left rear side and right lateral side respectively.Original fish eye images is converted to side image, not only eliminates the distortion of fish eye images, also make the target object originally needing detection various visual angles become single visual angle, improve Detection results and real-time.

According to perspective projection ultimate principle, there is equation (5).Wherein P (XYZ)^TRepresent bodywork reference frame next one three-dimensional point, p (uv1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, s represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix.

S ' p=K (RP+t) (5)

Only consider ground, make Y=0, it is possible to obtain:

$\left[\begin{array}{c}x\\ 0\\ z\end{array}\right]=R^{-1}({\mathrm{sK}}^{-1}p-t)---\left(6\right)$

According to equation (6), the pixel coordinate of input side elevation image, it is possible to calculate the X-direction under s ' and bodywork reference frame and coordinate on Z-direction the two direction (x, z).To 4 tunnel side elevation image I₁, I₂, I₃And I₄Generate the Distance matrix D under corresponding bodywork reference frame₁, D₂, D₃And D₄.Distance matrix and side elevation image have identical resolution, and wherein the value of each point represents the distance in X-direction and Z-direction on ground under bodywork reference frame of the point in corresponding side elevation image.

The sphere of action of blind area monitoring is generally in the side of vehicle and rear flank side, and this scope upright projection on the ground can represent with a rectangular area:

$\left\{\begin{array}{c}{X}^{-}\≤x\≤X^{+}\\ Z^{-}\≤z\≤Z^{+}\end{array}\right.---\left(7\right)$

Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum.

For 4 Distance matrix Ds₁, D₂, D₃And D₄, generate monitored area image respectively, be designated as ROI '₁, ROI '₂, ROI '₃With ROI '₄.The flow process generated is as follows: each point in traversal Distance matrix D, for certain point (u, v), its value is that (x, z), if this value meets inequation (7), then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0.

Monitored area image and distance matrix have identical resolution, and wherein the value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection.By generating monitored area image, only at the region detection target object of monitoring, it is possible to greatly accelerate that follow-up phase carries out the time of object detection.

The rectangular window needing input to represent with top left co-ordinate and width height when generally carrying out feature extraction and target classification, therefore converts the monitored area ROI ' that the following central point with tire minimum enclosed rectangle represents to represent with the top left co-ordinate of this rectangle monitored area ROI here.The detection window making target object is sized to M × N pixel, according to equation (8), according to from top to bottom, order from left to right, the above monitored area ROI ' represented with tire centerline point is changed into the monitored area ROI represented with window top left co-ordinate.

$ROI(u-\frac{M}{2},v-N)=\left\{\begin{array}{cc}{\mathrm{ROI}}^{\′}(u,v),& u\≥\frac{M}{2}andv\≥N\\ 0,& otherwise\end{array}\right.---\left(8\right)$

Simultaneously, monitored area ROI can also be converted into the array A of monitoring point and carry out concrete object detection again, so have only to the array A of traversal monitoring point when object detection, without traveling through whole image and judging in conjunction with monitored area ROI, promote system real time further.The method adopted is: for some ROI, according to from top to bottom, each point of order traversal from left to right, and by the position of point that value is 255, (u, v) joins in array A, the array of 4 monitoring points of composition, is designated as A₁, A₂, A₃And A₄.After obtaining the array of monitoring point, follow-up object detection has only to travel through this array.

To monitoring array A in element (u, the detection window represented by v) is (u, v, M, N), input detection window and side elevation image, wheel detection employing known technology: first detection window is carried out feature extraction and calculating, it is generally adopted Haar rectangular characteristic, LBP feature or HOG feature, then input to neutral net, cascade Weak Classifier by calculated eigenvalue, or the grader such as SVM judges, whether output is target object.The grader used is good generally according to the data training in advance of mark.

Employing is arranged on 3 of automobile left side, right side and rear side and looks around fisheye camera and realize native system.The original fish eye images F of collected by camera, image resolution ratio is 640 × 480 pixels.It is respectively as follows: by demarcating the Intrinsic Matrix obtained

$K_1=\left[\begin{array}{ccc}153.35& 0& 316.10\\ 0& 145.30& 243.06\\ 0& 0& 1\end{array}\right]$

$K_2=\left[\begin{array}{ccc}151.81& 0& 308.76\\ 0& 145.36& 241.74\\ 0& 0& 1\end{array}\right]$

$K_3=\left[\begin{array}{ccc}151.46& 0& 312.20\\ 0& 146.40& 245.72\\ 0& 0& 1\end{array}\right]$

Side elevation image I after conversion, image resolution ratio is 400 × 160 pixels.

Blind area monitoring range is set to:

Side elevation image corresponding to left camera is X-direction [-4m ,-1.5m], Z-direction [-1m, 3m].

Side elevation image corresponding to right camera is X-direction [0.5m, 3.0m], Z-direction [-1m, 3m].

The side elevation image in the left side that rear side camera is corresponding is X-direction [-4m ,-1.5m], Z-direction [-4m, 0m].

The side elevation image on the right side that rear side camera is corresponding is X-direction [0.5m, 3.0m], Z-direction [-4m, 0m].

By the determined monitored area ROI of above-mentioned blind area monitoring range.

The known cascade classifier based on Haar rectangular characteristic will be adopted when side view carries out tire checking, can be seen that testing result, it is arranged on the camera of automobile side and side elevation image that camera below obtains is complementary on detection region, and have certain overlapping region.In the image that the camera of side obtains, from rearview mirror to the tailstock, this region is more visible, starts to fog from tailstock region sternward again；In the image that camera below obtains more visible from the tailstock 3 to 5 meters of these regions backward, then start to fog backward.The advantage adopting 3 fisheye cameras is to have bigger monitoring range, and there is certain overlapping region blind area, left and right, makes detection more reliable.

Native system is realized only with two fisheye cameras being arranged on automobile left side and right side.The blind area in left camera left side corresponding to the side elevation image difference after the conversion of right camera and the blind area on right side.It is identical that other flow processs look around fisheye camera with installation 3.

Native system is realized only with the fisheye camera being arranged on automotive back.The fisheye camera of rear side can be simultaneously converted into the side elevation image dough-making powder side elevation image to the right towards left side, respectively the blind area of the blind area of corresponding left rear side and right lateral side.It is identical that other flow processs look around fisheye camera with installation 3.

The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all any amendment, equivalent replacement and improvement etc. made within the spirit and principles in the present invention, should be included within protection scope of the present invention.

Claims (10)

1. vehicle monitoring method in the blind area looking around camera based on automobile, it is characterised in that include looking around fisheye camera, processor and display unit, said method comprising the steps of:

A, look around fisheye camera catch original fish eye images, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, it is designated as Y-direction vertically upward, passes through equation

$\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$

$\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2\left(r^2+2r^{\′2}\right)\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1\left(r^2+2y^{\′2}\right)\end{array}\right.$

It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model；

B, carry out rotation and obtain virtual camera looking around fisheye camera, pass through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame；

C, according to perspective projection ultimate principle, obtained the Intrinsic Matrix of virtual camera and outer parameter matrix by equation s ' p=K (RP+t), only consider ground, make Y=0, pass through equationGenerate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, S represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction；

D, according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection；

E, the monitored area image ROI ' that will represent with tire centerline point, pass through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object；

F, to the element in the array A of monitoring point, (u, the detection window represented by v) is (u, v, M, N), input detection window and side elevation image, adopts known technology to detect, and whether finally export is target object.

2. vehicle monitoring method in blind area according to claim 1, it is characterised in that further comprise the steps of: in described step A

A1, obtain looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.

3. vehicle monitoring method in blind area according to claim 1, it is characterised in that further comprise the steps of: in described step B

B1, rotate and look around fisheye camera and obtain an optical axis and detect the vertical virtual camera of plane, can be eliminated the image after perspective distortion by virtual camera；

B2, in the normal driving process of vehicle, virtual video camera is pointed on the left of Current vehicle or right side, the undistorted tire image of side back car can be obtained.

4. vehicle monitoring method in blind area according to claim 1, it is characterised in that further comprise the steps of: in described step F

F1, detection window is carried out feature extraction and calculating, then calculated eigenvalue is inputed to neutral net and carry out the judgement of grader.

5. vehicle monitoring method in blind area according to claim 4, it is characterised in that the feature extraction in described step F1 includes Haar rectangular characteristic or LBP feature or HOG feature, the grader in described step F1 is cascade Weak Classifier or SVM classifier.

6. vehicle monitoring system in the blind area looking around camera based on automobile, described system includes looking around fisheye camera, processor and display unit, described in look around camera and be connected with described processor, described processor is connected with described display unit both-way communication, it is characterized in that

Described fluoroscopy images module, original fish eye images is caught for looking around fisheye camera, set up bodywork reference frame, with the following central point of the minimum enclosed rectangle of automobile upright projection on the ground for bodywork reference frame initial point, vertically it is designated as forward X-direction, level is designated as to the right Z-direction, is designated as Y-direction vertically upward, passes through equation

$\left\{\begin{array}{c}{x}^{\′}=\frac{u^{\′}-u_0}{f_x}\\ y^{\′}=\frac{v^{\′}-v_0}{f_y}\end{array}\right.$

$\left\{\begin{array}{c}{x}^{\′\′}=x^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\′}{y}^{\′}+p_2\left(r^2+2r^{\′2}\right)\\ y^{\′\′}=y^{\′}\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_2{x}^{\′}{y}^{\′}+p_1\left(r^2+2y^{\′2}\right)\end{array}\right.$

It is corrected, the fluoroscopy images after being corrected；Wherein (u ', v ') is the pixel coordinate on the fluoroscopy images after correction, (u₀, v₀) for principal point pixel coordinate, f_xAnd f_yThe respectively focal length in image level direction and vertical direction, k₁, k₂, k₃, k₄, k₅, k₆For radial distortion parameter, p₁And p₂For tangential distortion parameter, r²=x '²+y′², (u, v) for the respective pixel coordinate on original image, (x ', y ') is the coordinate of the actual picture point in units of millimeter, (x ", y ") is the ideal value of the picture point coordinate calculated by perspective model；

Described side elevation image module, for carrying out rotation and obtain virtual camera looking around fisheye camera, passes through equationReal camera and virtual camera are mapped, generates side elevation image；Wherein P_vAnd P₀The respectively point on virtual camera and real camera image, S is a zoom factor, K_vAnd K₀The respectively Intrinsic Matrix of virtual camera and real camera, K_0VRepresent the spin matrix between two cameras, R₀Represent the real camera spin matrix relative to reference frame, R_VRepresent the virtual camera spin matrix relative to bodywork reference frame；

Described distance matrix module, for according to perspective projection ultimate principle, obtaining the Intrinsic Matrix of virtual camera and outer parameter matrix, only consider ground, make Y=0, pass through equation by equation s ' p=K (RP+t)Generate ground distance matrix；Wherein P (X, Y, Z)^TRepresent bodywork reference frame next one three-dimensional point, p (u, v, 1)^TRepresenting the homogeneous coordinates of pixel under corresponding image coordinate system, s represents zoom factor, K, and R and t is the internal reference matrix of virtual camera and outer ginseng matrix, and (x, z) for the coordinate in the X-direction under bodywork reference frame and Z-direction the two direction；

Described monitored area image ROI ' module, for according to distance matrix upright projection rectangular area on the groundRepresenting, rectangular area, relative to distance matrix, generates the monitored area image ROI ' represented with tire centerline point；Traversal distance matrix in each point, for certain point (u, v), its value be (x, z), if this value meets inequation, then make ROI ' (u, v)=255, otherwise ROI ' (u, v)=0；Wherein, X^-And X⁺It is that under default bodywork reference frame, X-direction needs the minima of monitoring and the threshold value of maximum, Z^-And Z⁺It is that under default bodywork reference frame, Z-direction needs the minima of monitoring and the threshold value of maximum；The value of each point is that 0 or 255,0 these points of expression are made without follow-up object detection, and 255 represent that these points need to carry out follow-up object detection；

Described monitoring point array module, for the monitored area image ROI ' that will represent with tire centerline point, passes through equationConvert the monitored area image ROI represented with the top left co-ordinate of this rectangle to, and then convert monitoring point array A to and carrying out concrete object detection；Wherein, M, N are the detection window size M × N pixel of target object；

Described detection output module, for the element in the array A of monitoring point, (u, the detection window represented by v) is (u, v, M, N), inputs detection window and side elevation image, adopts known technology to detect, and whether finally export is target object.

7. vehicle monitoring system in blind area according to claim 6, it is characterised in that also include demarcating unit；

Described demarcation unit, for obtaining looking around the Intrinsic Matrix of fisheye camera, radial distortion parameter, tangential distortion parameter and looking around the fisheye camera outer parameter matrix relative to bodywork reference frame by Zhang Zhengyou standardizition.

8. vehicle monitoring system in blind area according to claim 6, it is characterised in that also include the elementary area after eliminating perspective distortion and undistorted tire image unit；

Elementary area after described elimination perspective distortion, looks around fisheye camera and obtains an optical axis for rotating and detect the vertical virtual camera of plane, can be eliminated the image after perspective distortion by virtual camera；

Described undistorted tire image unit, for, in the normal driving process of vehicle, pointing on the left of Current vehicle or right side, can obtain the undistorted tire image of side back car by virtual video camera.

9. vehicle monitoring system in blind area according to claim 6, it is characterised in that also include judging unit；

Described judging unit, for detection window is carried out feature extraction and calculating, then inputs to neutral net by calculated eigenvalue and carries out the judgement of grader.

10. vehicle monitoring system in the blind area according to any one of claim 6-9, it is characterised in that described in look around fisheye camera expansible.