CN103065323A - Subsection space aligning method based on homography transformational matrix - Google Patents

Abstract

The invention discloses a subsection space aligning method based on a homography transformational matrix. According to the subsection space aligning method based on the homography transformational matrix, large marked distance is sectioned, the homography transformational matrix between a camera coordinate system and a millimeter-wave radar coordinate system of each subsection is acquired, errors caused by using a same homography transformational matrix to represent coordinate relations of two sensors in the prior art are avoided, and space aligning of target detection of the large marked distance can be achieved. Relations of different coordinate systems between the camera and the millimeter-wave radar are deduced and represented, and finally the relations of the coordinate systems between the camera and the millimeter-wave radar are represented by the homography transformational matrix N. The two sensors are used for obtaining target data and solving the homography transformational matrix N, and a camera internal parameter matrix and a rotation matrix which are composed of solving scaling factors, focal distance and the like are avoided and a camera external parameter matrix composed of translation vectors is avoided. Therefore, operation process is greatly simplified and operation time is saved.

Description

A kind of segment space alignment methods based on the homography transformation matrix

Technical field

The present invention relates to unmanned vehicle multi-sensor information fusion technology field, be specifically related to a kind of segment space alignment methods based on the homography transformation matrix.

Background technology

Unmanned vehicle claims again Outdoor Intelligent Autonomous Mobile Robot, be a kind of environment sensing, dynamic decision and planning, behavior control device very high with the multi-functional intelligent degree such as carrying out that integrate, its rapidity to environment sensing, accuracy are inseparable with multi-sensor information fusion technology.Multi-sensor information fusion technology is exactly that computing machine takes full advantage of each sensor resource, by reasonable domination and the use to various measurement informations, on room and time, combine according to certain Optimality Criteria with redundant information complementary, generation is explained the consistance of observing environment or is described, produces simultaneously new fusion results.In the environment sensing module, vision sensor and millimetre-wave radar are two kinds of sensors commonly using.Size and profile information that the vision sensor sensing range is wide, can obtain target in the external environment, but it is subject to the extraneous factor impact, has target disappearance problem.And millimetre-wave radar resolution is high, antijamming capability is strong, can under various weather environments, accurately obtain distance, relative velocity and the azimuth information of target, but can not identify target shape and size, therefore, utilize this complementary characteristic to merge both information, obtain more comprehensive, reliable environmental information.And spacial alignment is the prerequisite of both information fusion.Spacial alignment essence is the transformation matrix relation of estimating between video camera and radar fix system.At present, in the Traditional Space alignment methods, in 20 meters demarcation distance range, on different range points, at random the point on the target is surveyed, obtain respectively that the coordinate system of target in video camera expressed and the expression of radar fix in being, the data of obtaining according to two sensors, the video camera external parameter matrix that the intrinsic parameters of the camera matrix that estimation is made of the scaling factor, focal length etc. and rotation matrix, translation vector consist of, this computation process is more loaded down with trivial details, also easily introduces error; In addition, when according to above-mentioned algorithm the target that surpasses 20 meters demarcation distance being found the solution transformation matrix, because scope is larger, therefore huge error causes the spacial alignment failure.

Summary of the invention

In view of this, the invention provides a kind of segment space alignment methods based on the homography transformation matrix, the video camera of unmanned vehicle loading and the spacial alignment of millimetre-wave radar can be in larger demarcation distance range, realized, the computation process of finding the solution the homography transformation matrix can also be simplified simultaneously.

A kind of segment space alignment methods based on the homography transformation matrix of the present invention comprises the steps:

Step 1: set up the relation based on the homography transformation matrix between camera coordinate system and the millimetre-wave radar coordinate system:

Image coordinate system O ' the uv of definition video camera, wherein O ' is positioned at the upper left corner on video camera imaging plane; The u axle is parallel with the camera-scanning line direction; The v axle is perpendicular to the camera-scanning line direction;

Definition O " " ρ θ is the millimetre-wave radar polar coordinate system, and O " " is the center on millimetre-wave radar surface; ρ is the air line distance between target and millimetre-wave radar; θ is the angle that target departs from millimetre-wave radar plane of scanning motion center line, and then the image coordinate system O ' uv of video camera and millimetre-wave radar polar coordinate system O " " ρ θ Relations Among are expressed as:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=N\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 1\end{array}\right]---\left(7\right)$

Wherein, $N=\left[\begin{array}{ccc}{n}_{11}& n_{12}& n_{13}\\ n_{21}& n_{22}& n_{23}\\ n_{31}& n_{32}& n_{33}\end{array}\right],$ Be defined as the homography transformation matrix;

Step 2: determine the suitable demarcation distance between unmanned vehicle and the spotting:

Definition O " " X _rY _rZ _rExpression millimetre-wave radar rectangular coordinate system, O " " is the center on millimetre-wave radar surface; Y _rAxle is millimetre-wave radar plane of scanning motion center line, and is perpendicular to the millimetre-wave radar surface, directed straight ahead; X _rAxle and Y _rVertically, point to the right side; Z _rAxle is perpendicular to X _r, Y _rThe plane of determining, points upwards;

Then the pass between millimetre-wave radar rectangular coordinate system and the millimetre-wave radar polar coordinate system is:

$\left[\begin{array}{c}{X}_r\\ Y_r\\ 1\end{array}\right]=\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 1\end{array}\right]---\left(7\right)\′$

The distance of spotting and unmanned vehicle is at millimetre-wave radar rectangular coordinate system longitudinal axis Y _rOn projection be called the demarcation distance; In the investigative range of millimetre-wave radar, according to the maximum movement speed of unmanned vehicle, determine suitable demarcation distance L; To demarcate distance L and be divided into from the near to the remote close range L1 and long-distance range L2, in close range L1, it will be divided into the m1 section, in long-distance range L2, it will be divided into the m2 section, and guarantee that L1/m1 is less than L2/m2;

Step 3: the image and the data message that gather respectively spotting by the video camera that loads in the unmanned vehicle and millimetre-wave radar:

Spotting is placed on respectively will demarcates different segmentations place that distance L is divided in the step 2, millimetre-wave radar and video camera are surveyed the target at above-mentioned m1+m2 segment distance place respectively, when surveying, for the target at every segment distance place, with target along Y _rIt is capable that direction of principal axis is divided into m, again with every delegation along X _rDirection of principal axis is divided into the h segment, and the control millimetre-wave radar obtains the coordinate data (X of each segment ^M _Rk, Y ^M _Rk), the control video camera is taken the view data f of every segment _k ^M, M=1 wherein ..., (m1+m2), k=1,2 ..., mh;

Step 4: for the view data f of each segment in each segmentation of video camera acquisition in the step 3 _k ^M, the center-of-mass coordinate (u of difference computed image _k ^M, v _k ^M);

Step 5: the homography spatial alternation matrix of finding the solution expression millimetre-wave radar coordinate system and camera coordinate system Relations Among:

The millimetre-wave radar coordinate data that all segments that obtain for each segment distance of telling in the whole demarcation distance L are corresponding (

Y ^M _Rk) and the view data (u of video camera ^M _k, v ^M _k) form data set corresponding in each segmentation, with each data set be updated to respectively formula (7) and (7) ' in, obtain:

$\left[\begin{array}{c}{u}_1^M\\ \·\\ \·\\ \·\\ u_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{11}^M\\ n_{12}^M\\ n_{13}^M\end{array}\right]---\left(8\right)$

$\left[\begin{array}{c}{v}_1^M\\ \·\\ \·\\ \·\\ v_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{21}^M\\ n_{22}^M\\ n_{23}^M\end{array}\right]---\left(9\right)$

With $\left[\begin{array}{c}1\\ \·\\ \·\\ \·\\ 1\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{31}^M\\ n_{32}^M\\ n_{33}^M\end{array}\right]---\left(10\right)$

Definition $P^M=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right],$ $N^M={\left[\begin{array}{ccc}{n}_{11}^M& n_{21}^M& n_{31}^M\\ n_{12}^M& n_{22}^M& n_{32}^M\\ n_{13}^M& n_{23}^M& n_{33}^M\end{array}\right]}^T,$ $U^M={\left[\begin{array}{ccc}{u}_1^M& \·\·\·& u_k^M\end{array}\right]}^T,$ $V^M={\left[\begin{array}{ccc}{v}_1^M& \·\·\·& v_k^M\end{array}\right]}^T,$ I _{K * 1}=[1 ... 1] ^T, homography spatial alternation matrix N then ^MLeast square solution can be expressed as: $N^M={\left[\begin{array}{ccc}{N}_1^{M^T}& N_2^{M^T}& N_3^{M^T}\end{array}\right]}^T,$ Wherein,

With Be respectively: $N_1^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{U}^M,$ $N_2^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{V}^M$ With $N_3={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{I}_{k\×1};$

Step 6: the spacial alignment of realizing vision sensor and millimetre-wave radar:

According to the actual range of the spotting of millimetre-wave radar scanning, judge that this distance in which segmentation, searches this apart from the homography spatial alternation matrix of correspondence, the implementation space aligning in step 2 in m1+m2 the result that step 5 calculates.

In step 2, when the demarcation distance was 50 meters, 0-20 rice was close range, is divided into 4 sections; 20-50 rice is long-distance range, is divided into 3 sections.

The method of the center-of-mass coordinate of the every a bit of image in the described step 4 in the calculating spotting image is as follows:

S40, manually choose the candidate region of containing spotting;

S41, the candidate region image is carried out medium filtering, the noise in the removal of images;

S42, the candidate region image is carried out the Sobel operator edge detection, obtain the spotting edge image of binaryzation;

S43, in the image coordinate system take pixel as unit, seek coordinate minimum value and peaked u axle pixel coordinate u in the spotting edge image along the u axle _Min, u _Max, along coordinate minimum value and the peaked v axle pixel coordinate v in the v axle searching spotting edge image _Min, v _Max, press clockwise or counterclockwise connect with straight line above-mentioned 4, form a quadrilateral area, in this quadrilateral area, utilize formula ${u_k}^M=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ With ${v^M}_k=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ Calculate the center-of-mass coordinate (u of spotting _k ^M, v _k ^M), f wherein _k ^MM gray-scale value apart from pixel (i, j) in quadrilateral area corresponding to k segment of target in the segmentation of (i, j) expression.

The present invention has following beneficial effect:

1) by larger demarcation distance is carried out segmentation, try to achieve respectively homography transformation matrix between video camera and the millimetre-wave radar coordinate system for each segmentation, avoid the error that causes owing to the coordinate relation of expressing with same homography transformation matrix between two sensors in the prior art, thereby can realize the spacial alignment to larger demarcation distance objective detection;

2) by the relation between the different coordinates of derive sign video camera and millimetre-wave radar, adopt at last the homograph matrix N to characterize both coordinate system relations, the target data that obtains respectively with two sensors solves the homograph matrix N again, avoided finding the solution the video camera external parameter matrix that intrinsic parameters of the camera matrix that the scaling factor, focal length etc. consist of and rotation matrix, translation vector consist of, greatly simplify calculating process, saved operation time;

3) different with the degree of concern of long distance target to low coverage according to the sensor in the unmanned vehicle, different to the segmentation fineness of low coverage and long distance, when guaranteeing spacial alignment, can also reduce calculated amount;

4) in image coordinate system, after having obtained the edge image of target, find respectively the pixel of two axial maximal values and minimum value, these 4 pixels are surrounded quadrilateral, find again this tetragonal barycenter, the method can be determined the barycenter of each object edge image fast, thereby can the simplified operation process, simultaneously can also be so that spacial alignment is more accurate.

Description of drawings

Fig. 1 is camera pin-hole model schematic diagram;

Fig. 2 is millimetre-wave radar coordinate system schematic diagram;

Fig. 3 image coordinate system and millimetre-wave radar rectangular coordinate system mapping relations schematic diagram.

Embodiment

Below in conjunction with the accompanying drawing embodiment that develops simultaneously, describe the present invention.

The invention provides a kind of segment space alignment methods based on the homography transformation matrix, comprise the steps:

Step 1: set up the relation based on the homography transformation matrix between camera coordinate system and the millimetre-wave radar coordinate system:

As shown in Figure 1, OX _cY _cZ _cThe expression camera coordinate system, initial point O is positioned at the photocentre of video camera; X _cAxle is parallel to the camera-scanning line direction, points to the direction that scanning element increases; Y _cAxle points to the direction that scan line increases perpendicular to the camera-scanning line direction; Z _cAxle points to the camera direction of visual lines perpendicular to imaging plane.O ' uv represents the image coordinate system take pixel as unit, and O ' is positioned at the upper left corner of imaging plane; U axle and X _cParallel; V axle and Y _cParallel.O " xy represents the image coordinate system take millimeter as unit, O " is the focus of imaging plane; X axle and X _cParallel; Y axle and Y _cParallel.F is the focal length of camera, and I represents imaging plane.Postulated point P is at coordinate system OX _cY _cZ _c, " coordinate under xy and the O ' uv is respectively (X to O _c, Y _c, Z _c), (x, y) and (u, v), by the P point at coordinate system OX _cY _cZ _cWith coordinate system O " the geometric proportion relation under the xy:

Above-mentioned relation is expressed as homogeneous form to be had:

$Z_c\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]---\left(1\right)$

" zooming and panning under xy and the coordinate system O ' uv coordinate system concern: x=S (u-u at coordinate system O by the P point ₀), y=S (v-v ₀), being expressed as homogeneous form has:

$\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=S\left[\begin{array}{ccc}1& 0& -u_0\\ 0& 1& -v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]---\left(2\right)$

Wherein S is the scaling factor, (u ₀, v ₀) be the coordinate of coordinate system O " xy initial point O " under coordinate system O ' uv.Suppose world coordinate system O " ' X _wY _wZ _w, the coordinate of some P under this coordinate system is (X _w, Y _w, Z _w), the relation of two coordinate systems is represented to have with homogeneous form:

$\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]=\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(3\right)$

Wherein R and T represent respectively rotation matrix and translation vector.Composite type (1), formula (2) and formula (3) then have relation:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{\mathrm{\β}}\left[\begin{array}{ccc}1& 0& u_0\\ 0& 1& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(4\right)$

Wherein, β=Z _cS.

As shown in Figure 2, O " " X _rY _rZ _rExpression millimetre-wave radar rectangular coordinate system, O " " is the center on millimetre-wave radar surface; Y _rAxle is millimetre-wave radar plane of scanning motion center line, and is perpendicular to the millimetre-wave radar surface, directed straight ahead; X _rAxle and Y _rVertically, point to the right side; Z _rAxle is perpendicular to X _r, Y _rThe plane of determining, points upwards.O " " ρ θ represents the millimetre-wave radar polar coordinate system, initial point and coordinate system O " " X _rY _rZ _rInitial point overlap; ρ is the air line distance between target and millimetre-wave radar; θ is the angle that target departs from millimetre-wave radar plane of scanning motion center line.Point P is at O " " ρ θ and O " " X _rY _rZ _rUnder coordinate be respectively (ρ, θ) and (X _r, Y _r, Z _r), because the plane of scanning motion of millimetre-wave radar is coordinate system O " " ρ θ next one two dimensional surface, Z is arranged _r=0, some P is at O " " ρ θ and O " " X _rY _rZ _rUnder triangle relation be expressed as:

$\left[\begin{array}{c}{X}_r\\ Y_r\\ Z_r\end{array}\right]=\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 0\end{array}\right]---\left(5\right)$

Supposing that the millimetre-wave radar rectangular coordinate is world coordinate system, is intermediate variable and convolution (4) and formula (5) by world coordinates, and then image coordinate system O ' uv and millimeter wave polar coordinate system O " " ρ θ relation table are shown:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{\mathrm{\β}}\left[\begin{array}{ccc}1& 0& u_0\\ 0& 1& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 0\\ 1\end{array}\right]---\left(6\right)$

As shown in Figure 3, when camera and millimetre-wave radar are observed same target, but the impact point that through type (6) scans millimetre-wave radar projects in the image of camera collection.But need to estimate the camera inner parameters such as zoom factor S, camera focal distance f and the external parameters such as rotation matrix R, translation vector T, computation process is complicated, calculate for simplifying, utilize the transformation relation N of an equivalence to represent the relation of O ' uv and O " " ρ θ, be expressed as:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=N\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 1\end{array}\right]---\left(7\right)$

Wherein, $N=\left[\begin{array}{ccc}{n}_{11}& n_{12}& n_{13}\\ n_{21}& n_{22}& n_{23}\\ n_{31}& n_{32}& n_{33}\end{array}\right],$ Be called the homography transformation matrix.

Step 2: determine suitable demarcation distance and carry out equitable subsection to demarcating distance:

We call the demarcation distance to the spotting that measures under the millimetre-wave radar polar coordinate system apart from ρ, distance L be will demarcate and close range L1 and long-distance range L2 will be divided into from the near to the remote, in close range L1, it is divided into the m1 section, in long-distance range L2, it is divided into the m2 section, because in the unmanned vehicle detection process, need to accurately survey nearer target, and for only need judgement roughly of distant object, therefore, it is larger that the long-distance range segmentation can be compared the close range segmentation, shown in L1/m1 less than L2/m2;

In the present embodiment, the maximal rate of unmanned vehicle regulation is at 36km/h, the distance areas of paying close attention in the process of moving according to the unmanned vehicle relation relevant with its velocity magnitude as can be known, we determine that 50m is suitable demarcation distance.In the demarcation distance of 50m, learn by experiment: utilize single homography spatial alternation matrix can cause aiming at unsuccessfully, therefore, according to the difference of unmanned vehicle rate request and concern distance areas, demarcating apart from being in the 0-20m, interval 5m is one section totally 4 sections; Demarcating apart from being in the 20-50m, interval 10m is one section totally 3 sections.

Step 3: the image and the data message that gather respectively spotting by the video camera that loads in the unmanned vehicle and millimetre-wave radar:

For guaranteeing the accuracy of spacial alignment, in each segmentation, vertically getting the five-element respectively, every row is laterally being got 7 groups of image and data messages that spotting is corresponding.The millimetre-wave radar that utilization is installed in the camera of the IEEE1394 interface on the unmanned vehicle and CAN bus interface gathers image and the data message of spotting under the Same Scene, and with communication in industrial computer.

Step 4: for the view data f of each segment in each segmentation of video camera acquisition in the step 3 _k ^M, the center-of-mass coordinate (u of difference computed image _k ^M, v _k ^M);

Generally, the clustering method that millimetre-wave radar adopts is nearest field method, and therefore, we think that the coordinate figure of millimetre-wave radar barycenter position under the spotting data message that scans under the millimetre-wave radar coordinate system and image coordinate system is mutually corresponding.To digital picture f, f (i, j) represents gray-scale value, the point in (i, j) presentation video zone.The center-of-mass coordinate of spotting in the computed image information, adopt following steps:

S40, manually choose the candidate region of containing spotting, to reduce background to the interference of spotting centroid calculation.

S41, medium filtering is carried out in the candidate region, noise in the removal of images.Medium filtering is realized by expression formula f (i, j)=Median{f (i-k, j-l).Wherein, (k, l) ∈ ω, ω are 3 * 3 fields of pixel.

S42, the candidate region image is carried out the Sobel operator edge detection, obtain containing the candidate region edge image of spotting.The Sobel operator adopts masterplate $\mathrm{mask}=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right],$ Candidate region image and mask are made convolution algorithm, obtain the bianry image of candidate region.

S43, in the image coordinate system take pixel as unit, seek coordinate minimum value and peaked u axle pixel coordinate u in the spotting edge image along the u axle _Min, u _Max, along coordinate minimum value and the peaked v axle pixel coordinate v in the v axle searching spotting edge image _Min, v _Max, press clockwise or counterclockwise connect with straight line above-mentioned 4, form a quadrilateral area, in this quadrilateral area, utilize formula ${u_k}^M=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ With ${v^M}_k=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ Calculate the center-of-mass coordinate (u of spotting _k ^M, v _k ^M), f wherein _k ^MM gray-scale value apart from pixel (i, j) in quadrilateral area corresponding to k segment of target in the segmentation of (i, j) expression;

Step 5: the homography spatial alternation matrix of finding the solution expression millimetre-wave radar coordinate system and camera coordinate system Relations Among:

The millimetre-wave radar coordinate data that all segments that obtain for each segment distance of telling in the whole demarcation distance L are corresponding (

Y ^M _Rk) and the view data (u of video camera ^M _k, v ^M _k) form data set corresponding in each segmentation, each data set is updated to respectively in formula (5) and (7), obtain:

$\left[\begin{array}{c}{u}_1^M\\ \·\\ \·\\ \·\\ u_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{11}^M\\ n_{12}^M\\ n_{13}^M\end{array}\right]---\left(8\right)$

$\left[\begin{array}{c}{v}_1^M\\ \·\\ \·\\ \·\\ v_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{21}^M\\ n_{22}^M\\ n_{23}^M\end{array}\right]---\left(9\right)$

With $\left[\begin{array}{c}1\\ \·\\ \·\\ \·\\ 1\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{31}^M\\ n_{32}^M\\ n_{33}^M\end{array}\right]---\left(10\right)$

Definition $P^M=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right],$ $N^M={\left[\begin{array}{ccc}{n}_{11}^M& n_{21}^M& n_{31}^M\\ n_{12}^M& n_{22}^M& n_{32}^M\\ n_{13}^M& n_{23}^M& n_{33}^M\end{array}\right]}^T,$ $U^M={\left[\begin{array}{ccc}{u}_1^M& \·\·\·& u_k^M\end{array}\right]}^T,$ $V^M={\left[\begin{array}{ccc}{v}_1^M& \·\·\·& v_k^M\end{array}\right]}^T,$ I _{K * 1}=[1 ... 1] ^T, homography spatial alternation matrix N then ^MLeast square solution can be expressed as: $N^M={\left[\begin{array}{ccc}{N}_1^{M^T}& N_2^{M^T}& N_3^{M^T}\end{array}\right]}^T,$ Wherein,

With

Be respectively: $N_1^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{U}^M,$ $N_2^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{V}^M$ With $N_3={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{I}_{k\×1};$

Step 6: the spacial alignment of realizing vision sensor and millimetre-wave radar:

According to the actual range of the spotting of millimetre-wave radar scanning, at first, judge this distance in step 2 in which segmentation, then, the N of homography spatial alternation matrix in this segmentation that calculates according to step 5 ^M(M=1,2 ..., 7), the implementation space is aimed at.

In sum, above is preferred embodiment of the present invention only, is not for limiting protection scope of the present invention.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.

Claims (3)

1. the segment space alignment methods based on the homography transformation matrix is characterized in that, comprises the steps:

Step 1: set up the relation based on the homography transformation matrix between camera coordinate system and the millimetre-wave radar coordinate system:

Image coordinate system O ' the uv of definition video camera, wherein O ' is positioned at the upper left corner on video camera imaging plane; The u axle is parallel with the camera-scanning line direction; The v axle is perpendicular to the camera-scanning line direction;

Definition O " " ρ θ is the millimetre-wave radar polar coordinate system, and O " " is the center on millimetre-wave radar surface; ρ is the air line distance between target and millimetre-wave radar; θ is the angle that target departs from millimetre-wave radar plane of scanning motion center line, and then the image coordinate system O ' uv of video camera and millimetre-wave radar polar coordinate system O " " ρ θ Relations Among are expressed as:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=N\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 1\end{array}\right]---\left(7\right)$

Wherein, $N=\left[\begin{array}{ccc}{n}_{11}& n_{12}& n_{13}\\ n_{21}& n_{22}& n_{23}\\ n_{31}& n_{32}& n_{33}\end{array}\right],$ Be defined as the homography transformation matrix;

Step 2: determine the suitable demarcation distance between unmanned vehicle and the spotting:

Definition O " " X _rY _rZ _rExpression millimetre-wave radar rectangular coordinate system, O " " is the center on millimetre-wave radar surface; Y _rAxle is millimetre-wave radar plane of scanning motion center line, and is perpendicular to the millimetre-wave radar surface, directed straight ahead; X _rAxle and Y _rVertically, point to the right side; Z _rAxle is perpendicular to X _r, Y _rThe plane of determining, points upwards;

Then the pass between millimetre-wave radar rectangular coordinate system and the millimetre-wave radar polar coordinate system is:

$\left[\begin{array}{c}{X}_r\\ Y_r\\ 1\end{array}\right]=\left[\begin{array}{c}\mathrm{\ρ}\sin \mathrm{\θ}\\ \mathrm{\ρ}\cos \mathrm{\θ}\\ 1\end{array}\right]---\left(7\right)\′$

The distance of spotting and unmanned vehicle is at millimetre-wave radar rectangular coordinate system longitudinal axis Y _rOn projection be called the demarcation distance; In the investigative range of millimetre-wave radar, according to the maximum movement speed of unmanned vehicle, determine suitable demarcation distance L; To demarcate distance L and be divided into from the near to the remote close range L1 and long-distance range L2, in close range L1, it will be divided into the m1 section, in long-distance range L2, it will be divided into the m2 section, and guarantee that L1/m1 is less than L2/m2;

Step 3: the image and the data message that gather respectively spotting by the video camera that loads in the unmanned vehicle and millimetre-wave radar:

Spotting is placed on respectively will demarcates different segmentations place that distance L is divided in the step 2, millimetre-wave radar and video camera are surveyed the target at above-mentioned m1+m2 segment distance place respectively, when surveying, for the target at every segment distance place, with target along Y _rIt is capable that direction of principal axis is divided into m, again with every delegation along X _rDirection of principal axis is divided into the h segment, and the control millimetre-wave radar obtains the coordinate data (X of each segment ^M _Rk, Y ^M _Rk), the control video camera is taken the view data f of every segment _k ^M, M=1 wherein ..., (m1+m2), k=1,2 ..., mh;

Step 4: for the view data f of each segment in each segmentation of video camera acquisition in the step 3 _k ^M, the center-of-mass coordinate (u of difference computed image _k ^M, v _k ^M);

Step 5: the homography spatial alternation matrix of finding the solution expression millimetre-wave radar coordinate system and camera coordinate system Relations Among:

The millimetre-wave radar coordinate data that all segments that obtain for each segment distance of telling in the whole demarcation distance L are corresponding (

Y ^M _Rk) and the view data (u of video camera ^M _k, v ^M _k) form data set corresponding in each segmentation, with each data set be updated to respectively formula (7) and (7) ' in, obtain:

$\left[\begin{array}{c}{u}_1^M\\ \·\\ \·\\ \·\\ u_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{11}^M\\ n_{12}^M\\ n_{13}^M\end{array}\right]---\left(8\right)$

$\left[\begin{array}{c}{v}_1^M\\ \·\\ \·\\ \·\\ v_k^M\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{21}^M\\ n_{22}^M\\ n_{23}^M\end{array}\right]---\left(9\right)$

With $\left[\begin{array}{c}1\\ \·\\ \·\\ \·\\ 1\end{array}\right]=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right]\left[\begin{array}{c}{n}_{31}^M\\ n_{32}^M\\ n_{33}^M\end{array}\right]---\left(10\right)$

Definition $P^M=\left[\begin{array}{ccc}{X}_{r1}^M& Y_{r1}^M& 1\\ \·& \·& \·\\ \·& \·& \·\\ \·& \·& \·\\ X_{\mathrm{rk}}^M& Y_{\mathrm{rk}}^M& 1\end{array}\right],$ $N^M={\left[\begin{array}{ccc}{n}_{11}^M& n_{21}^M& n_{31}^M\\ n_{12}^M& n_{22}^M& n_{32}^M\\ n_{13}^M& n_{23}^M& n_{33}^M\end{array}\right]}^T,$ $U^M={\left[\begin{array}{ccc}{u}_1^M& \·\·\·& u_k^M\end{array}\right]}^T,$ $V^M={\left[\begin{array}{ccc}{v}_1^M& \·\·\·& v_k^M\end{array}\right]}^T,$ I _{K * 1}=[1 ... 1] ^T, homography spatial alternation matrix N then ^MLeast square solution can be expressed as: $N^M={\left[\begin{array}{ccc}{N}_1^{M^T}& N_2^{M^T}& N_3^{M^T}\end{array}\right]}^T,$ Wherein,

With

Be respectively: $N_1^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{U}^M,$ $N_2^M={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{V}^M$ With $N_3={\left(P^M{P}^{M^T}\right)}^{-1}{P}^{M^T}{I}_{k\×1;}$

Step 6: the spacial alignment of realizing vision sensor and millimetre-wave radar:

According to the actual range of the spotting of millimetre-wave radar scanning, judge that this distance in which segmentation, searches this apart from the homography spatial alternation matrix of correspondence, the implementation space aligning in step 2 in m1+m2 the result that step 5 calculates.

2. a kind of segment space alignment methods based on the homography transformation matrix as claimed in claim 1 is characterized in that, when the demarcation distance of step 2 was 50 meters, 0-20 rice was close range, is divided into 4 sections; 20-50 rice is long-distance range, is divided into 3 sections.

3. a kind of segment space alignment methods based on the homography transformation matrix as claimed in claim 1 is characterized in that, the method for the center-of-mass coordinate of the every a bit of image in the described step 4 in the calculating spotting image is as follows:

S40, manually choose the candidate region of containing spotting;

S41, the candidate region image is carried out medium filtering, the noise in the removal of images;

S42, the candidate region image is carried out the Sobel operator edge detection, obtain the spotting edge image of binaryzation;

S43, in the image coordinate system take pixel as unit, seek coordinate minimum value and peaked u axle pixel coordinate u in the spotting edge image along the u axle _Min, u _Max, along coordinate minimum value and the peaked v axle pixel coordinate v in the v axle searching spotting edge image _Min, v _Max, press clockwise or counterclockwise connect with straight line above-mentioned 4, form a quadrilateral area, in this quadrilateral area, utilize formula ${u_k}^M=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ With ${v^M}_k=\frac{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{{\mathrm{if}}_k}^M(i,j)}{\underset{i}{\mathrm{\Σ}}\underset{j}{\mathrm{\Σ}}{f_k}^M(i,j)}$ Calculate the center-of-mass coordinate (u of spotting _k ^M, v _k ^M), f wherein _k ^MM gray-scale value apart from pixel (i, j) in quadrilateral area corresponding to k segment of target in the segmentation of (i, j) expression.

Images