CN103177442A - Calibrating method for two-dimensional laser and camera without overlapped viewing fields - Google Patents

Abstract

The invention discloses a calibrating method for two-dimensional laser and a camera without overlapped viewing fields. A calibrating system comprises a two-dimensional laser scanner, the camera, a plane mirror and the like. The calibrating method includes calibrating intrinsic parameters of the camera by the aid of the traditional chessboard table via a pinhole camera model; reasonably configuring the laser scanner, the camera and the plane mirror, adhering the black and white chessboard table on the plane mirror, locating the chessboard table within the range of a viewing field of the camera and determining a transformation matrix between a coordinate system of the chessboard and a coordinate system of the camera; determining a transformation relation between the coordinate system of the chessboard and a coordinate system of a camera image reflected in the mirror according to a symmetric relation of mirror surface images so as to acquire a transformation relation between the actual camera and the camera image reflected in the mirror; performing observation at selected sufficient frequency by changing the position and gestures of a black triangular calibrating plate to acquire sufficient observation samples, estimating by a Gauss-Newton iterative process to acquire a transformation matrix between the coordinate system of the camera image and the two-dimensional laser scanner; and acquiring a rotation and translation transformation matrix between the coordinate system of the camera and a coordinate system of the two-dimensional laser scanner via relations between positions and gestures of the camera and the camera image and relations between positions and gestures of the camera image and the two-dimensional laser scanner.

Description

The camera of a kind of zero lap visual field and the scaling method of two-dimensional laser

Technical field

The present invention relates to camera in the Fusion field and the demarcation of two-dimensional laser.

Background technology

In the vision guided navigation of intelligent robot, use multi-sensor fusion technology, can obtain about more reliable, unified, the meticulous description of environment, so that decision-making and planning and control.When carrying out data fusion, the demarcation of multisensor is one of problem needing to overcome.Demarcate and generally be divided into the demarcation of each sensor self parameter and the combined calibrating of multisensor.The demarcation of each sensor self parameter is the accuracy of the image data in order to guarantee, and the combined calibrating of multisensor is the accurate coupling for multi-sensor data.

In the technology of camera and two dimensional laser scanning instrument data fusion, the combined calibrating of camera and two-dimensional laser is a vital link.The vision sensor investigative range is wide, and signal is abundant, but is subject to ectocine, to the ambient lighting sensitive, has target disappearance, the problem such as fuzzy.Laser is subjected to the external environment condition variable effect little, but exists data point sparse, the shortcoming that sweep frequency is low.To both merge, can make up disappearance separately.

In recent years, the Data fusion technique of camera and two dimensional laser scanning instrument becomes the focus of research.By reasonably setting up camera coordinates system and laser scanner coordinate system, utilize the space constraint of corresponding point in camera image and laser scanner depth map to find the solution the spatial alternation relation of two coordinate systems.But there are many problems in the combined calibrating method of present existing camera and laser scanner:

1. general passing through of adopting seeks the matching process that the staggered information of black and white lattice is completed vision and two-dimensional laser data point, depends on the characteristic of scanning substances and apart from the distance of laser, the sparse degree of laser data point also greatly affects calibration result simultaneously.

2. mate right selective dependency in manually clicking, the personal error of introducing can't well be added up and control, and precision is lower, manually clicks cost in time simultaneously larger.

3. require camera and laser scanner that overlapping visual field must be arranged, the demarcation thing detected simultaneously, greatly limited the putting position of camera and laser scanner.

Summary of the invention

The objective of the invention is to propose the high camera of a kind of precision and two-dimensional laser combined calibrating method in order to overcome the deficiency of existing camera and two-dimensional laser scaling method.

The present invention adopts following technical scheme:

The camera of a kind of zero lap visual field and the calibration system of two-dimensional laser comprise two dimensional laser scanning instrument, camera, level crossing etc.

The camera of a kind of zero lap visual field and the scaling method of two-dimensional laser comprise the following steps:

Step 1: adopt pinhole camera modeling to consider the lens distortion factor, the intrinsic parameter that uses traditional chessboard form to carry out camera is demarcated, and obtains the Intrinsic Matrix M of camera.

Step 2: the relation between reasonable disposition laser scanner, camera, level crossing, and paste a chequered with black and white chessboard form on level crossing.

Step 3: determine that minute surface chessboard coordinate system and camera coordinates are position orientation relation.Timing signal, the chessboard paper plane is positioned at the viewing field of camera scope, use the harris angular-point detection method determine in angle point at the coordinate of image coordinate system, use Gauss's Newton iteration method to try to achieve rotation matrix R and translation matrix T between chessboard coordinate system and camera coordinates system.

Step 4: try to achieve in chessboard coordinate system and mirror camera as the transformation relation between coordinate system according to the mirror image symmetric relation, thereby can determine the transformation relation between camera in actual camera and mirror.

Step 5: use the black triangle scaling board, determine that camera is as the relation between coordinate system and two dimensional laser scanning instrument coordinate system.

Detailed step wherein is as follows:

5-1. reasonably put scaling board, make that in two dimensional laser scanning instrument and mirror, camera can detect scaling board simultaneously.

5-2. extract by profile and approach three summits that leg-of-mutton method obtains scaling board imaging in camera, two summits of the intersection of the detection laser plane of scanning motion and scaling board, two points (laser coordinate system) that three summits (image coordinate system) of detecting in image and laser and scaling board edge is crossing are as a sample.

5-3. repeating step 5-1,5-2 obtains enough samples, sets the initial value of rotation amount and translational movement, uses Gauss's Newton iteration to carry out optimal estimation, is rotated and translation matrix.

Step 6: by the position orientation relation between camera and its picture and the position orientation relation between camera picture and two dimensional laser scanning instrument, obtain the rotation translation transformation matrix between camera coordinates system and two dimensional laser scanning instrument coordinate system.

Advantage of the present invention has:

1. camera and two dimensional laser scanning instrument do not have overlapping visual field, and both data fusion have larger field range.

2. need not manually click coupling right, only need demarcate the selection in zone, improve the automaticity of demarcating, have the demarcation rapidity, the characteristics of robotization.

3. the present invention has higher stated accuracy, for the camera in later stage and the data fusion of two dimensional laser scanning instrument are laid a good foundation.

Description of drawings

Fig. 1 is system's composition diagram of the combined calibrating method that proposes of the present invention.

Fig. 2 is that in the present invention, defined camera coordinates is.

Fig. 3 is the chessboard schematic diagram that uses in the present invention.

Fig. 4 is that in the present invention, camera and picture coordinate system and chessboard coordinate system transformation concern schematic diagram.

Fig. 5 is camera picture and laser calibration schematic diagram in the present invention.

Embodiment

Below in conjunction with accompanying drawing, the present invention is further illustrated.

Be illustrated in figure 1 as the calibration system composition diagram of camera of the present invention and two-dimensional laser.This system mainly comprises: two dimensional laser scanning instrument, camera, level crossing etc.

Step 1: adopt pinhole camera modeling to consider the lens distortion factor, the intrinsic parameter that uses traditional chessboard form to carry out camera is demarcated.In the plane of delineation, take in the upper left corner as true origin, horizontal direction is the u axle, vertical direction is the v axle, and unit is pixel, because rectangular coordinate system u-v has only represented the residing line number of pixel and columns in digital picture, and do not express the position of this pixel in image with physical unit, need to set up again the coordinate system x-y that physical unit represents, the coordinate of the image coordinate system of (x, y) expression take millimeter as unit.In the x-y coordinate system, initial point O ₁Be defined in the intersection point place of camera optical axis and the plane of delineation, establish O ₁Coordinate in the u-v coordinate system is (u ₀, v ₀), the physical size size of each pixel on x axle and y direction of principal axis is dx, dy, and the coordinate of each pixel under two coordinate systems has following relation in image:

$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{\mathrm{dx}}& 0& u_0\\ 0& \frac{1}{\mathrm{dy}}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]---\left(1\right)$

A world coordinate system of describing position relationship between three-dimensional object is selected as shown in Figure 2 by camera coordinates system, and camera coordinates system and world coordinate system can be described by rotation matrix R and translation vector T.The spatial point homogeneous coordinates under world coordinate system and camera coordinates system again is respectively (X _w, Y _w, Z _w, 1) ^T, (X _c, Y _c, Z _c, 1) ^T, following relation will be arranged:

$\left[\begin{array}{c}{X}_c\\ Y_{\mathrm{<figure-callout\; id="1"\; label="c\; Z\; c"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">c</figure-callout>}}& \\ Z_c{}_{}\\ 1\end{array}\right]=\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=M\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]---\left(2\right)$

(1) (2) two formulas are brought into the relational expression of the coordinate in the image coordinate system of this point and camera coordinates system:

$Z_c\left[\begin{array}{c}x\\ \mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">y</figure-callout>}\\ 1\end{array}\right]=\left[\begin{array}{cccc}\mathrm{<figure-callout\; id="0"\; label="f"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">f</figure-callout>}& 0& 0& 0\\ 0& \mathrm{<figure-callout\; id="0"\; label="f"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">f</figure-callout>}& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_{\mathrm{<figure-callout\; id="1"\; label="c"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">c</figure-callout>}}\\ 1\end{array}\right]$

Obtain the relational expression between image coordinate and world coordinates:

$Z_{\mathrm{<figure-callout\; id="1"\; label="c\; u\; v"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">c</figure-callout>}}\left[\begin{array}{c}u\\ v\end{array}\right]\left[\begin{array}{c}\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{\mathrm{dx}}& 0& u_0\\ 0& \frac{1}{\mathrm{dy}}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}\mathrm{<figure-callout\; id="0"\; label="f"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">f</figure-callout>}& 0& 0& 0\\ 0& \mathrm{<figure-callout\; id="0"\; label="f"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">f</figure-callout>}& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=\left[\begin{array}{cccc}\mathrm{\&alpha;}& 0& u_0& 0\\ 0& \mathrm{\&beta;}& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]$

Wherein, α=f/dx, β=f/dy.α, β, u ₀, v ₀Only relevant with the inner structure of camera, be the inner parameter of camera.

Step 2: the relation between reasonable disposition laser scanner, camera, level crossing, and paste a chequered with black and white chessboard form on level crossing.

Step 3: determine that minute surface chessboard coordinate system and camera coordinates are position orientation relation.Timing signal, the chessboard paper plane is positioned at the viewing field of camera scope, is expressed as matrix form for the interior angular coordinate on the chessboard paper plane to be

P _board(i,j)=[x y z] ^T=[ia ja 0] ^T

Wherein, a is the chessboard black and white square length of side, and i is angle point numeral index in the coordinate system directions X shown in Figure 3, i=1, and 2 ..., m, j are the interior angle piont mark index along Y-direction, j=1, and 2 ..., n.

Use the harris angular-point detection method to determine that interior angle point at the coordinate of image coordinate system is

P _cam(i,j)=[u _i,j v _i,j 1] ^T，

Obtained so respectively two groups of corresponding point under chessboard coordinate system and camera image coordinate system, its transformation relation is

P _cam(i,j)=M(RP _board(i,j)+T)

M is the camera Intrinsic Matrix, and T is for being tied to the translation matrix of camera coordinates system, T=[t from the chessboard coordinate _xt _yt _z] ^T, R is for be tied to the rotation matrix of camera coordinates system from the chessboard coordinate, if with θ _x, θ _y, θ _zRepresent respectively the axle around x, the y axle, the rotation amount of z axle can be expressed as

$R=R_x\left({\mathrm{\&theta;}}_x\right)R_y\left({\mathrm{\&theta;}}_y\right)R_z\left({\mathrm{\&theta;}}_z\right)$

$=\left[\begin{array}{ccc}\mathrm{cycz}& -\mathrm{cysz}& \mathrm{sy}\\ \mathrm{sxsycz}+\mathrm{cxsz}& -\mathrm{sxsysz}+\mathrm{cxcy}& -\mathrm{sxcy}\\ -\mathrm{cxsycz}+\mathrm{sxsz}& \mathrm{cxsysz}+\mathrm{sxcz}& \mathrm{cxcy}\end{array}\right]$

Wherein, c (x|y|z)=cos (θ _(x|y|z)), s (x|y|z)=sin (θ _(x|y|z))

In order to ask for optimum R and T, use Gauss's Newton iteration method to make the following formula result minimize:

Here the rotation amount translational movement all can be initialized as zero, after the several times iteration, following formula is convergence rapidly, during less than setting threshold, has namely obtained the best fit approximation solution of required rotation matrix and translation matrix when following formula.

To find the solution that the R that obtains and T can obtain being tied to camera coordinates from the chessboard coordinate be the homogeneous transformation matrix by above

For

$A_{\mathrm{board}}^{\mathrm{cam}}=\left[\begin{array}{cc}\mathrm{<figure-callout\; id="0"\; label="R\; T"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">R</figure-callout>}& T\\ 0& 1\end{array}\right]$

Step 4: try to achieve transformation relation between camera coordinates in chessboard coordinate system and mirror system according to the mirror image symmetric relation.As Fig. 4, if camera coordinates is the (θ that is transformed to respect to the chessboard coordinate system _x, θ _y, θ _z, t _x, t _y, t _z), the picture that becomes through chessboard coordinate system oxy face of camera is with respect to the first conversion (θ of chessboard coordinate system _x,-θ _y, θ _z, t _x, t _y,-t _z), then the z axle is reverse again, namely obtains the camera imaging coordinate system, chessboard coordinate system O ₀-xyz process rotational transform matrix R ', translation transformation T ' conversion, and then reciprocal transformation obtains coordinate system O through the z axle ₂-xyz.

Can obtain R ' according to the parity of sine and cosine, be shown below:

$R^{\&prime;}=R_x(-{\mathrm{\&theta;}}_x)R_y(-{\mathrm{\&theta;}}_y)R_z\left({\mathrm{\&theta;}}_z\right)$

$=\left[\begin{array}{ccc}\mathrm{cycz}& -\mathrm{cysz}& (-1)\&times;\left(\mathrm{sy}\right)\\ \mathrm{sxsycz}+\mathrm{cxsz}& -\mathrm{sxsysz}+\mathrm{cxcy}& (-1)\&times;(-\mathrm{sxcy})\\ (-1)\&times;(-\mathrm{cysycz}+\mathrm{sxsz})& (-1)\&times;(\mathrm{cxsysz}+\mathrm{sxcz})& \mathrm{cxcy}\end{array}\right]$

By observing following formula as can be known, R ' matrix for the R matrix, is only that wherein four elements have been done the computing of sign symbol, therefore be easy to obtain R ' matrix by the R matrix.In like manner, T ' also is easy to obtain by T, has only done a sign reversing at the translational movement of z direction.

Therefore can have following formula to describe by chessboard coordinate system transformation to the homogeneous transformation matrix of the coordinate system of camera imaging

$A_{\mathrm{board}}^{{\mathrm{cam}}^{\&prime;}}=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}{R}^{\&prime;}& T^{\&prime;}\\ 0& 1\end{array}\right]$

The relation between coordinate system of looking like of now having determined to become in chessboard coordinate system and camera and mirror thereof is used respectively homogeneous matrix With

Expression, camera is tied to the homogeneous transformation matrix use of camera coordinates system as coordinate

Expression, for:

$A_{{\mathrm{cam}}^{\&prime;}}^{\mathrm{cam}}=A_{\mathrm{board}}^{\mathrm{cam}}{A}_{{\mathrm{cam}}^{\&prime;}}^{\mathrm{board}}=A_{\mathrm{board}}^{\mathrm{cam}}{\left(A_{\mathrm{board}}^{{\mathrm{cam}}^{\&prime;}}\right)}^{-1}$

Step 5: use the black triangle scaling board, using method determines that camera is as the relation between coordinate system and two-dimensional laser coordinate system.

As Fig. 5, detailed step wherein is as follows:

5-1. the picture that black triangle scaling board ABC becomes on image is made as abc, extracts by profile and approaches leg-of-mutton method and obtain a, b, c three summits.

5-2. laser scanning plane and scaling board AB, the AC limit meets at M, and the N point carries out Treatment Analysis to laser data, be easy to find M, and the position of N point under laser coordinate system, with M, the N spot projection is made as m to the virtual camera imaging plane, the n point.From a m, n is to ab, and the distance on ac limit can obtain by following formula:

$d(m,\mathrm{ab})=\frac{\left|\right|\stackrel{\&RightArrow;}{\mathrm{mb}}\&times;\stackrel{\&RightArrow;}{\mathrm{ab}}\left|\right|}{\left|\right|\stackrel{\&RightArrow;}{\mathrm{ab}}\left|\right|},d(n,\mathrm{ac})=\frac{\left|\right|\stackrel{\&RightArrow;}{\mathrm{nc}}\&times;\stackrel{\&RightArrow;}{\mathrm{ac}}\left|\right|}{\left|\right|\stackrel{\&RightArrow;}{\mathrm{ac}}\left|\right|}$

If O ₃-xyz coordinate is tied to O ₂The transformation relation of-xyz is R ' ', T ' ', and the spot projection under laser coordinate system is to being obtained by following formula as the plane:

P _c=[u v 1] ^T＝M(R′′P _L+T′′)

Can obtain some groups of data (M by the pose that changes scaling board _i, N _i, a _i, b _i, c _i), if can find one group of R ' ', T ' ' makes M _i, N _iSubpoint m _i, n _iFrom a _ib _i, a _ic _iThe limit is as far as possible little, is equivalent to following formula:

As long as gather abundant sample (M _i, N _i, a _i, b _i, c _i), can be with R ' ', T ' ' initial value all is made as zero, uses Gauss's Newton iteration method to make following formula converge to setting threshold by the several times iteration, thereby obtain rotation matrix R ' ' and translation matrix T ' '.

Laser coordinate is tied to the homogeneous transformation matrix of camera coordinates in mirror system and is:

$A_{\mathrm{laser}}^{{\mathrm{cam}}^{\&prime;}}=\left[\begin{array}{cc}{R}^{\&prime;\&prime;}& T^{\&prime;\&prime;}\\ 0& 1\end{array}\right]$

Step 6: by the position orientation relation between camera and its picture and the position orientation relation between camera picture and two dimensional laser scanning instrument, obtain the rotation translation transformation matrix between camera coordinates system and two dimensional laser scanning instrument coordinate system:

$A_{\mathrm{laser}}^{\mathrm{cam}}=A_{{\mathrm{cam}}^{\&prime;}}^{\mathrm{cam}}{A}_{\mathrm{laser}}^{{\mathrm{cam}}^{\&prime;}}=A_{\mathrm{board}}^{\mathrm{cam}}{A}_{{\mathrm{cam}}^{\&prime;}}^{\mathrm{board}}{A}_{\mathrm{laser}}^{{\mathrm{cam}}^{\&prime;}}$

$=\left[\begin{array}{cc}\mathrm{<figure-callout\; id="0"\; label="R\; T"\; filenames="HDA00002880663300022.png"\; state="\{\{state\}\}">R</figure-callout>}& T\\ 0& 1\end{array}\right]{\left(\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& -1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}{R}^{\&prime;}& T^{\&prime;}\\ 0& 1\end{array}\right]\right)}^{-1}\left[\begin{array}{cc}{R}^{\&prime;\&prime;}& T^{\&prime;\&prime;}\\ 0& 1\end{array}\right]$ 。

Claims (4)

1. the camera of a zero lap visual field and the scaling method of two-dimensional laser, is characterized in that, comprises two dimensional laser scanning instrument, camera, level crossing etc.

2. the camera of a kind of zero lap claimed in claim 1 visual field and the scaling method of two-dimensional laser, is characterized in that, camera and two dimensional laser scanning instrument zero lap visual field.

3. the camera of a zero lap visual field and the scaling method of two-dimensional laser, is characterized in that, comprises following steps:

Step 1: adopt pinhole camera modeling to consider the lens distortion factor, the intrinsic parameter that uses traditional chessboard form to carry out camera is demarcated, and obtains the Intrinsic Matrix M of camera.

Step 2: reasonable disposition laser scanner, camera, level crossing, and paste a chequered with black and white chessboard form on level crossing.

Step 3: determine that minute surface chessboard coordinate system and camera coordinates are position orientation relation.Timing signal, the chessboard paper plane is positioned at the viewing field of camera scope, use the harris angular-point detection method determine in angle point at the coordinate of image coordinate system, use Gauss's Newton iteration method to try to achieve rotation matrix R and translation matrix T between chessboard coordinate system and camera coordinates system.

Step 4: can try to achieve in chessboard coordinate system and mirror camera as the transformation relation between coordinate system according to the mirror image symmetric relation.Thereby can determine the transformation relation between camera in actual camera and mirror.

Step 5: use the black triangle scaling board, determine that camera is as the relation between coordinate system and two dimensional laser scanning instrument coordinate system.

Step 6: by the position orientation relation between camera and its picture and the position orientation relation between camera picture and two dimensional laser scanning instrument, obtain the rotation translation transformation matrix between camera coordinates system and two dimensional laser scanning instrument coordinate system.

4. the camera of a kind of zero lap claimed in claim 3 visual field and the scaling method of two-dimensional laser, is characterized in that, step 5 concrete steps are as follows:

5-1. reasonably put scaling board, make that in two dimensional laser scanning instrument and mirror, camera can detect scaling board simultaneously.

5-2. extract by profile and approach three summits that leg-of-mutton method obtains scaling board imaging in camera, two summits of the intersection of the detection laser plane of scanning motion and scaling board two points (laser coordinate system) that three summits (image coordinate system) of detecting in image and laser and scaling board edge is crossing are as a sample.

5-3. repeating step 5-1,5-2 obtains enough samples, sets the initial value of rotation amount and translational movement, uses Gauss's Newton iteration to carry out optimal estimation, is rotated and translation matrix.

Images