CN104574406B - A kind of combined calibrating method between 360 degree of panorama laser and multiple vision systems - Google Patents

Abstract

The invention provides a joint calibration method of a 360-degree panoramic laser and multiple vision systems, belonging to the technical field of autonomous environment perception of mobile robots. The biggest innovation of the present invention is that simple black cardboard is used as the calibration device to quickly realize the simultaneous calibration of multiple vision systems and panoramic lasers. For this reason, the present invention can make the laser beam irradiated on its surface have lower reflectivity characteristics based on the black cardboard. After generating reflection value map, range filtering, binarization and point clustering, the collected three-dimensional laser data Extract the feature points belonging to the calibration device, and use the plane extraction method to filter out the noise objects in the environment, and then use the iterative optimization method to solve the three-dimensional feature points based on laser data and the two-dimensional feature points based on image data, so as to obtain the sensor Between rotation matrix and translation matrix. The invention lays the foundation for multi-sensor information fusion, and can be used in fields such as mobile robot scene reconstruction.

Description

A joint calibration method between 360-degree panoramic laser and multiple vision systems

technical field

The invention belongs to the technical field of environment perception, relates to data fusion between a three-dimensional laser ranging system and multiple visual systems, and particularly relates to a joint calibration method between the three-dimensional laser ranging system and multiple visual systems.

Background technique

In complex scenes, a single sensor cannot meet the task requirements of environment perception and scene understanding. Therefore, data matching and fusion between multiple sensors is a necessary means to improve the performance of environment perception and scene understanding. The joint calibration between multiple sensors is its key step. At present, there are a variety of external parameter calibration methods for 3D laser and monocular vision. The most common method is to use black and white calibration plates to calibrate the external parameters between the 3D laser and the vision system (Joung J H, AnK H, Kang J W, et al .3D environment reconstruction using modified color ICPalgorithm by fusion of a camera and a 3D laser range finder[C],IEEE/RSJInternational Conference on Intelligent Robots and Systems(IROS),2009:3082-3088.), due to the calibration board itself It is used to calibrate the internal parameters of the vision system, so it is convenient for the vision system to process. However, for the 3D laser system, the scanning effect of the black and white calibration plate is very demanding during the calibration process, and this method is only suitable for the calibration of nearby scenes. There are certain restrictions on the scanning angle. In order to facilitate the acquisition of calibration features by the 3D laser system, literature (García-Moreno A I, Gonzalez-Barbosa J J, Ornelas-Rodriguez F J, et al. LIDAR and Panoramic Camera Extrinsic Calibration Approach Using a Pattern Plane[M], Pattern Recognition. Springer Berlin Heidelberg, 2013: 104-113.) Using the edge corners of multiple diamond-shaped holes as features to calibrate the parameters of 3D lidar and panoramic camera, but due to the influence of the edge effect of laser points, the acquisition of corner features is not particularly accurate. Stability cannot be guaranteed. In addition, the literature (ThomasJ Osgood, Yingping Huang.Calibration of laser scanner and camera fusion system for intelligent vehicles using Nelder-Mead optimization[J], MeasurementScience&Technology, 2013, vol.24, no.3, pp:1-10.) uses suspension The circular white card in the air is used as a calibration object for external reference calibration. Although this method is convenient for determining the position of the feature object in the 3D laser data and image, it requires an open experimental environment during calibration, and because the volume of the calibration object is relatively small, The influence of the edge effect of the laser point on the three-dimensional coordinates of the calibration object is amplified, which affects the calibration accuracy. Patent (Zheng Yan; Wang Wei; Chen Dong; Yang Shengpeng, Dalian University of Technology, an automatic calibration method between 3D laser and monocular vision, patent number 200910187344,) proposed to use a black and white calibration plate with holes for 3D laser and monocular vision There are two kinds of information on the board: color information and void information. The color information is obtained by the camera, and the intersection point (i.e., the feature point) of the black and white grid is detected by the relevant corner detection algorithm provided in OpenCV; the hole information is obtained by the laser rangefinder, and the black and white grid is found in the laser data through the relevant algorithm. The intersection point, and then the feature points in the two scenes correspond to complete the matching of the scene. However, this method has certain limitations, requiring a small distance between the camera and the laser range finder, which is not suitable for the situation where the camera and the laser range finder are far away, and can only be applied to the calibration of the laser and a single vision system.

Contents of the invention

The problem to be solved by the present invention is the automatic joint calibration between the three-dimensional laser ranging system and multiple vision systems, and the external parameter calculation of multiple systems can be conveniently realized through only one data collection, which solves the problem of the laser ranging system and the vision system. The limitation that the system cannot be too far away reduces the requirements of the calibration method on the calibration object and the calibration environment, reduces the influence of the edge effect of the laser point on the calibration results, and enhances the practicability and accuracy of the calibration method.

Technical scheme of the present invention is as follows:

1. Design of three-dimensional panoramic laser characteristic analysis and calibration device

The three-dimensional panoramic laser system used in the present invention is composed of a two-dimensional laser sensor and a rotating platform with a stepping motor. The rotating platform rotates on a horizontal plane, and the scanning plane of the two-dimensional laser sensor is fan-shaped and perpendicular to the rotating plane of the platform. , each set of laser data contains ranging data and reflection value data, and the two data correspond to each other. Since the laser sensor has a certain measurement error, and the laser data at the edge of the object is affected by the edge effect, when the distance between the foreground and the background object is relatively close, it is difficult to accurately distinguish the data of the foreground and the background object according to the ranging data; and Reflection value data is affected by various object properties such as object material and color at the same time, and is not limited by the distance between objects, which can facilitate the division of multi-type object data.

According to the characteristics of the above-mentioned 3D panoramic laser, a joint calibration device of 3D panoramic laser and multiple vision systems was designed. The selected material was black cardboard, and the unified shape and specification were cut (as shown in Figure 1). After experimental verification, the choice of black cardboard can make the laser beam irradiated on its surface have a lower reflectivity, design a uniform shape for the calibration device, and facilitate the extraction of subsequent feature points to ensure the robustness of the calibration algorithm and calibration results accuracy. In order to obtain the best calibration effect, the size of the calibration black paper can be properly adjusted according to the visual range of the camera and the density of the laser data.

2. Extraction of laser data feature points required for calibration

The feature point of the laser data to be extracted in the calibration is the center point of the calibration black paper. First, the three-dimensional laser point projected on the calibration black paper needs to be extracted. The extraction of calibration black paper data is divided into two steps: pre-detection and denoising. In the pre-detection part, the reflection value characteristics of the black cardboard are used to find the serial number of the laser point on the calibration black paper from the two-dimensional reflection value map, and determine the laser The 3D coordinates of the point. The denoising link analyzes and processes the acquired 3D laser data to remove falsely detected 3D laser data to ensure calibration accuracy.

The pre-detection steps of the calibration black paper based on laser data are as follows:

1) Generate a binarized reflection value map. The total number of data sets and the number of laser points in each set of data are used as the number of pixels in the x-direction and y-direction to generate a two-dimensional image, so that one laser point corresponds to one image pixel. The gray value of each pixel is obtained by processing the image according to the reflection value of the laser point using formula (1), and then the two-dimensional reflection value map corresponding to the three-dimensional laser point cloud is obtained (as shown in Figure 5(a) Show).

where d _i and g _i are the reflection value and gray value of laser point i respectively, and d _max and d _min are the maximum and minimum reflection values of all laser points.

In order to facilitate processing and avoid more interference, the laser data within a certain range from the three-dimensional panoramic laser where the calibration black paper is located is selected. This range can be adjusted appropriately according to the position of the calibration black paper (as shown in Figure 5(b) ). Binarize the reflection value map:

Where g _i ′ is the pixel gray value after binarization, is the gray mean value of all pixels, and k _g is the gray adjustment threshold, which affects the binarization effect. After binarization, the gray value of the data in the black paper area is 0, and only black and white are left in the picture, making the black paper area clearer (as shown in Figure 5(c)).

2) Cluster the black pixels.

The binary image generated above will have some black noise points. In order to determine the laser points belonging to each calibration black paper and eliminate interference, the neighborhood search method will be used to cluster the black pixels in the binary grayscale image. . The clustering process is as follows:

Clustering Algorithm:

Through the above clustering algorithm, the black pixels are clustered into multiple pixel clusters. Because the calibrated black paper occupies a large area in the binary image, it can be determined whether it is a calibrated black paper by judging the size of the pixel cluster. If the number of points in the pixel cluster is less than a certain range, it is judged as a noise cluster, and then this cluster The gray value of all points in becomes 255. Through the above processing, only the calibration black paper is left in the reflection value map (as shown in Figure 5(d)).

Since the laser ranging data and the reflection value data are in one-to-one correspondence in order, according to the image index ( _xi , y _{i )} of each pixel in the clustered pixel cluster _, the corresponding The serial number of the ranging data, where m is the number of laser points of each set of laser data, so each pixel cluster can find the corresponding laser point cluster.

The above algorithm completes the preliminary detection of the laser point on the calibration black paper. The advantages of this detection method are: using the reflection value of the laser as the detection basis, the calibration black paper does not need to be far away from the background environment, which reduces the requirements for the calibration environment; clustering is carried out on the basis of the two-dimensional reflection value map, compared with the three-dimensional laser point Reduce one-dimensional difficulty.

3) Denoising algorithm of 3D laser feature points

Affected by interfering objects in the environment, not all detected laser point clusters correspond to the calibration black paper. Therefore, the algorithm given below is required to verify the preliminary detected laser point clusters to remove the influence of environmental noise and ensure calibration accuracy.

Since the reflection value of the laser is used as the detection basis, objects with similar colors and materials to the calibration black paper in the environment are the main interference. For this main interference, the flatness evaluation method is used to distinguish between calibration objects and non-calibration objects. Since the laser points belonging to the calibration object are completely on a plane, the non-planar calibration objects can be distinguished by judging whether the plane formed by the laser points is flat. Suppose a laser point cluster contains p=[p ₁ ,p ₂ ,...,p _N ] a total of N laser points, then the coordinate covariance matrix of the laser points is:

in, is the midpoint of the laser point. Calculate the three eigenvalues of the covariance matrix: λ ₀ , λ ₁ , λ ₂ , and when the smallest eigenvalue satisfies λ ₂ <<λ ₁ ≈λ ₀ , the laser points cluster into plane rows. In this case, the flatness of the formed plane can be evaluated by the ratio of the _two eigenvalues λ2 and _λ0 . When the ratio is less than a given threshold, the laser point cluster is considered to belong to the calibration object:

where k _λ is the flatness evaluation threshold.

In addition, when the laser beam hits the glass mirror, due to the combined effect of reflection and refraction, the reflection value of the laser is relatively small. In this case, it is difficult to effectively distinguish it from the calibration black paper in the grayscale image. For this This kind of interference can be started from the characteristics of the laser sensor. After the laser beam passes through the glass mirror, it hits an obstacle and reflects back. The actual measurement distance is much greater than the distance between the glass mirror and the laser sensor. Therefore, when the center point distance of the laser point cluster is less than a certain threshold, the point cluster is considered as the calibration object:

in with is the midpoint of the laser point The three-dimensional coordinates of , k _d is the distance threshold.

3. Iterative calculation of the transformation relationship between 3D laser data and 2D visual data

Since the visual range of the visual sensor is limited, which is much smaller than the measurement range of the 3D laser sensor, when performing calibration, it is possible to design a background with a color difference from the black paper, and then use the pixel grayscale information to extract the black cardboard in the image. For the corresponding pixels, the black pixel point clustering method is used for reference to segment the pixel points, and the average image coordinates of all pixels in each pixel cluster are used as image feature points.

According to the matching pairs of laser data feature center points and visual data feature corner points, the iterative optimization method is used to solve the projective transformation from three-dimensional space to two-dimensional space. Here, the Gauss-Newton iterative method is used to optimize the external parameters. For the convenience of calculation, both image feature points and laser feature points will be represented by homogeneous coordinates. definition is the two-dimensional homogeneous coordinate vector of the image feature point, p _i =[ _xi ,y _i , _zi ,1] is the homogeneous coordinate vector of the three-dimensional laser point, after the laser point is converted, the corresponding pixel point coordinate vector in the image is The purpose of calibration is to calculate a set of transformation parameters makes:

Where f _x , f _y are the focal lengths of the visual sensor in the x and y directions respectively, and (u _x , u _y ) are the offset vectors of the midpoint of the photosensitive element relative to the image center. f _x , f _y , u _x , u _y are the internal parameters of the vision system, which can be obtained by traditional internal reference calibration methods, so here are known quantities. r ₁ , r ₂ , r ₃ are the rotation angles around the x, y, and z axes respectively, and t _x , _ty , t _z are the translation amounts in the directions of the three coordinate axes respectively.

By the Gauss-Newton iterative method, the transformation parameters at time k are given make exist Nearby are:

where the Jacobian matrix Find the next iteration point makes:

make Get the normalization equation:

The iterative format of the Gauss-Newton method substituted into the previous formula:

The iterative format is used to obtain the transformation parameters satisfying the formula (6), and the calibration result is obtained.

The effect and advantage of the present invention is that it can effectively reduce the interference of noise, distance, incident angle, etc. on the calibration between the three-dimensional laser and the visual system, the calibration effect is relatively stable, and the calibration between the three-dimensional panoramic laser and multiple visual systems can be realized at the same time. The practical applicability has been effectively improved. The design of the calibration method is simple, and the calibration device is light in weight and easy to carry. It can be used not only indoors, but also in complex outdoor environments to quickly and correctly calibrate the 3D panoramic laser and vision system, and then realize the 3D laser data and The fusion of visual data realizes scene reconstruction in complex environments and lays a solid foundation for the development of machine intelligence technology based on multi-sensor data fusion.

Description of drawings

Figure 1 shows the black cardboard used for calibration.

Figure 2 is a distribution diagram of equipment, including four vision systems and a panoramic laser

Figure 3 is a picture of the calibrated black paper collected by the four vision systems

Figure 4 shows the three-dimensional laser data collected by the panoramic laser sensor

Figure 5 is the process of extracting the calibration black paper from the laser data. (a) The two-dimensional reflection value map corresponding to the three-dimensional laser point cloud, (b) is the reflection value map within the selected range, (c) is the reflection value map after binarization, (d) is the reflection after clustering processing value graph.

Fig. 6 is to extract the 3D laser points projected onto the human body from the 3D laser data.

Fig. 7 is an effect diagram of back-projecting the laser point in Fig. 5 into four pictures using the calibration results.

detailed description

In order to verify the effectiveness of this method, the sensor system constructed in Figure 2 is used to verify the calibration method. The panoramic laser sensor consists of a Hokuyo UTM-30LX laser sensor and a rotating pan/tilt. The plane scanning angle of the laser sensor is 0-270 degrees, and the frequency range of the stepping motor of the pan/tilt is 500-2500Hz. Use the motor to drive the laser sensor to obtain the three-dimensional laser ranging data of the scene. The four visual systems are all common ANC FULL HD1080P network cameras, using USB2.0 interface, with a viewing angle of 60 degrees and a resolution of 1280×960. The calibration device uses nine 300mm×100mm black papers, which are placed at different positions in the scene.

The pictures of the calibration device in the scene are collected from the four vision systems (as shown in FIG. 3 ), and the calibration device can be extracted from the picture by using the corresponding image processing method. The panoramic laser sensor can obtain the 3D point cloud data of the entire space (as shown in Figure 4) and the reflection value corresponding to each laser point, and then after generating the reflection value map, range filtering, binarization, and point clustering, it can The laser point of the data calibration device is extracted from the laser data (as shown in FIG. 5 ). The rotation and translation matrix can be calculated by iteratively matching the feature points of the two-dimensional calibration device in the picture and the feature points of the three-dimensional calibration device in the laser data.

The obtained external parameters of the 3D panoramic laser ranging system and the four vision systems are:

From a qualitative point of view, the calibration results are visually verified by projecting laser points onto pictures. Taking the person as a reference object, reacquire the corresponding pictures and 3D laser data, extract the laser points belonging to the body part from the 3D laser points (as shown in Figure 6), and use the rotation matrix and translation matrix obtained from the calibration to convert the laser point Projected onto the four pictures to check the calibration effect, the white dots on the human body in the four pictures in Figure 7 are the projected laser dots, it can be seen that the laser dots can be accurately projected onto the corresponding areas of the pictures.

Claims (1)

1. A joint calibration method between a 360-degree panoramic laser and multiple vision systems, characterized in that: simple black cardboard is used as the calibration device, and the laser beam irradiated on its surface has a lower Reflectivity characteristics, extract the laser points belonging to the black cardboard from the two-dimensional reflection value map, and take the midpoint as the feature point, match the corresponding feature points in the images of different vision systems, and iteratively calculate the transformation relationship between the systems , to complete the joint calibration of 3D panoramic laser and multiple vision systems; the specific steps are as follows: a) First cut out several rectangular black cardboard, the length of the cardboard is not less than 30 cm, the width is not less than 10 cm, and randomly placed on different planes in the indoor environment; b) Use the panoramic laser to collect the three-dimensional data and reflection value data of the environment, and use Perform grayscale processing on the image to obtain the two-dimensional reflection value map corresponding to the three-dimensional laser point cloud, where d _i and g _i are the reflection value and gray value of laser point i respectively, and d _max and d _min are the maximum values of all laser points and the minimum reflection value; select the laser data within a certain range from the three-dimensional panoramic laser where the black paper is calibrated, and perform binary processing on the pixels corresponding to the data, so that only black and white are left in the picture, and the black paper area is more Clear; Neighborhood search method is used to cluster the black pixels after binarization to eliminate interference; c) Use the flatness evaluation method to distinguish the calibration object from the non-calibration interference object, assuming that a laser point cluster contains p=[p ₁ ,p ₂ ,...,p _N ] a total of N laser points, then the coordinate coordinates of the laser points The variance matrix is in is the midpoint of the laser point; calculate the three eigenvalues of the covariance matrix: λ ₀ , λ ₁ , λ ₂ , when λ ₂ <<λ ₁ ≈λ ₀ and the ratio is less than a given threshold Where k _λ is the flatness evaluation threshold, it is considered that the laser point cluster belongs to the calibration object; d) Definition is the feature point of the image, p _i =[ _xi ,y _i , _zi ,1] is the three-dimensional laser point, and the coordinate vector of the pixel point in the image corresponding to the laser point after conversion is The purpose of calibration is to calculate a set of transformation parameters Satisfy Where f _x , f _y are the focal lengths of the visual sensor in the x and y directions respectively, (u _x , u _y ) are the offset vectors of the midpoint of the photosensitive element relative to the image center, both of which are known quantities; r ₁ , r ₂ , r ₃ are the rotation angles around the three axes of x, y, and z respectively, t _x , ty _y , t _z are the translation amounts in the direction of the three coordinate axes respectively; the transformation parameters at a given time k make exist nearby where the Jacobian matrix Find the next iteration point makes: make get the normalized equation thus get Using this iterative format to obtain the transformation parameters and obtain the calibration results.