TWI659390B - Data fusion method for camera and laser rangefinder applied to object detection - Google Patents

Abstract

A data fusion method for a camera and a laser rangefinder applied to object detection, based on image processing combined with a laser rangefinder, the image processing is a method of fusion of geometric features and laser ranging data to reconstruct The 3D size of the object.

Description

Data fusion method for camera and laser rangefinder applied to object detection

The invention is aimed at making an autonomous robot to operate an object or perform a task in an unknown environment, and it is necessary to reconstruct a three-dimensional scene of the environment in which the robot is located. The robot must be designed as a three-dimensional object or scene with a cognitive environment outside the ability to move. Reconstructing the perception system is a basic problem in the research of machine vision. The stereo camera system is designed to reconstruct the 3D information of the scene. The present invention is a 3D image processing method for reconstructing a scene that combines laser ranging and image features.

The advantages of conventional 3D image capture are large range, high resolution and short data capture time. At the same time, color information can be obtained, and the volume of the orientation device is gradually reduced. The quality of the scene depth reconstructed by image processing depends on several factors Characteristics, such as lighting conditions, the texture of objects in the scene, and the complexity of objects in the scene, etc. In general, searching for the corresponding points of a stereo image on the edges and textured areas will get quite good results, but in the absence of images The feature area will fail, so using stereo images to reconstruct the depth of the surface of the object without texture features can not get good results. The laser rangefinder can directly obtain the distance of the points, plus line scanning or area scanning, you can get two Dimensional or three-dimensional distance without being affected by ambient light sources.

Cameras and laser rangefinders offer different advantages for different tasks. Cameras can be used to identify the geometry or color of objects, and laser rangefinders can easily obtain depth of field information. Although the three-dimensional laser rangefinder can measure three-dimensional information in space, it is expensive It is expensive, so the present invention hopes to fuse the cheaper 1D laser ranging measurement data with the camera image to reconstruct the 3D information of the scene.

Because the position and shape of the objects placed on the production line are random, how to fuse the data of the laser ranging scan with the image to reconstruct the 3D size and position of the object is the focus of the invention. In the machine tool industry, the invention combines machinery The arm can increase the fluency of the production line and the possibility of automation, achieve high productivity and high accuracy, and reduce personnel costs. The distance obtained by laser scanning can be calculated by the geometric relationship between the measured point and the reference plane The image processing can obtain the edge features of the scene. If the corresponding points of the feature points and the laser scan can be obtained, the 3D information of the feature points can be combined to reconstruct the 3D scene. Therefore, the present invention discusses a method for processing image features. Then, a method of fusion of laser scanning information and image feature points is proposed.

The above-mentioned edge detection is to identify points with obvious changes in brightness in the image. Significant changes in image attributes usually reflect important events and changes in attributes. These include (1) depth discontinuities, (2) surface discontinuities, (3) changes in material properties, and (4) changes in scene lighting. Edge detection is in black and white images, usually identified by the gradient of the image. Detection methods It can be roughly divided into the local extreme value of the first derivative function and the zero-crossing point of the second derivative function to identify the edge region.

Point, line, and circle detection algorithms are being extensively researched. The most common algorithm is the Hough transform algorithm. Most algorithms use feature points on the circumference to detect concentric circles. According to the region segmentation method, different areas of concentric circles are divided to obtain Center and radius, find 2 characteristic points in a circle, then connect the two points to get a chord, and according to the geometric characteristics, that is, the vertical bisector of any chord on the circle will pass the center of the circle.

Therefore, the appropriate constraint conditions need to be set according to the algorithm of the feature point, and have certain limitations. The algorithm based on the feature point has bad anti-interference and high requirements for preprocessing. Therefore, when there is noise, distortion or discontinuous edges, it will increase the complexity and time of the operation.

Some people have proposed a method based on different positions of correction patterns and related physical constraint calibration, integrating features into a two-dimensional plane to minimize the distance between different sensor features; or, other methods are based on obstacle correction In the process, specific patterns are detected in different sensors, allowing the detection of physical constraints. The use of a calibration object with a CAD model allows the matching to be performed by a single frame. Based on the triangular pattern correction, a circle-based camera using a monocular camera is proposed. Similar system for shape pattern correction.

Most of the above is through the use of a specific pattern for calibration, which involves the definition of the driver or the user, which makes it impossible to perform the calibration at any time or at any place. The calibration is limited to some specific times and places, especially the special The conditions require or require manual operation.

The three-dimensional measurement system used to solve the above problems is very expensive and the architecture is very complicated, so it is not commonly used.

Therefore, the present invention applies a data fusion method of a camera and a laser rangefinder for object detection, and uses a pitch-actuated laser rangefinder (PALRF) in combination with a camera to complete the design and reconstruction of a three-dimensional depth image. The PALRF technology uses a The laser rangefinder is installed on the axis of the pitch actuator. In each set angular increment, the laser rangefinder will capture one-dimensional scanning depth information, and these vectors will be projected to a local coordinate. System, for the purpose of generating a 3D image.

The data fusion method of a camera and a laser rangefinder applied to object detection according to the present invention uses a laser rangefinder and a camera, and relates to methods such as image capture, image processing, edge extraction, and laser. Method for rangefinder line plane correction and projection to stereo geometry For spatial algorithms, the main methods of image processing are to use grayscale adjustment, binarization, morphological operations, and graphic segmentation methods to identify and analyze objects in the image. Laser ranging is measured using laser ranging data. Conversion and correction are used to fuse with the geometric features obtained by image processing. By looking for the correspondence between the laser rangefinder scan line and the camera image, we can find the image features corresponding to the laser scan points, and then match these matching features. The image is compared with the engineering drawing file of the object to calculate the surface contour, center and depth of the object. The obtained fusion algorithm can use 2D object images and 1D laser ranging data to reconstruct the 3D object. size.

1‧‧‧Image reading

2‧‧‧RGB to HSV color conversion

3‧‧‧Critical value selection

4‧‧‧ binarization

5‧‧‧ Morphology

6‧‧‧Image enhancement

8‧‧‧ contour tracking

9‧‧‧ analysis

Figure 1: Image processing flowchart

Figure 2: The relationship between image projection and laser ranging

Figure 3: Schematic of 3D scanning

Figure 4: Contour map

Figure 5: Relationship between projection points in three-dimensional space

Figure 6: Method for reconstructing a 3D scene by combining a camera and a laser rangefinder

Figure 7: Measurement scenario

Figure 8: Height curve of the measurement scene represented by α and γ

Figure 9: Contour map of the measurement scene represented by α and γ

Figure 10: Height curve of the measurement scene in world coordinates

Figure 11: Contour map of the measurement scene in world coordinates

Figure 12: Point cloud diagram of laser ranging measurement data

Figure 13: Point cloud image fusion of image pixels and laser measurement data

Figure 14 (a): Cross-shaped object diagram

Figure 14 (b): Figure of angular object

Figure 14 (c): Multi-object situation

Figure 14 (d): The situation of the object is out of range

Figure 15 (a): the result of image processing for glyph objects

Figure 15 (b): The image processing result of the object

Figure 15 (c): Image processing results for multiple objects

Figure 15 (d): The result of the image processing beyond the scope

Figure 16 (a): X-direction gradient map of object gray scale

Figure 16 (b): Y-direction gradient diagram of the gray scale of the cross-shaped object

Figure 16 (c): X-direction gradient diagram of the gray scale of a cross-shaped object

Figure 16 (d): Y-direction gradient diagram of the gray scale of a cross-shaped object

Figure 16 (e): X-direction gradient map of multi-object gray scale

Figure 16 (f): Y-direction gradient of multi-object gray scale

Figure 16 (g): X-direction gradient map of the gray scale of the object out of range

Figure 16 (h): Y-direction gradient of the gray scale of the object out of range

Figure 17 (a): Reconstruction results of shadow-shaped objects for cross-shaped objects

Fig. 17 (b): operation diagram of the cross-shaped object through the mask of Fig. 15 (a)

Figure 17 (c): Reconstruction result of shadow shape for angular object

Fig. 17 (d): the calculation diagram of the angular object through the mask of Fig. 15 (b)

Figure 17 (e): Reconstruction results of shadow shaping for multiple objects

Fig. 17 (f): the calculation diagram of multiple objects passing through the mask of Fig. 15 (c)

Figure 17 (g): Reconstruction result of shadow forming for objects that are out of range

Fig. 17 (h): The calculation diagram of the object out of range after masking in Fig. 15 (d)

Fig. 18 (a): Height curve expressed by α and γ

Figure 18 (b): Altitude curve in world coordinates

Figure 18 (c): Contour maps expressed by α and γ

Figure 18 (d): Contour map in world coordinates

Figure 19 (a): Height graphs expressed by α and γ

Figure 19 (b): Height curve in world coordinates

Figure 19 (c): Contour maps represented by α and γ

Figure 19 (d): Contour map in world coordinates

Fig. 20 (a): Height graphs expressed by α and γ

Figure 20 (b): Height curve in world coordinates

Figure 20 (c): Contour maps represented by α and γ

Figure 20 (d): Contour map in world coordinates

Fig. 21 (a): Height graphs expressed by α and γ

Figure 21 (b): Altitude curve in world coordinates

Figure 21 (c): Contour maps represented by α and γ

Figure 21 (d): Contour map in world coordinates

Figure 22 (a) Laser scanning point cloud image of a cross-shaped object

Figure 22 (b) Point cloud image of data fusion of cross-shaped object

Figure 22 (c) Laser scanning point cloud image of angular objects

Figure 22 (d) Point cloud image of data fusion of angular objects

Figure 22 (e) Laser scanning point cloud image of multiple objects

Figure 22 (f) Point cloud image of multi-object data fusion

Figure 22 (g) Laser scanning point cloud image of object out of range

Figure 22 (h) Point cloud diagram of data fusion of objects out of range

Data fusion method of camera and laser rangefinder applied to object detection. The method is as follows: the image processing flow of detecting the outline of the object is shown in Figure 1. The original scene image reading 1 is usually affected by the light source. Generate background images with different brightness. In order to obtain the contour features of the target, first convert the image format, then use the color segmentation method to perform RGB to HSV color conversion2, set the threshold value to 3, then binarize4, and then The effect of light source unevenness will remove the noise from the binarized color segmented image through the opening and closing method of morphological operation 5, and strengthen the image to enhance the outline of 6 lines through the sharpening filter. Finally, use Canny operator to extract edge features And all the pixels are marked in a linked manner for contour tracking 8 and then analysis of 9 contours. 2, the center c of the lens of the camera, the laser light source is placed at the point c, the laser light can be obtained from a distance measuring method and emission scanning center point of the scene for the scene camera via a video projection relationship, FIG. L Projection of scene L, available Means. The distance z ( α ) obtained by the laser scanning scene L can be expressed by the following first formula: Where α is the laser scanning angle, so as long as the value measured by the laser rangefinder is multiplied by cos α , the distance between the object L and the reference plane can be obtained. Therefore, the object L measured by the laser rangefinder and The relationship of the reference plane can be expressed by the following second formula: P ( z _i , α _i ) = P _ref -LData ( α _i ) × cos α _i

In the second formula above, i is the sampling point, P _ref is the distance between the laser rangefinder and the reference plane, P ( z _i , α _i ) is the distance between the measurement point and the reference plane, and LData ( α _i ) is the lightning If the measurement value of the radio rangefinder at the sampling angle α _i can be combined with the projection geometry of the camera to find the corresponding point between the image l and the laser scan Z ( α _i ), then the image plane can be fused. The distance of the line segment l from its corresponding object length L to reconstruct the 3D information of the scene.

As shown in Figure 3, the 2D relationship between the image projection of Figure 2 and laser ranging is extended to 3D. If the laser rangefinder can be controlled to have the ability to move vertically with the laser scanning direction, then three-dimensional measurement of the scene is performed It is possible, as shown in FIG 3 we drive the stepping motor to tilt the scanning laser rangefinder, laser scanning plane in each three-dimensional space may be projected, the laser scan plane through the drive of the stepping motor pitch angle γ _i is known, and the distance P ( z _i , α _i , γ _i ) obtained by the measurement object S is used to represent the distance between the measurement object and the reference plane, as shown in the following third formula: P ( z _i , α _i , γ _i ) = P _ref -LData ( α _i , γ _i ) × cos α _i × cos γ _i

In the above formula, γ _i is the angle of the stepping motor, and LData ( α _i , γ _i ) is the measurement value of the laser rangefinder at the sampling angle α _i .

After scanning the object with the laser rangefinder in the manner shown in Figure 3, the measurement data obtained is converted in the third form and corresponds to the grid represented by α _i and γ _i . The contour curve of the object can be obtained, as shown in Figure 4 As shown, dense contour lines indicate that the curvature of the surface of the object changes greatly. The results of the gradient calculation and edge detection of the image of the object can be fused with the contour lines to find the corresponding points of the laser scan point of the object and its image.

Generally speaking, the laser light source cannot overlap the center of the camera lens, and there will be an offset b between them as shown in Figure 5. Point O is the camera projection center and the origin of the camera coordinate system. The base distance b is known, and the Z axis and the camera's optical axis overlap. Therefore, the image plane is located at the focal length f , and the coordinate P of the target point P in the camera coordinate system is ( X _O , Y _O , Z _O ). The two-dimensional coordinates on the image plane are p ( x, y ).

According to the pinhole imaging principle, the following fourth formula can be obtained: The following fifth formula is obtained by using a trigonometric function: And from the fourth and fifth formulas, the following sixth formula is obtained: After finishing, the following seventh formula can be obtained:

The eighth formula of the three-dimensional coordinate P ( X _O , Y _O , Z _O ) of the P point is as follows: In the above formula, α is the scanning angle of the laser rangefinder in the X direction, the x and y values of the p point can be obtained from the camera parameters, and the focal length f value can be obtained from the following ninth formula: In the above formula, l is the measurement value of the laser rangefinder, and the tenth formula is obtained as follows:

The method of reconstructing a 3D scene by combining a camera and a laser rangefinder is as shown in FIG. 6. The distance L ( z, α, γ ) obtained by the laser ranging measurement is used in conjunction with the known α and γ. The contour line of the scene can be obtained by using the formula. The image of the scene captured by the camera can be used to obtain the edge lines of each object in the scene. If the contour points can be obtained from the contour points, you can use the tenth formula to reconstruct the 3D scene.

Through the parameters of the camera and the laser rangefinder, such as the focal length f of the camera, the resolution of the laser rangefinder and the micro-stepping motor, etc., the above method can be used to fuse the camera image and the data of the laser rangefinder. First obtain the current position of the micro-stepping motor, the distance measured by the laser rangefinder, and the camera image, and then perform a corresponding point search to obtain the fused data.

Because the actual grid size (△ X, △ Y ) of the interval projection of laser scanning points in world coordinates can be expressed by the scanning angle α and elevation angle γ of the laser, △ X = Ldata × (tan ( α _{i +1} ) -tan ( α _i )), △ Y = Ldata × (tan ( γ _{i +1} ) -tan ( γ _i )), we can know that the actual grid length △ X and width △ Y are different between the laser scanning points. Distance, so the Ldata obtained by laser scanning should be placed at the coordinates of ( X, Y ), and the scanning coordinates ( α, γ ) must be converted into spatial coordinates ( X, Y ). The relationship between the image coordinates and the real-world coordinates ( X, Y ) can be obtained from the ratio of the size of the reference object to the image pixel value of the reference object, and the image coordinates ( x, y ) are converted into spatial coordinates ( X, Y ), The pixels of the image can be matched with the ranging value of the laser scan.

As shown in Figure 7, the rectangular object is placed in front of the reference plate, the scan range α of the laser rangefinder is set to be negative 8 degrees to positive 32 degrees, and the pitch angle of the laser rangefinder is driven by the stepping motor. The gamma range is plus or minus 18 degrees. The measured laser distance measurement value is firstly extended by one-dimensional interpolation to 564 * 456 records, and then converted by the third formula to obtain a height curve based on the reference plane, such as Figure 8 shows the height curve of the measurement scene represented by α and γ . In Figure 8, there are a total of 564 laser scans obtained by laser scanning. The x-axis direction is the α angle and the y-axis direction. gamma] is the angle, the distance from the reference plane is set to 0, the trajectory of the laser scan line in each row, the scene to be measured can be obtained by gamma] and [alpha] defined point from the reference plane _{P (z i, α i,} γ _i ). After the laser measurement distance L ( z, α, γ ) is processed in the third formula, and the plane coordinates expressed by α and γ are used, the contour lines of the scene can be obtained as shown in Figure 9, and the measurements expressed by α and γ The contour map of the scene can be clearly seen from the figure. The dense area represents a large change in height, which is the edge line of the object.

After converting the grids of α and γ in FIG. 8 to world coordinates, the laser distance measurement curve obtained by the line scan of 564 laser rangefinders can be obtained. As shown in FIG. 10, the quantity expressed in world coordinates The height curve of the scene is measured, and the distance of the reference plane is set to 0. With the coordinates of the plane represented by x and y , the contour map of the scene in the world coordinates can be obtained as shown in Figure 11. In the contour map, it can be clearly seen in FIG. 11 that the dense areas of the contour lines represent a large change in height. As the edge line of the object, the edge size and the plane height of the object can be obtained.

Next, the laser ranging measurement data is expressed in the form of a point cloud diagram. As shown in the point cloud diagram of the laser ranging measurement data shown in FIG. 12, the surface contour and size of the rectangular object to be measured can be seen. There is no texture information on the surface of the measured object, and the material and characteristics of this object cannot be discriminated. Therefore, the fusion method shown in Fig. 6 will be used to obtain the contour map of the data measured by laser ranging with world coordinates. The edge features obtained from the binarized image are matched to obtain a point cloud image as shown in Figure 13 which is the fusion of image pixels and laser measurement data. Figure 13 is based on parameters such as the bounding box and centroid of the object to be measured. The distance measurement data obtained from the radio scan are compared, and feature matching is performed with the data amount of the same resolution, and the non-matching parts are removed. Therefore, in Figure 13, only 520 * 434 data were successfully merged. The obtained point cloud image can completely obtain the surface texture characteristics and surface size of the object to be measured. The fusion result of Fig. 13 includes the number of detection points, the number of matching error points, and the success rate. FIG laser scanning spot 13 and the rectangular object image pixel integration results in the following table: The calculation of success rate is as follows: Multi-object 3D surface reconstruction results, as shown in Figure 14, the tests in this section will explore cross-shaped objects, angular objects, multiple objects, and cases where objects are out of range, as shown in Figure 14 (a) (b ) (c) (d). As described above, the image processing algorithm and resampling method are used to extract object features. The processed binarized image is shown in FIG. 15, and the processed result of the cross-shaped object image in FIG. (a); and, FIG. 14 (b) is an angular object, and its image processing result is 15 (b); and, FIG. 14 (c) is a multi-object, its image processing.

The results are shown in Fig. 15 (c) and Fig. 14 (d). The objects are out of range, and the results after image processing are shown in Fig. 15 (d).

The image characteristics of the object to be measured obtained by image processing are shown in Table 2. It includes parameters such as chroma, saturation, brightness, bounding box, centroid, and area, and then calculates the X-direction and Y-direction gradients of the grayscale image in FIG. 14 As shown in FIG. 16, the gradient values obtained in FIGS. 16 (a) and 16 (b) are mapped to a combination of integrable basic functions in the frequency domain by Fourier transformation and the global integral calculation is performed based on the Fourier transform. The image depth calculation method is shown in Figure 17 (a). The image characteristics of the object to be measured are shown in the following table:

FIG. 17 (b) is the result after recalculating FIG. 17 (a) using FIG. 15 (a) as a mask. Similarly, FIG. 16 (c), FIG. 16 (d), FIG. 16 (e), and FIG. The gradient values obtained from 16 (f), 16 (g), and 16 (h) are reconstructed using shadow shaping algorithms. The results are shown in Figs. 17 (c), 17 (e), and 17 (g), respectively. As shown in Fig. 17 (d), Fig. 17 (f), and Fig. 17 (h) are the results after mask recalculation in Fig. 15 (b), Fig. 15 (c), and Fig. 15 (d), respectively. From Figures 17 (a), 17 (c), 17 (e), and 17 (g), it can be seen that although the surface height of the object to be measured can be reconstructed by the shadow forming algorithm, it is susceptible to surface texture changes. Influence, which leads to the reconstruction error situation.

For example, the black and white grids of the reference plate are reconstructed at the highest point and the lowest point respectively. Therefore, in this paper, the binarized image after image processing is used as a mask, and the mask calculation is performed on the reconstruction result of the shadow forming algorithm, as shown in Figure 17. (b), Fig. 17 (d), Fig. 17 (f), and Fig. 17 (h).

In Fig. 17 (c), there is an angular object to be measured. Using the result of the shadow forming algorithm, the edge of the corner object is reconstructed to a height, but the oblique part is reconstructed into a plane with a slope opposite. In Figure 17 (e), the position of three square objects is improperly placed. You can see the side of the square object. The shaded part is reconstructed on the negative height, and the front part is reconstructed on the positive height. As shown in Figure 17 (g), the triangular objects in the upper left corner and the square objects in the lower right corner are darker brown objects, which are reconstructed at a negative height, while the other two lighter square objects are reconstructed at At a positive height, it can be seen that this algorithm is greatly affected by color and texture, and it is easy to fail to reconstruct corner objects. As shown in Figure 17 (a), because the image obtained by this object only captures the front part , And the color and texture of the front face change little, so the shadow height algorithm can be used to successfully reconstruct the surface height of this object.

After analyzing the problem of the shadow forming algorithm, the method of the present invention is then used to reconstruct the 3D surface of the object. As described in the previous section, the laser sensor is used to perform a 2D surface scan of the measurement scene in FIG. 14 to capture the captured image. After taking the laser ranging measurement data and interpolating and expanding it to 564 * 456 records, use the third formula to convert the measurement data into a height curve represented by α and γ, as shown in Figure 18 (a) and Figure 19 respectively. (a), Fig. 20 (a) and Fig. 21 (a), set the distance of the reference plane to 0, and coordinate with the plane coordinates represented by α and γ to obtain the contour maps of the scene as shown in Figure 18 (c). 19, (c), 20 (c) and 21 (c). Because the actual grid sizes represented by α and γ are not equidistant, they need to be converted into height curves represented by world coordinates x and y, as shown in Figure 18 (b), Figure 19 (b), and Figure 20 (b). And as shown in Fig. 21 (b), the distance of the reference plane is set to 0, and with the plane coordinates represented by x and y , the contour maps of the scene in the world coordinates can be obtained as shown in Fig. 18 (d) and Fig. 19 (d) As shown in Figures 20 (d) and 21 (d), it can be clearly seen from the figure that the dense contours represent a large change in height, which is the edge line of the object.

The laser ranging measurement data of Fig. 18 (a), Fig. 19 (a), Fig. 20 (a), and Fig. 21 (a) are expressed in the form of point cloud diagrams, as shown in Fig. 22 (a) and Fig. 22 ( c), as shown in Figure 22 (e) and Figure 22 (g), the surface profile and size of the object to be measured can be seen, but without any texture information, the materials and characteristics of these objects cannot be discriminated, so the present invention The fusion method shown in Figure 22 (b), Figure 22 (d), Figure 22 (f), and Figure 22 (h).

The parameters such as the bounding box and centroid of the object to be measured are compared with the distance measurement data obtained by laser scanning, and feature matching is performed with the data amount of the same resolution, and the non-matching parts are removed to obtain a fused The results are shown in Figure 22 (b), including 509 * 410 records, Figure 22 (d) 486 * 374 records, Figure 22 (f) 529 * 411 records, and Figure 22 (h) 519 * 411 records. If the matching is successful, the surface texture features, surface size and surface height of the object to be measured are obtained from the point cloud images of these fusion points. The fusion results of Figure 22 (b), Figure 22 (d), Figure 22 (f), and Figure 22 (h) are compared according to the number of detected points, the number of mismatched points, and the success rate. Laser scanning of different objects The result of the fusion of points and image pixels is shown in the following table:

As described above, the image (2D) obtained by the camera shooting the object (3D) loses the depth information of the object; and the distance obtained by laser ranging is the distance from the object to the laser light source, and the algorithm is:

a. Develop an algorithm to convert the object distance obtained by laser ranging to the contour curve of the reference surface according to the laser scanning angle.

b. Develop an algorithm to find the geometric characteristics of the measurement object

c. Develop an algorithm to fuse a and b to measure the 3D information of the object.

Its fusion calculation procedure is:

1. In the part of image processing, first use the color segmentation method to segment the outline of the object to be measured, filter out noise with morphological operations to obtain a labeled binary image, and then use the magnitude and direction of the gradient. Use global integration to calculate the depth of the image, and use the binary image segmented by the object as a mask to perform the masking operation. The resulting 3D reconstructed image formed by the shadow is more likely to appear due to changes in texture or special shapes. Rebuilding the wrong situation.

2. The present invention uses the binarized image and its feature parameters segmented by the object to compare with the edge features of the contour map such as the bounding box and centroid, and performs feature matching with the same amount of data. The unmatched parts are removed, and the result of the fusion is presented in the form of a point cloud image. The data fusion method of the present invention can convert the data obtained by laser ranging into contour lines based on the reference plane and edge lines obtained from the image. Corresponding point matching is performed to successfully fuse the laser ranging data with the pixels corresponding to the camera image.

3. From the experiments of feature extraction and shadow forming algorithms, we can know that the global integration algorithm is used to reconstruct the 3D object surface. If no mask is used for scene filtering of non-test objects, the scene will be measured. And the experimental results of the fusion of laser scan data and image pixels, it can be seen that the length and width of the actual scanning grid between laser scan points are not equidistant, so you need to measure the laser ranging first. The data is converted into contour maps expressed in world plane coordinates. Feature matching is performed with the same amount of data. Then the unmatched parts are removed, and the laser ranging data and image features are displayed in the form of point cloud diagrams. Surface texture characteristics and surface dimensions of the object under test.

The present invention uses rectangular objects, cross-shaped objects, angular objects, multiple objects, out-of-range objects, non-solid objects, and non-solid objects with oblique angles as test conditions for the test scene. The advantages are: (1) in rectangular objects In the experiment, the matching rate was 99.8%, while the matching rates for cross-shaped objects, angular objects, multi-objects, and objects out of range were 98.80%, 99.95%, 97.10%, and 96.36%, respectively. (2) The match rate of one layer is 93.30%. The match rate of the first layer and the second layer of non-solid objects with tilt angles are 94.88% and 98.54%, respectively. (3) The experimental results of 3D surface reconstruction from multiple objects show that the plane Measurement objects based on the method of the present invention can obtain good reconstruction results higher than 95% for non-solid objects and non-solid objects with oblique angles. The texture map reconstruction results are poor. After verifying the designed data fusion method of the present invention, 2D object images and 1D laser ranging data can be used to reconstruct the 3D size of the object.

Claims (3)

A method for data fusion of a camera and a laser rangefinder applied to object detection refers to combining a laser rangefinder based on the image processing of a camera, and the image processing of the camera and scanning by the laser rangefinder Data fusion method: The laser rangefinder uses an algorithm to convert the distance of the 3D object obtained by laser ranging into the contour curve of the reference surface according to the laser scanning angle. The geometry of the measured object is calculated by the algorithm. Feature, the contour curve and geometric features are merged to obtain the 3D contour information of the object. The image processing of this camera uses geometric features to segment the 3D contour of the object to be measured by color segmentation. The operation filters out noise to obtain a labeled binary image, then uses the magnitude and direction of the gradient, uses the global integration to calculate the image depth, and uses the binary image segmented by the object as a mask to perform the masking operation. 3D reconstruction image obtained from the operation; and the binarized image segmented by the object and its feature parameters are performed with the edge feature parameters of the contour map Yes, with the same resolution data volume, feature matching is performed, and the non-matching parts are removed. The result of the fusion is presented in the form of point cloud diagrams, so that these high curves and geometric features are obtained by fusion algorithms to obtain data and cameras. Corresponding pixel fusion of image processing to obtain the 3D size of the object. A data fusion method for a camera and a laser rangefinder applied to object detection. The data fusion method uses a laser rangefinder and a camera. The camera obtains data such as image capture, image processing, and edge extraction. The laser rangefinder line obtains the calculation data of plane correction and projection into the three-dimensional geometric space. The data obtained through image processing uses gray-scale adjustment, binarization, morphological operations, and graphic segmentation methods to identify objects in the image and Analysis, the laser ranging measurement is converted and corrected by laser ranging data to fuse with the geometric features obtained by image processing, find the correspondence between the laser rangefinder scan line and the camera image, and find out the laser distance Scan the corresponding image characteristics of the points, and then compare these matching images with the object's engineering drawing file to calculate the surface contour, center, and depth of the object. The resulting fusion algorithm can convert the 2D image and lightning of the object. Shoot 1D data to get the 3D size of the object. A data fusion method for a camera and a laser rangefinder applied to object detection. The data fusion method uses a laser rangefinder and a camera, and combines the laser rangefinder based on the image processing of the camera. It is characterized in that a pitch-actuated laser rangefinder combined with a camera is used to complete the design and reconstruction of a three-dimensional depth image, and a one-dimensional laser rangefinder is installed on the axis of the pitch actuator in each set angular increment The laser rangefinder will capture 1D scan depth information, and these vectors will be projected onto a local coordinate system to generate a 3D image. The image processing uses gray scale adjustment, binarization, morphological operations, and graphic segmentation methods. For object identification and analysis in the image, laser ranging measurement is converted and corrected with laser ranging data, and the converted and corrected data is then fused with geometric features obtained from camera image processing. Correspondence between the scan line of the telemeter and the geometric features of the camera image processing, find the image features corresponding to the laser scan points, and then map the matching drawings of these images and objects Compare the parameters, calculate the surface contour, center and depth of the object, and use the image processing data and 1D laser ranging data to obtain the 3D size of the object.