CN114241298A - Tower crane environment target detection method and system based on laser radar and image fusion - Google Patents
Tower crane environment target detection method and system based on laser radar and image fusion Download PDFInfo
- Publication number
- CN114241298A CN114241298A CN202111388748.XA CN202111388748A CN114241298A CN 114241298 A CN114241298 A CN 114241298A CN 202111388748 A CN202111388748 A CN 202111388748A CN 114241298 A CN114241298 A CN 114241298A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- target
- dimensional
- cloud data
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 104
- 230000004927 fusion Effects 0.000 title claims description 17
- 238000000034 method Methods 0.000 claims abstract description 27
- 238000012216 screening Methods 0.000 claims abstract description 21
- 238000010276 construction Methods 0.000 claims abstract description 18
- 239000011159 matrix material Substances 0.000 claims description 16
- 238000006243 chemical reaction Methods 0.000 claims description 14
- 238000012544 monitoring process Methods 0.000 claims description 14
- 230000011218 segmentation Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 8
- 238000012549 training Methods 0.000 claims description 8
- 230000036544 posture Effects 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000013519 translation Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000009435 building construction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 229910001294 Reinforcing steel Inorganic materials 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
Abstract
The invention provides a method and a system for detecting a tower crane environment target fused with a laser radar and an image, wherein the method comprises the steps of constructing a combined calibration model of the camera and the laser radar, calibrating internal parameters of the camera and external parameters between the camera and the laser radar, carrying out two-dimensional image target detection on obtained image information by adopting a target detection network to obtain a two-dimensional target object detection frame, projecting three-dimensional point cloud data into a two-dimensional image, extracting corresponding target point cloud data falling into the two-dimensional target object detection frame, clustering, finely dividing and screening the target point cloud data, judging whether the distance between a target object and a tower crane is a safe distance, and outputting alarm information when the target object is detected to be less than the safe distance. The system of the present invention is applied to the above method. The invention is applied to the tower crane in the construction site to detect the target object in real time and realize real-time alarm, thereby reducing the labor consumption and effectively improving the use safety of the tower crane.
Description
Technical Field
The invention relates to the technical field of building construction, in particular to a tower crane environment target detection method based on laser radar and image fusion and a tower crane environment target detection system applying the method.
Background
At present, the building industry develops well, numerous building equipment is widely applied, and in the building construction process, the tower crane is used as important hoisting equipment in a construction site, is mainly applied to hoisting materials such as reinforcing steel bars, steel pipes and the like for construction, has the characteristics of high hoisting height and high operation efficiency, and is widely used in construction. Because the high characteristic of tower crane lifting height, in such a complicated operational environment of building site, the safety problem of tower crane construction is prominent gradually, in order to get into the target object of tower crane arm within range in the building site and in time detect and the early warning, need carry out the target detection to the target object. The traditional detection method is used for monitoring the safety problem of tower crane construction in a construction site through manual identification and monitoring, life and property safety in construction is difficult to guarantee in time, and a more convenient and efficient detection method needs to be provided.
In recent years, with the development of artificial intelligence technology, many target detection algorithms appear in data processing work based on a single sensor, wherein the most common sensor is based on a camera or a laser radar, and abundant two-dimensional visual information can be obtained through the camera, but a monocular camera cannot obtain depth information of the environment, so that distance information of a target is lacked, accurate positioning of the target is difficult to achieve, and in addition, the camera is only used for visual imaging, and imaging is often unstable due to interference of light. The laser radar can accurately provide depth distance information of a target object, the sensing mode of the laser radar is not influenced by light rays and severe weather, data acquisition is stable, but only point cloud data are obtained by the laser radar, and visual information such as color texture and the like is lacked, so that the target object which is far away and has few point clouds is difficult to accurately identify and detect only by the radar data. Due to the single sensing mode: the camera and the laser radar have the advantages and the disadvantages respectively, and the information acquired by different sensors is fused, so that the defect of a single sensor can be overcome, and redundant information can be provided for the system.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a tower crane environment target detection method and system based on laser radar and image fusion.
In order to solve the problems, the technical scheme adopted by the invention is as follows:
a tower crane environment target detection method based on laser radar and image fusion comprises the following steps: constructing a combined calibration model of the camera and the laser radar, and calibrating internal parameters of the camera and external parameters between the camera and the laser radar; acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, and performing two-dimensional image target detection on the acquired image information by adopting a target detection network to obtain a two-dimensional target object detection frame; projecting the three-dimensional point cloud data into a two-dimensional image based on a joint calibration model, extracting corresponding target point cloud data falling into a two-dimensional target object detection frame, and clustering, finely dividing and screening the target point cloud data; and judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting alarm information when the target object is detected to be smaller than the safe distance.
The further scheme is that the method for constructing the combined calibration model of the camera and the laser radar and calibrating the internal parameters of the camera and the external parameters between the camera and the laser radar comprises the following steps: providing a checkerboard calibration plate, respectively placing the checkerboard calibration plate in a camera lens at a plurality of different positions and postures, and extracting checkerboard corner point coordinate information of a corresponding image and laser point cloud for each picture; and calculating the internal reference matrix K of the camera by adopting a self-contained calibration tool box of an ROS system or MATLAB for the extracted calibration plate image information.
The further scheme is that the extracted image and the checkerboard corner point coordinate information of the laser point cloud are calculated to obtain a conversion parameter between the camera and the laser radar, and the conversion parameter is expressed as a formula (1):
wherein, R is a rotation matrix, and t is a translation vector.
According to a further scheme, the acquiring of the three-dimensional point cloud data and the camera image of the tower crane in the surrounding environment comprises the following steps: and acquiring three-dimensional point cloud data and a corresponding two-dimensional image of a surrounding construction site by adopting a security monitoring ball machine and a laser radar which are installed on a tower crane.
A further scheme is that the two-dimensional image target detection is performed on the acquired image information by using a target detection network to obtain a two-dimensional target detection frame, and the method includes: preprocessing the acquired image; training the preprocessed image data set to obtain a training model of a YOLOv5 network; and detecting the experimental image by using the trained model to acquire the category information of the target and the position information of the target candidate frame in the image.
The further scheme is that the three-dimensional point cloud data is projected to the two-dimensional image based on the combined calibration model, and corresponding target point cloud data falling into the two-dimensional target object detection frame is extracted, and the method comprises the following steps: based on the combined calibration result of the camera and the laser radar, projecting the three-dimensional point cloud data into a two-dimensional image according to an internal parameter matrix K of the camera, a rotation matrix R and a translational vector t between the camera and the laser radar; extracting corresponding target point cloud data falling into a two-dimensional target object detection frame according to the position of the projected three-dimensional point cloud data in the two-dimensional image, wherein the point cloud data is a polyhedral cone in a 3D form.
Further, the clustering, fine segmentation and screening of the target point cloud data includes: clustering the objects in the vertebral body of the target point cloud data by using an Euclidean clustering algorithm; performing fine segmentation on the clustered point cloud by utilizing the width information of the target object corresponding to the depth of the target point cloud data; and screening by using the size of the target point cloud data in the two-dimensional image and the position of the target point cloud data in the two-dimensional target object detection frame.
Further, the fine segmentation of the clustered point cloud by using the width information of the target object corresponding to the depth of the target point cloud data includes: and when the depth of certain point cloud data in the clustered point clouds exceeds a set threshold, the certain point cloud data is regarded as point cloud noise and discarded, and the rest point cloud data is stored, wherein the size of the set threshold is determined according to the size of the target.
In a further aspect, the screening using the size of the target point cloud data in the two-dimensional image and the position of the target point cloud data in the two-dimensional target object detection frame includes: and transferring the candidate target point cloud data after the fine segmentation into an image coordinate system by using a coordinate conversion formula, counting point cloud projection pixel points in a small frame at the center of a two-dimensional target object detection frame in the two-dimensional image, comparing the point cloud projection pixel points in the small frame of each clustering result, and screening out a clustering result with the largest pixel point number as final target object point cloud, wherein the width of the small frame is higher than half of the width of the detection frame.
Therefore, compared with the prior art, the invention comprises the following improvement points and beneficial effects:
(1) the two-dimensional image data and the three-dimensional laser point cloud are fused, the advantages and the disadvantages of the collected data characteristics can be made up, the full and effective utilization is realized, the accuracy and the effectiveness of target detection are enhanced, and especially in the complex scene of a construction site, the detection effect of a single sensor can be greatly improved by a multi-sensor fusion mode.
(2) Compared with the most common manual monitoring method at present, the algorithm disclosed by the invention has the advantages that the target object is detected in real time by the tower crane on the construction site, the detection efficiency is greatly improved, and the labor cost can be saved.
(3) By combining the data characteristics of the application scene, compared with the traditional front-back fusion mode, the invention adopts a new data processing method of series fusion, improves the efficiency and accuracy of the algorithm, and ensures the real-time performance and detection accuracy of the algorithm in the actual construction site application.
A system for detecting a tower crane environment target fused by a laser radar and an image is applied to the method for detecting the tower crane environment target fused by the laser radar and the image for detecting the target, and comprises the following steps; the calibration unit is used for constructing a combined calibration model of the camera and the laser radar and calibrating the internal parameters of the camera and the external parameters between the camera and the laser radar;
the system comprises a detection unit, a target detection unit and a data processing unit, wherein the detection unit is used for acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, and performing two-dimensional image target detection on the acquired image information by adopting a target detection network to obtain a two-dimensional target object detection frame;
the processing unit is used for projecting the three-dimensional point cloud data into the two-dimensional image based on the combined calibration model, extracting corresponding target point cloud data falling into the two-dimensional target object detection frame, and clustering, finely dividing and screening the target point cloud data;
and the early warning unit is used for judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting warning information when the target object is detected to be smaller than the safe distance.
Therefore, the target detection of the environment below the tower crane is completed through a cooperation system consisting of the calibration unit, the detection unit and the early warning unit, various parameters and images can be obtained through the camera and the laser radar, all the parameters and the images are further fused into a monitoring target, so that workers can directly watch the monitoring target to judge the safety, and a machine is used for safety assistance, so that the working of multiple persons is avoided, the labor consumption is reduced, and the use safety of the tower crane is improved.
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Drawings
FIG. 1 is a flowchart of an embodiment of a method for detecting a tower crane environment target by fusing a laser radar and an image.
FIG. 2 is a schematic diagram of coordinate conversion between a camera and a laser radar in an embodiment of a method for detecting a tower crane environment target by fusing the laser radar and an image.
FIG. 3 is a schematic diagram of an original point cloud and a target point cloud view cone in the embodiment of the tower crane environment target detection method based on laser radar and image fusion.
FIG. 4 is a schematic diagram of an embodiment of a system for detecting a tower crane environment target based on laser radar and image fusion.
Detailed Description
The embodiment of a tower crane environment target detection method with laser radar and image fusion comprises the following steps:
referring to fig. 1, when the target detection is performed on the environment below the tower crane, the method for detecting the tower crane environment target by fusing the laser radar and the image provided by the invention executes the following steps:
and step S1, constructing a combined calibration model of the camera and the laser radar, and calibrating the internal parameters of the camera and the external parameters between the camera and the laser radar.
And step S2, acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, and performing two-dimensional image target detection on the acquired image information by adopting a target detection network to obtain a two-dimensional target object detection frame.
And step S3, projecting the three-dimensional point cloud data into the two-dimensional image based on the joint calibration model, extracting corresponding target point cloud data falling into the two-dimensional target object detection frame, and clustering, finely dividing and screening the target point cloud data.
And S4, judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting alarm information when the target object is detected to be smaller than the safe distance.
Therefore, according to the method for fusing the laser point cloud and the two-dimensional image and detecting the target of the environment below the tower crane, firstly, three-dimensional point cloud data and a camera image under the surrounding environment are obtained, a camera and laser radar combined calibration model is built, and internal reference calibration of a monocular camera and external reference calibration between the camera and the laser radar are carried out. Then, a single-stage image target detection network YOLOv5 is adopted to perform target two-dimensional detection and obtain a 2D detection frame. And then, roughly dividing the point cloud data by adopting a 2D detection frame, extracting the target point cloud data by utilizing the traditional clustering, and screening the point cloud by adopting a semantic division network. And then, judging according to the obtained target category and the distance between the target and the tower crane, and performing early warning when the target is detected to be smaller than the safe distance.
In step S1, a joint calibration model of the camera and the lidar is constructed, and the calibration of the camera intrinsic parameters and the extrinsic parameters between the camera and the lidar includes:
firstly, providing a checkerboard calibration board, respectively placing the checkerboard calibration board in a camera lens at 12 different positions and postures, and extracting checkerboard corner point coordinate information of corresponding images and laser point clouds for each picture. And calculating the internal reference matrix K of the camera by adopting a self-contained calibration tool box of an ROS system or MATLAB for the extracted calibration plate image information.
Then, as shown in fig. 2, calculating the extracted image and the coordinate information of the checkered corner points of the laser point cloud to obtain a conversion parameter between the camera and the laser radar, which is expressed as formula (1):
wherein, R is a rotation matrix, and t is a translation vector.
In step S2, acquiring three-dimensional point cloud data and a camera image of the tower crane in the surrounding environment includes: and acquiring three-dimensional point cloud data and a corresponding two-dimensional image of a surrounding construction site by adopting a security monitoring ball machine and a laser radar which are installed on a tower crane. The laser radar of the embodiment is a DJI Mid-70 laser radar.
In step S2, performing two-dimensional image target detection on the acquired image information by using a target detection network to obtain a two-dimensional target detection frame, includes:
preprocessing the acquired image: and cutting the original image and inputting the cut image into a network for detection.
And training the preprocessed image data set to obtain a training model of the YOLOv5 network.
The method comprises the steps of detecting an experimental image by using a trained model, obtaining category information of a target object, wherein the category information comprises people, a trolley, a bicycle, a hanging object and position information of a target object candidate frame in the image, and specifically adopting pixel coordinates of four corner points of the target object candidate frame in the image to represent.
In the step S3, projecting the three-dimensional point cloud data into the two-dimensional image based on the joint calibration model, and extracting corresponding target point cloud data falling into the two-dimensional target detection frame, including:
and based on the joint calibration result of the camera and the laser radar, projecting the three-dimensional point cloud data into the two-dimensional image according to the internal parameter matrix K of the camera, the rotation matrix R and the translational vector t between the camera and the laser radar. The camera of the present embodiment is an RGB camera.
Extracting corresponding target point cloud data falling into the two-dimensional target object detection frame according to the position of the projected three-dimensional point cloud data in the two-dimensional image, wherein the point cloud data is a polyhedral cone in a 3D form, as shown in FIG. 3.
In step S3, the clustering, the fine segmentation, and the screening are performed on the target point cloud data, including:
firstly, clustering the objects in the vertebral body of the target point cloud data by using an Euclidean clustering algorithm. In order to ensure the searching accuracy, the searching radius of the target point cloud data is set to be 1m, and meanwhile, in order to avoid clustering errors caused by the fact that the point cloud number of a small target at a distance is too small, the minimum clustering point cloud number is set to be 5.
And then, performing fine segmentation on the clustered point cloud by utilizing the width information of the target object corresponding to the depth of the target point cloud data. When the depth of one point cloud data in the clustered point cloud exceeds a set threshold, the point cloud is regarded as point cloud noise and discarded, and the rest point cloud data are stored, wherein the size of the set threshold is determined according to the target size, the pedestrian is set to be 0.5m, the bicycle is set to be 1.2m, the trolley is set to be 4m, and the suspended object is set to be 5 m.
Then, the size of the target point cloud data in the two-dimensional image and the position of the target point cloud data in the two-dimensional target object detection frame are used for screening. Specifically, the candidate target point cloud data after being finely divided is transferred to an image coordinate system by using a coordinate conversion formula, point cloud projection pixel points in a small frame in the center of a two-dimensional target object detection frame in a two-dimensional image are counted, then the point cloud projection pixel points in the small frame of each clustering result are compared, and a clustering result with the largest pixel point number is screened out to serve as the final target object point cloud, wherein the width of the small frame is higher than half of the width of the detection frame.
In the step S4, the obtained target category information and the depth information of the target laser point cloud data are used to determine whether the distance between the target object and the tower crane is less than the safety distance, and if it is detected that the target object enters the range of the tower crane arm, the system sends out early warning information.
In practical application, taking a frame of image and a corresponding frame of point cloud data as an example, the following synchronization control steps are described in detail:
firstly, calibrating internal and external parameters of a security monitoring dome camera and a DJI Mid-70 laser radar by using a checkerboard calibration plate and a coordinate conversion relation between the camera and the laser radar.
And then, mounting a security monitoring ball machine and a DJI Mid-70 laser radar on a tower crane arm to obtain an image and a laser point cloud of the environment below the tower crane.
Then, the image obtained in the above steps is cut and input to a YOLOv5 network for detection, so as to obtain the 2D candidate frame position information and the object type information of the object, in this embodiment, the number of input images is 1 frame, the original image size is 1280 × 1280, and the obtained object type is a pedestrian.
Then, the obtained laser point clouds corresponding to the image of the present embodiment are converted into an image coordinate system by using the coordinate conversion parameter K, R, t in the above step 4, and then point clouds that fall in the 2D candidate frame of the target object obtained in the previous step are obtained, which are three-dimensionally a view cone.
And then, carrying out point cloud clustering operation on the point cloud view cone obtained in the last step.
Then, the size of the target object is estimated by using the target object category information obtained in the above steps, then the point cloud visual cone is finely divided in the depth direction, the target object depth threshold set in this embodiment is 0.5m, and point cloud noise is further filtered according to the threshold.
And then, screening the obtained clustering point cloud by using the size of the 2D candidate frame of the target object obtained in the step to obtain the final target object point cloud.
And finally, estimating the distance between the target object point cloud obtained in the last step and the tower crane according to the depth information, and if the target object is in the range of the arm of the tower crane, giving an early warning.
Therefore, compared with the prior art, the invention comprises the following improvement points and beneficial effects:
(1) the two-dimensional image data and the three-dimensional laser point cloud are fused, the advantages and the disadvantages of the collected data characteristics can be made up, the full and effective utilization is realized, the accuracy and the effectiveness of target detection are enhanced, and especially in the complex scene of a construction site, the detection effect of a single sensor can be greatly improved by a multi-sensor fusion mode.
(2) Compared with the most common manual monitoring method at present, the algorithm disclosed by the invention has the advantages that the target object is detected in real time by the tower crane on the construction site, the detection efficiency is greatly improved, and the labor cost can be saved.
(3) By combining the data characteristics of the application scene, compared with the traditional front-back fusion mode, the invention adopts a new data processing method of series fusion, improves the efficiency and accuracy of the algorithm, and ensures the real-time performance and detection accuracy of the algorithm in the actual construction site application.
The embodiment of a system for detecting the tower crane environment target by fusing a laser radar and an image comprises the following steps:
a system for detecting a tower crane environment target fused by a laser radar and an image is applied to the method for detecting the tower crane environment target fused by the laser radar and the image for detecting the target, as shown in figure 4, the system comprises;
and the calibration unit 10 is used for constructing a combined calibration model of the camera and the laser radar and calibrating the internal parameters of the camera and the external parameters between the camera and the laser radar.
And the detection unit 20 is used for acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, performing two-dimensional image target detection on the acquired image information by adopting a target detection network, and obtaining a two-dimensional target object detection frame.
And the processing unit 30 is configured to project the three-dimensional point cloud data into the two-dimensional image based on the joint calibration model, extract corresponding target point cloud data falling into the two-dimensional target object detection frame, and perform clustering, fine segmentation and screening on the target point cloud data.
And the early warning unit 40 is used for judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting warning information when the target object is detected to be smaller than the safe distance.
In the calibration unit 10, a combined calibration model of the camera and the lidar is constructed, and the calibration of the camera internal parameters and the external parameters between the camera and the lidar includes:
firstly, providing a checkerboard calibration board, respectively placing the checkerboard calibration board in a camera lens at 12 different positions and postures, and extracting checkerboard corner point coordinate information of corresponding images and laser point clouds for each picture. And calculating the internal reference matrix K of the camera by adopting a self-contained calibration tool box of an ROS system or MATLAB for the extracted calibration plate image information.
Then, calculating the coordinate information of the checkerboard corner points of the extracted image and the laser point cloud to obtain a conversion parameter between the camera and the laser radar, wherein the conversion parameter is expressed as a formula (1):
wherein, R is a rotation matrix, and t is a translation vector.
In the detection unit 20, acquiring three-dimensional point cloud data and a camera image of the tower crane in the surrounding environment, including: and acquiring three-dimensional point cloud data and a corresponding two-dimensional image of a surrounding construction site by adopting a security monitoring ball machine and a laser radar which are installed on a tower crane.
In the detection unit 20, performing two-dimensional image target detection on the acquired image information by using a target detection network to obtain a two-dimensional target detection frame, includes:
preprocessing the acquired image: and cutting the original image and inputting the cut image into a network for detection.
And training the preprocessed image data set to obtain a training model of the YOLOv5 network.
The method comprises the steps of detecting an experimental image by using a trained model, obtaining category information of a target object, wherein the category information comprises people, a trolley, a bicycle, a hanging object and position information of a target object candidate frame in the image, and specifically adopting pixel coordinates of four corner points of the target object candidate frame in the image to represent.
In the processing unit 30, the three-dimensional point cloud data is projected into the two-dimensional image based on the joint calibration model, and the corresponding target point cloud data falling into the two-dimensional target object detection frame is extracted, including:
and based on the joint calibration result of the camera and the laser radar, projecting the three-dimensional point cloud data into the two-dimensional image according to the internal parameter matrix K of the camera, the rotation matrix R and the translational vector t between the camera and the laser radar.
Extracting corresponding target point cloud data falling into a two-dimensional target object detection frame according to the position of the projected three-dimensional point cloud data in the two-dimensional image, wherein the point cloud data is a polyhedral cone in a 3D form.
In the processing unit 30, clustering, fine-segmenting and screening the target point cloud data includes:
firstly, clustering the objects in the vertebral body of the target point cloud data by using an Euclidean clustering algorithm. In order to ensure the searching accuracy, the searching radius of the target point cloud data is set to be 1m, and meanwhile, in order to avoid clustering errors caused by the fact that the point cloud number of a small target at a distance is too small, the minimum clustering point cloud number is set to be 5.
And then, performing fine segmentation on the clustered point cloud by utilizing the width information of the target object corresponding to the depth of the target point cloud data. When the depth of one point cloud data in the clustered point cloud exceeds a set threshold, the point cloud is regarded as point cloud noise and discarded, and the rest point cloud data are stored, wherein the size of the set threshold is determined according to the target size, the pedestrian is set to be 0.5m, the bicycle is set to be 1.2m, the trolley is set to be 4m, and the suspended object is set to be 5 m.
Then, the size of the target point cloud data in the two-dimensional image and the position of the target point cloud data in the two-dimensional target object detection frame are used for screening. Specifically, the candidate target point cloud data after being finely divided is transferred to an image coordinate system by using a coordinate conversion formula, point cloud projection pixel points in a small frame in the center of a two-dimensional target object detection frame in a two-dimensional image are counted, then the point cloud projection pixel points in the small frame of each clustering result are compared, and a clustering result with the largest pixel point number is screened out to serve as the final target object point cloud, wherein the width of the small frame is higher than half of the width of the detection frame.
In the early warning unit 40, the obtained target category information and the depth information of the target laser point cloud data are used for judging whether the distance between the target object and the tower crane is smaller than the safe distance, and if the target object is detected to enter the range of the arm of the tower crane, the system sends out early warning information.
Therefore, the target detection of the environment below the tower crane is completed through the cooperation system consisting of the calibration unit 10, the detection unit 20 and the early warning unit 40, various parameters and images can be obtained through the camera and the laser radar, all the parameters and the images are further fused into a monitoring target, so that workers can directly watch the monitoring target to judge the safety, and a machine is used for safety assistance, so that the working of multiple persons is avoided, the labor consumption is reduced, and the use safety of the tower crane is improved.
The above embodiments are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby, and any insubstantial changes and substitutions made by those skilled in the art based on the present invention are within the protection scope of the present invention.
Claims (10)
1. A tower crane environment target detection method based on laser radar and image fusion is characterized by comprising the following steps:
constructing a combined calibration model of the camera and the laser radar, and calibrating internal parameters of the camera and external parameters between the camera and the laser radar;
acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, and performing two-dimensional image target detection on the acquired image information by adopting a target detection network to obtain a two-dimensional target object detection frame;
projecting the three-dimensional point cloud data into a two-dimensional image based on a joint calibration model, extracting corresponding target point cloud data falling into a two-dimensional target object detection frame, and clustering, finely dividing and screening the target point cloud data;
and judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting alarm information when the target object is detected to be smaller than the safe distance.
2. The method of claim 1, wherein constructing a joint calibration model of the camera and the lidar, and calibrating the camera intrinsic parameters and the extrinsic parameters between the camera and the lidar comprise:
providing a checkerboard calibration plate, respectively placing the checkerboard calibration plate in a camera lens at a plurality of different positions and postures, and extracting checkerboard corner point coordinate information of a corresponding image and laser point cloud for each picture;
and calculating the internal reference matrix K of the camera by adopting a self-contained calibration tool box of an ROS system or MATLAB for the extracted calibration plate image information.
3. The method of claim 2, wherein:
calculating the coordinate information of the checkerboard corner points of the extracted image and the laser point cloud to obtain a conversion parameter between the camera and the laser radar, wherein the conversion parameter is expressed as a formula (1):
wherein, R is a rotation matrix, and t is a translation vector.
4. The method of claim 1, wherein the obtaining three-dimensional point cloud data and camera images of the environment around the tower crane comprises:
and acquiring three-dimensional point cloud data and a corresponding two-dimensional image of a surrounding construction site by adopting a security monitoring ball machine and a laser radar which are installed on a tower crane.
5. The method according to claim 1, wherein the performing two-dimensional image target detection on the acquired image information by using a target detection network to obtain a two-dimensional target detection frame comprises:
preprocessing the acquired image;
training the preprocessed image data set to obtain a training model of a YOLOv5 network;
and detecting the experimental image by using the trained model to acquire the category information of the target and the position information of the target candidate frame in the image.
6. The method of claim 3, wherein the projecting the three-dimensional point cloud data into the two-dimensional image based on the joint calibration model, extracting corresponding target point cloud data falling into the two-dimensional target detection box, comprises:
based on the combined calibration result of the camera and the laser radar, projecting the three-dimensional point cloud data into a two-dimensional image according to an internal parameter matrix K of the camera, a rotation matrix R and a translational vector t between the camera and the laser radar;
extracting corresponding target point cloud data falling into a two-dimensional target object detection frame according to the position of the projected three-dimensional point cloud data in the two-dimensional image, wherein the point cloud data is a polyhedral cone in a 3D form.
7. The method of claim 6, wherein the clustering, fine segmenting, and screening the target point cloud data comprises:
clustering the objects in the vertebral body of the target point cloud data by using an Euclidean clustering algorithm;
performing fine segmentation on the clustered point cloud by utilizing the width information of the target object corresponding to the depth of the target point cloud data;
and screening by using the size of the target point cloud data in the two-dimensional image and the position of the target point cloud data in the two-dimensional target object detection frame.
8. The method of claim 7, wherein the performing the fine segmentation of the clustered point cloud using the width information of the target object corresponding to the depth of the target point cloud data comprises:
and when the depth of certain point cloud data in the clustered point clouds exceeds a set threshold, the certain point cloud data is regarded as point cloud noise and discarded, and the rest point cloud data is stored, wherein the size of the set threshold is determined according to the size of the target.
9. The method of claim 7, wherein the screening using the size of the target point cloud data in the two-dimensional image and the location in the two-dimensional target detection box comprises:
and transferring the candidate target point cloud data after the fine segmentation into an image coordinate system by using a coordinate conversion formula, counting point cloud projection pixel points in a small frame at the center of a two-dimensional target object detection frame in the two-dimensional image, comparing the point cloud projection pixel points in the small frame of each clustering result, and screening out a clustering result with the largest pixel point number as final target object point cloud, wherein the width of the small frame is higher than half of the width of the detection frame.
10. A system for detecting a target in a tower crane environment with a laser radar and image fusion function, which is applied to the method for detecting the target in the tower crane environment with the laser radar and image fusion function according to any one of claims 1 to 9, and comprises:
the calibration unit is used for constructing a combined calibration model of the camera and the laser radar and calibrating the internal parameters of the camera and the external parameters between the camera and the laser radar;
the system comprises a detection unit, a target detection unit and a data processing unit, wherein the detection unit is used for acquiring three-dimensional point cloud data and a camera image in the surrounding environment of the tower crane, and performing two-dimensional image target detection on the acquired image information by adopting a target detection network to obtain a two-dimensional target object detection frame;
the processing unit is used for projecting the three-dimensional point cloud data into the two-dimensional image based on the combined calibration model, extracting corresponding target point cloud data falling into the two-dimensional target object detection frame, and clustering, finely dividing and screening the target point cloud data;
and the early warning unit is used for judging whether the distance between the target object and the tower crane is a safe distance according to the obtained target category, and outputting warning information when the target object is detected to be smaller than the safe distance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111388748.XA CN114241298A (en) | 2021-11-22 | 2021-11-22 | Tower crane environment target detection method and system based on laser radar and image fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111388748.XA CN114241298A (en) | 2021-11-22 | 2021-11-22 | Tower crane environment target detection method and system based on laser radar and image fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114241298A true CN114241298A (en) | 2022-03-25 |
Family
ID=80750369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111388748.XA Pending CN114241298A (en) | 2021-11-22 | 2021-11-22 | Tower crane environment target detection method and system based on laser radar and image fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114241298A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114693187A (en) * | 2022-05-31 | 2022-07-01 | 杭州未名信科科技有限公司 | Operation analysis method and device of tower crane cluster, storage medium and terminal |
| CN114885133A (en) * | 2022-07-04 | 2022-08-09 | 中科航迈数控软件(深圳)有限公司 | Depth image-based equipment safety real-time monitoring method and system and related equipment |
| CN114973564A (en) * | 2022-04-28 | 2022-08-30 | 北京机械设备研究所 | Remote personnel intrusion detection method and device under non-illumination condition |
| CN115273268A (en) * | 2022-07-27 | 2022-11-01 | 中建三局集团有限公司 | Large-scene construction site inspection method based on tower crane carrying zoom camera |
| CN115547003A (en) * | 2022-11-24 | 2022-12-30 | 北京数字绿土科技股份有限公司 | External damage prevention alarm method and system for power transmission line |
| CN115588121A (en) * | 2022-11-03 | 2023-01-10 | 腾晖科技建筑智能(深圳)有限公司 | Tower crane lifting object type detection method and system based on sensing data and image sequence |
| CN115909272A (en) * | 2022-11-09 | 2023-04-04 | 杭州枕石智能科技有限公司 | Method for acquiring obstacle position information, terminal device and computer medium |
| CN115994854A (en) * | 2023-03-22 | 2023-04-21 | 智洋创新科技股份有限公司 | Method and system for registering marker point cloud and image |
| CN116862922A (en) * | 2023-06-20 | 2023-10-10 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
| CN117329971A (en) * | 2023-12-01 | 2024-01-02 | 海博泰科技(青岛)有限公司 | Compartment balance detection method and system based on three-dimensional laser radar |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150003723A1 (en) * | 2013-06-27 | 2015-01-01 | Chevron U.S.A. Inc. | System and method of detecting objects in scene point cloud |
| CN109444911A (en) * | 2018-10-18 | 2019-03-08 | 哈尔滨工程大学 | A kind of unmanned boat waterborne target detection identification and the localization method of monocular camera and laser radar information fusion |
| CN110264416A (en) * | 2019-05-28 | 2019-09-20 | 深圳大学 | Sparse point cloud segmentation method and device |
| CN111612845A (en) * | 2020-04-13 | 2020-09-01 | 江苏大学 | Laser radar and camera combined calibration method based on mobile calibration plate |
| CN112669393A (en) * | 2020-12-31 | 2021-04-16 | 中国矿业大学 | Laser radar and camera combined calibration method |
| CN112785651A (en) * | 2019-11-11 | 2021-05-11 | 北京京邦达贸易有限公司 | Method and apparatus for determining relative pose parameters |
| CN113255481A (en) * | 2021-05-11 | 2021-08-13 | 北方工业大学 | Crowd state detection method based on unmanned patrol car |
| CN113506318A (en) * | 2021-07-12 | 2021-10-15 | 广东工业大学 | Three-dimensional target perception method under vehicle-mounted edge scene |
| US20210342620A1 (en) * | 2018-10-30 | 2021-11-04 | Mitsubishi Electric Corporation | Geographic object detection apparatus and geographic object detection method |
-
2021
- 2021-11-22 CN CN202111388748.XA patent/CN114241298A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150003723A1 (en) * | 2013-06-27 | 2015-01-01 | Chevron U.S.A. Inc. | System and method of detecting objects in scene point cloud |
| CN109444911A (en) * | 2018-10-18 | 2019-03-08 | 哈尔滨工程大学 | A kind of unmanned boat waterborne target detection identification and the localization method of monocular camera and laser radar information fusion |
| US20210342620A1 (en) * | 2018-10-30 | 2021-11-04 | Mitsubishi Electric Corporation | Geographic object detection apparatus and geographic object detection method |
| CN110264416A (en) * | 2019-05-28 | 2019-09-20 | 深圳大学 | Sparse point cloud segmentation method and device |
| CN112785651A (en) * | 2019-11-11 | 2021-05-11 | 北京京邦达贸易有限公司 | Method and apparatus for determining relative pose parameters |
| CN111612845A (en) * | 2020-04-13 | 2020-09-01 | 江苏大学 | Laser radar and camera combined calibration method based on mobile calibration plate |
| CN112669393A (en) * | 2020-12-31 | 2021-04-16 | 中国矿业大学 | Laser radar and camera combined calibration method |
| CN113255481A (en) * | 2021-05-11 | 2021-08-13 | 北方工业大学 | Crowd state detection method based on unmanned patrol car |
| CN113506318A (en) * | 2021-07-12 | 2021-10-15 | 广东工业大学 | Three-dimensional target perception method under vehicle-mounted edge scene |
Non-Patent Citations (2)
| Title |
|---|
| DENG Z, LATECKI L J.: "Amodal detection of 3D objects: inferring 3D bounding boxes from 2D ones in RGB-depth images", 《PROCEEDINGS OF THE 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》 * |
| 宋一凡,张鹏,宗立波等: "改进的基于冗余点过滤的3D目标检测方法", 《计算机应用》 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114973564A (en) * | 2022-04-28 | 2022-08-30 | 北京机械设备研究所 | Remote personnel intrusion detection method and device under non-illumination condition |
| CN114693187A (en) * | 2022-05-31 | 2022-07-01 | 杭州未名信科科技有限公司 | Operation analysis method and device of tower crane cluster, storage medium and terminal |
| CN114885133A (en) * | 2022-07-04 | 2022-08-09 | 中科航迈数控软件(深圳)有限公司 | Depth image-based equipment safety real-time monitoring method and system and related equipment |
| CN115273268B (en) * | 2022-07-27 | 2023-05-19 | 中建三局集团有限公司 | Large scene construction site inspection method based on tower crane carrying zoom camera |
| CN115273268A (en) * | 2022-07-27 | 2022-11-01 | 中建三局集团有限公司 | Large-scene construction site inspection method based on tower crane carrying zoom camera |
| CN115588121B (en) * | 2022-11-03 | 2023-07-04 | 腾晖科技建筑智能(深圳)有限公司 | Tower crane object type detection method and system based on sensing data and image sequence |
| CN115588121A (en) * | 2022-11-03 | 2023-01-10 | 腾晖科技建筑智能(深圳)有限公司 | Tower crane lifting object type detection method and system based on sensing data and image sequence |
| CN115909272A (en) * | 2022-11-09 | 2023-04-04 | 杭州枕石智能科技有限公司 | Method for acquiring obstacle position information, terminal device and computer medium |
| CN115547003A (en) * | 2022-11-24 | 2022-12-30 | 北京数字绿土科技股份有限公司 | External damage prevention alarm method and system for power transmission line |
| CN115994854A (en) * | 2023-03-22 | 2023-04-21 | 智洋创新科技股份有限公司 | Method and system for registering marker point cloud and image |
| CN116862922A (en) * | 2023-06-20 | 2023-10-10 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
| CN116862922B (en) * | 2023-06-20 | 2024-03-19 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
| CN117329971A (en) * | 2023-12-01 | 2024-01-02 | 海博泰科技(青岛)有限公司 | Compartment balance detection method and system based on three-dimensional laser radar |
| CN117329971B (en) * | 2023-12-01 | 2024-02-27 | 海博泰科技(青岛)有限公司 | Compartment balance detection method and system based on three-dimensional laser radar |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114241298A (en) | Tower crane environment target detection method and system based on laser radar and image fusion | |
| WO2020253308A1 (en) | Human-machine interaction behavior security monitoring and forewarning method for underground belt transportation-related personnel | |
| CN104091369B (en) | Unmanned aerial vehicle remote-sensing image building three-dimensional damage detection method | |
| WO2018028103A1 (en) | Unmanned aerial vehicle power line inspection method based on characteristics of human vision | |
| CN112216049A (en) | Construction warning area monitoring and early warning system and method based on image recognition | |
| CN102496232B (en) | Transmission facility monitoring method and system | |
| Rau et al. | Bridge crack detection using multi-rotary UAV and object-base image analysis | |
| CN113255481A (en) | Crowd state detection method based on unmanned patrol car | |
| CN104821056A (en) | Intelligent guarding method based on radar and video integration | |
| CN103559703A (en) | Crane barrier monitoring and prewarning method and system based on binocular vision | |
| CN110189375B (en) | Image target identification method based on monocular vision measurement | |
| CN110910350B (en) | Nut loosening detection method for wind power tower cylinder | |
| CN109506628A (en) | Object distance measuring method under a kind of truck environment based on deep learning | |
| CN113139437B (en) | Helmet wearing inspection method based on YOLOv3 algorithm | |
| CN112528979B (en) | Transformer substation inspection robot obstacle distinguishing method and system | |
| CN112184773A (en) | Helmet wearing detection method and system based on deep learning | |
| CN113052903A (en) | Vision and radar fusion positioning method for mobile robot | |
| CN110885019A (en) | Tower crane monitoring and real-time alarm system based on BIM model | |
| CN115588043A (en) | Excavator operation pose monitoring method based on vision | |
| CN107767366A (en) | A kind of transmission line of electricity approximating method and device | |
| CN111461079A (en) | Binocular image-based method for detecting personnel under suspension arm | |
| CN112101211A (en) | Personnel and suspension arm position calculation method based on target detection and binocular ranging | |
| WO2023109664A1 (en) | Monitoring method and related product | |
| CN116486287A (en) | Target detection method and system based on environment self-adaptive robot vision system | |
| CN113643544B (en) | Intelligent detection method and system for illegal parking in parking lot based on Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220325 |