CN112462373A - Cabin position detection method based on multi-sensor fusion - Google Patents
Cabin position detection method based on multi-sensor fusion Download PDFInfo
- Publication number
- CN112462373A CN112462373A CN202110141999.1A CN202110141999A CN112462373A CN 112462373 A CN112462373 A CN 112462373A CN 202110141999 A CN202110141999 A CN 202110141999A CN 112462373 A CN112462373 A CN 112462373A
- Authority
- CN
- China
- Prior art keywords
- container
- point cloud
- coordinate
- image
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
- B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
- B65G67/00—Loading or unloading vehicles
- B65G67/60—Loading or unloading ships
- B65G67/603—Loading or unloading ships using devices specially adapted for articles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
Abstract
The invention relates to a cabin position detection method based on multi-sensor fusion.A laser radar and a camera are arranged on a trolley of a lifting appliance, wherein an image is obtained through the camera, a container in the image and box type and box number information of the container are identified, and pixel coordinates of the position of the container in the image are obtained; acquiring a point cloud through a laser radar, converting a point cloud coordinate into a pixel coordinate, comparing the container pixel coordinate in an image with the pixel coordinate of the point cloud, and segmenting the point cloud of the container from an environmental point cloud; and calculating the centroid coordinate of the container point cloud, corresponding the centroid coordinate to a world coordinate system through the position information of the trolley to obtain a corresponding world coordinate, and giving the box type and box number information of the container to the container point cloud. The method adopts a multi-sensor data fusion method to detect the cabin position, and combines the accurate positioning of the laser radar and richer semantic information of the camera, so that the cabin position detection can cope with more complex conditions.
Description
Technical Field
The invention belongs to the technical field of container freight at a wharf, and particularly relates to a cabin position detection method based on multi-sensor fusion.
Background
The cabin position detection is to determine the position of the container in the cabin. During the loading and unloading process of the container, the position of the container to be grabbed is determined, and secondary transportation caused by mistaken loading and unloading of the container is prevented. Therefore, the position detection of the ship cabin is very important in the process of loading and unloading the port container, which directly influences the accuracy and efficiency of loading and unloading and further influences the economic efficiency of international shipping companies. With the development and progress of science and technology, each shipping company hopes to replace the traditional manual confirmation mode by means of intelligent technology so as to improve the efficiency of cabin position detection.
The currently used technical solutions are roughly classified into two types:
1) and acquiring the current position of the trolley and the position of the lifting appliance by a PLC (programmable logic controller) arranged on the shore bridge. In the patent "automatic identification system and method for container loading bay" (application number: CN 202010187618.9), the inventor confirms whether the current container position is consistent with the preset position according to the trolley position and the hanger position during operation. The method has low efficiency, and can not find the problem quickly and solve the problem in time. When tide fluctuation and irregular containers occur, the data of the PLC are irregular, and the problem of identification failure can occur by adopting single PLC data.
2) And acquiring the position of the container by adopting a laser identification technology, and judging whether the position of the current container is consistent with the preset position. In the patent "an automatic identification system and identification method of container loading and unloading ship's station" (application number: CN 201711145462.2), the inventor calculates the ship's station by means of a laser scanner and the size of the container. Such solutions are often used in automated docks, and traditional non-automated docks are difficult to deploy. However, the simultaneous existence of the single and double boxes cannot be distinguished only by data of a single laser radar, and the application range is limited.
Reference documents:
CN 109795892A-an automatic identification system and identification method for container position of container loading and unloading ship;
CN 111170158A-automatic identification system and method for the positions of the containers when the containers are loaded;
CN 103336282B-an automatic cabin positioning device and its positioning method;
CN 103196434B-a port container positioning device and method;
CN 106291622A-a container positioning device and its positioning method;
CN 108460800A-Container image positioning method and System;
CN 109655855A-a positioning device for container and container;
CN 111243016A-an automatic container identification and location method.
Disclosure of Invention
The invention aims to provide a cabin position detection method based on multi-sensor fusion.
In order to achieve the purpose, the invention provides the following technical scheme:
a cabin position detection method based on multi-sensor fusion is characterized by comprising the following steps: mounting a laser radar and a camera on the trolley, wherein
Acquiring an image through a camera, identifying a container in the image and box type and box number information of the container, and acquiring pixel coordinates of the position of the container in the image;
acquiring a point cloud through a laser radar, converting a point cloud coordinate into a pixel coordinate, comparing the container pixel coordinate in an image with the pixel coordinate of the point cloud, and segmenting the point cloud of the container from an environmental point cloud;
and calculating the centroid coordinate of the container point cloud, corresponding the centroid coordinate to a world coordinate system through the position information of the trolley to obtain a corresponding world coordinate, and giving the box type and box number information of the container to the container point cloud.
Further, the world coordinates of all containers are obtained, the position information is calculated and compared with the preset positions, and a cabin position detection result is obtained.
Furthermore, the trolley is driven from the land side to the sea side until reaching the last container, and the laser radar point cloud, the camera image and the trolley position information are synchronously acquired at regular intervals.
Compared with the prior art, the technical scheme comprises the following improvement points and beneficial effects:
(1) the invention adopts a multi-sensor data fusion method to detect the cabin position, and combines the accurate positioning of the laser radar and richer semantic information of the camera, so that the cabin position detection can cope with more complicated single-box and double-box conditions.
(2) Compared with the traditional scheme of confirming while working, the scheme of scanning the layer has higher detection efficiency.
Drawings
Fig. 1 is a schematic diagram of a cabin location detection process.
Fig. 2 is a schematic view of the installation positions of the laser radar and the camera.
FIG. 3 is a flow chart for cabin level detection based on multi-sensor data fusion.
In the figure: 1-trolley, 2-shore bridge, 3-ship, 4-container, 5-laser radar, 6-camera.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Installing a laser radar and a camera: as shown in fig. 2, the lidar and the camera are mounted at the forefront of the trolley (world coordinate system) of the lifting appliance, and are vertically mounted vertically downwards, the lidar can be a 2D or 3D laser, and the scanning plane is consistent with the moving direction of the trolley.
As shown in fig. 3, a cabin position detection method based on multi-sensor fusion includes:
when the ship just approaches the shore or finishes working on one floor, the trolley reaches the position above the warning position where the ship approaches the shore, the trolley is driven from the land side to the sea side until the trolley reaches the last container, and the laser radar point cloud, the camera image and the trolley position information are synchronously acquired at regular intervals.
The method comprises the steps of acquiring an image through a camera, identifying a container in the image through a convolutional neural network, identifying the position, the center, the box type and the box number information of the container, and acquiring pixel coordinates of the container in the image. The position of the container in the image coordinate system is described by the coordinates (u, v) of the container center pixel and the length (w, h) of the container pixel, and the box type and the box number are described by character strings and can be stored by using the structure body and the position of the container. The container type of the container can comprise single 20-foot, double 20-foot, single 40-foot, single 45-foot and the like, and the information of the container number corresponds to a unique container.
And acquiring the point cloud through a laser radar, converting the point cloud coordinate into a pixel coordinate, comparing the container pixel coordinate in the image with the pixel coordinate of the point cloud, and segmenting the point cloud of the container from the environmental point cloud. The method comprises the following specific steps:
the laser radar acquires point cloud of the container and the environment, and the coordinate under the laser radar coordinate system is
The internal reference matrix of the camera can be known through calibration
And the transformation matrix of the laser radar under the camera coordinate system
。
The point cloud coordinates under the pixel coordinate system can be obtained by using the following four coordinate transformation formulas;
the formula is:
the formula (II):
formula (c):
the formula (IV):
a formula is:
the formula I is to convert the coordinates of the points in the laser radar coordinate system into homogeneous coordinates, namely to increase the coordinates by 1.
And a formula II is to transform the coordinates under the laser radar coordinate system to the coordinates under the camera coordinate system.
Formula III is that the homogeneous coordinate in the coordinate system of the camera is converted into the inhomogeneous coordinate, namely that 1 of the last bit in the coordinate is removed.
The formula (iv) is to transform the coordinates of this point in the camera coordinate system to the coordinates in the pixel coordinate system (de-Z axis), where Z is the depth coordinates of the point in the camera coordinate system.
The formula (v) is to convert the left side of this pixel into non-homogeneous coordinates.
The following takes the calculation of one point as an example:
internal reference matrix of camera
，
Transformation matrix of laser radar under camera coordinate system
。
Firstly, a formula is utilized to convert the coordinates of points in a laser radar system into homogeneous coordinates
。
Then obtaining the coordinate of the point in the camera coordinate system by a formula II
。
Then, the homogeneous coordinate of the point in the camera coordinate system is converted into non-homogeneous coordinate by formula
。
And then, transforming the camera coordinate system to a pixel coordinate system by using a formula IV to obtain the homogeneous coordinate of the point in the pixel coordinate system as follows:
。
since the pixel coordinates are typically integers, the calculation of equation (r) has a rounding operation.
Converting the homogeneous coordinate into non-homogeneous coordinate by formula to obtain the actual pixel coordinate of the point in the image
。
Assume now that the pixel location in the image where the container is identified in the image is (968, 546, 707, 148).
Taking the calculation of four points in the point cloud as an example (the actual point cloud number is more), the pixel coordinates of the four points and the coordinates under the laser radar coordinate system are respectively:
，
if it is not
And is and
then this point is considered to be the point of the container.
Comparing the pixel coordinates of the point cloud with the positions of the containers in the image can know that P1, P2 and P3 are point clouds of the containers, P4 is a point cloud of the environment, and the point cloud of the containers can be removed, so that the point cloud of the containers can be segmented from the environment.
Then, calculating an average value by using coordinates of the three points P1, P2 and P3 in a laser radar coordinate system, and taking the average value as a coordinate (centroid coordinate) of the centroid of the container in the laser radar coordinate system as follows:
because the laser radar can move along with the movement of the trolley, the position of the trolley is utilized to transform the coordinates of the mass center of the container to the coordinates under a world coordinate system
。
Thus, each container has corresponding point cloud, centroid coordinates (world coordinate system), box type information and box number information.
And then calculating the decibel information according to the centroid coordinate sequence of each container. The column number and layer number of the container are calculated according to the following iterative formula, using the container with the minimum y in the centroid coordinates as the starting container.
Wherein
，
。
For example, the coordinates of the three container centroids in the world coordinate system are obtained through the previous calculation:
selecting
Is the starting container location;
number of first container
Layer number
；
The column number and layer number of the second container are respectively:
the column number and layer number of the third container are respectively:
comparing the calculated result (the scallop bit information) with the preset scallop bit information result can be used for confirming the scallop bit information, and finally, the cabin bit detection result can be obtained.
Description of the principles of the invention: as shown in fig. 1, when the ship just lands on the shore or finishes working on one floor, the trolley drives from the land side to the sea side to reach the last container, and at this time, the information of all containers on the topmost floor can be obtained. According to the technical scheme, the position information of the container on the surface of the whole ship is acquired by fusing a laser radar, a camera and the position (world coordinate system) of a trolley in a PLC. Whether the actual shellfish position of the current ship body is consistent with the preset shellfish position can be judged by comparing the actual shellfish position with the preset shellfish position information.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (3)
1. A cabin position detection method based on multi-sensor fusion is characterized by comprising the following steps: mounting a laser radar and a camera on the trolley, wherein
Acquiring an image through a camera, identifying a container in the image and box type and box number information of the container, and acquiring pixel coordinates of the position of the container in the image;
acquiring a point cloud through a laser radar, converting a point cloud coordinate into a pixel coordinate, comparing the container pixel coordinate in an image with the pixel coordinate of the point cloud, and segmenting the point cloud of the container from an environmental point cloud;
and calculating the centroid coordinate of the container point cloud, corresponding the centroid coordinate to a world coordinate system through the position information of the trolley to obtain a corresponding world coordinate, and giving the box type and box number information of the container to the container point cloud.
2. The multi-sensor fusion-based hold detection method of claim 1, wherein: and acquiring world coordinates of all containers, calculating the position information, and comparing the position information with preset positions to obtain a cabin position detection result.
3. The multi-sensor fusion-based hold detection method of claim 1, wherein: the trolley is driven from the land side to the sea side until reaching the last container, and the laser radar point cloud, the camera image and the trolley position information are synchronously collected at regular intervals.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110141999.1A CN112462373A (en) | 2021-02-02 | 2021-02-02 | Cabin position detection method based on multi-sensor fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110141999.1A CN112462373A (en) | 2021-02-02 | 2021-02-02 | Cabin position detection method based on multi-sensor fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112462373A true CN112462373A (en) | 2021-03-09 |
Family
ID=74802455
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110141999.1A Pending CN112462373A (en) | 2021-02-02 | 2021-02-02 | Cabin position detection method based on multi-sensor fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112462373A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113551611A (en) * | 2021-06-15 | 2021-10-26 | 西安交通大学 | Stereo vision measuring method, system, equipment and storage medium for large-size moving object |
| CN114671266A (en) * | 2022-05-26 | 2022-06-28 | 浙江天新智能研究院有限公司 | Collapse coal unloading process for unattended screw ship unloader |
| CN116193262A (en) * | 2023-04-25 | 2023-05-30 | 上海安维尔信息科技股份有限公司 | Container PTZ camera selective aiming method and system in storage yard |
| WO2024060792A1 (en) * | 2022-09-22 | 2024-03-28 | 中车资阳机车有限公司 | Lock hole locating system and method for split-type container spreader |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203284078U (en) * | 2013-05-08 | 2013-11-13 | 北京国泰星云科技有限公司 | Container collision prevention automatic control system for RTG/RMG lifting appliance in container wharf |
| CN203439940U (en) * | 2013-09-04 | 2014-02-19 | 贾来国 | Automatic control system for RTG/RMG dual-laser sling crash-proof box at container terminal |
| CN103803419A (en) * | 2014-03-06 | 2014-05-21 | 上海振华重工电气有限公司 | Safety system of stacking crane and driving method |
| CN105565167A (en) * | 2016-03-09 | 2016-05-11 | 任贤定 | Visual inspection system and implementation method for direction locating of travelling crab |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| CN208688445U (en) * | 2018-09-14 | 2019-04-02 | 上海川丰机电科技发展有限公司 | Ship type outline detection system |
| CN109815833A (en) * | 2018-12-29 | 2019-05-28 | 江苏集萃智能制造技术研究所有限公司 | A kind of tea point recognition methods based on CCD Yu the feature extraction of three-dimensional laser sensor information fusion |
| CN110902570A (en) * | 2019-11-25 | 2020-03-24 | 上海驭矩信息科技有限公司 | Dynamic measurement method and system for container loading and unloading operation |
| CN110929692A (en) * | 2019-12-11 | 2020-03-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional target detection method and device based on multi-sensor information fusion |
| CN211698183U (en) * | 2019-12-31 | 2020-10-16 | 安徽华电芜湖发电有限公司 | Ship unloader |
| CN111832345A (en) * | 2019-04-17 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Container monitoring method, device and equipment and storage medium |
| CN111830526A (en) * | 2020-09-17 | 2020-10-27 | 上海驭矩信息科技有限公司 | Container positioning method and device based on multi-line laser data fusion |
| CN112149550A (en) * | 2020-09-21 | 2020-12-29 | 华南理工大学 | Automatic driving vehicle 3D target detection method based on multi-sensor fusion |
-
2021
- 2021-02-02 CN CN202110141999.1A patent/CN112462373A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN203284078U (en) * | 2013-05-08 | 2013-11-13 | 北京国泰星云科技有限公司 | Container collision prevention automatic control system for RTG/RMG lifting appliance in container wharf |
| CN203439940U (en) * | 2013-09-04 | 2014-02-19 | 贾来国 | Automatic control system for RTG/RMG dual-laser sling crash-proof box at container terminal |
| CN103803419A (en) * | 2014-03-06 | 2014-05-21 | 上海振华重工电气有限公司 | Safety system of stacking crane and driving method |
| CN105565167A (en) * | 2016-03-09 | 2016-05-11 | 任贤定 | Visual inspection system and implementation method for direction locating of travelling crab |
| CN208688445U (en) * | 2018-09-14 | 2019-04-02 | 上海川丰机电科技发展有限公司 | Ship type outline detection system |
| CN109472831A (en) * | 2018-11-19 | 2019-03-15 | 东南大学 | Obstacle recognition range-measurement system and method towards road roller work progress |
| CN109815833A (en) * | 2018-12-29 | 2019-05-28 | 江苏集萃智能制造技术研究所有限公司 | A kind of tea point recognition methods based on CCD Yu the feature extraction of three-dimensional laser sensor information fusion |
| CN111832345A (en) * | 2019-04-17 | 2020-10-27 | 杭州海康威视数字技术股份有限公司 | Container monitoring method, device and equipment and storage medium |
| CN110902570A (en) * | 2019-11-25 | 2020-03-24 | 上海驭矩信息科技有限公司 | Dynamic measurement method and system for container loading and unloading operation |
| CN110929692A (en) * | 2019-12-11 | 2020-03-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional target detection method and device based on multi-sensor information fusion |
| CN211698183U (en) * | 2019-12-31 | 2020-10-16 | 安徽华电芜湖发电有限公司 | Ship unloader |
| CN111830526A (en) * | 2020-09-17 | 2020-10-27 | 上海驭矩信息科技有限公司 | Container positioning method and device based on multi-line laser data fusion |
| CN112149550A (en) * | 2020-09-21 | 2020-12-29 | 华南理工大学 | Automatic driving vehicle 3D target detection method based on multi-sensor fusion |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113551611A (en) * | 2021-06-15 | 2021-10-26 | 西安交通大学 | Stereo vision measuring method, system, equipment and storage medium for large-size moving object |
| CN113551611B (en) * | 2021-06-15 | 2022-04-22 | 西安交通大学 | Stereo vision measuring method, system, equipment and storage medium for large-size moving object |
| CN114671266A (en) * | 2022-05-26 | 2022-06-28 | 浙江天新智能研究院有限公司 | Collapse coal unloading process for unattended screw ship unloader |
| WO2024060792A1 (en) * | 2022-09-22 | 2024-03-28 | 中车资阳机车有限公司 | Lock hole locating system and method for split-type container spreader |
| CN116193262A (en) * | 2023-04-25 | 2023-05-30 | 上海安维尔信息科技股份有限公司 | Container PTZ camera selective aiming method and system in storage yard |
| CN116193262B (en) * | 2023-04-25 | 2023-09-01 | 上海安维尔信息科技股份有限公司 | Container PTZ camera selective aiming method and system in storage yard |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112462373A (en) | Cabin position detection method based on multi-sensor fusion | |
| CN105431370B (en) | For method and system container automatically unloaded using container crane in unloading target | |
| CN103781717B (en) | Determine the position of the capture member of hoisting crane and the system and method for measure of skewness | |
| Mi et al. | Ship identification algorithm based on 3D point cloud for automated ship loaders | |
| Zong et al. | Container ship cell guide accuracy check technology based on improved 3D point cloud instance segmentation | |
| Shen et al. | A positioning lockholes of container corner castings method based on image recognition | |
| CN112150388B (en) | Continuous ship unloader ship and material identification sensing method | |
| US20230324554A1 (en) | Container positioning method and apparatus based on multi-line laser data fusion | |
| CN114988283A (en) | High-precision cabin position real-time detection system for automatic bridge crane | |
| WO2022127311A1 (en) | Left lock pad detection method and system of automated wharf bridge crane | |
| US20230348237A1 (en) | Mapping of a Crane Spreader and a Crane Spreader Target | |
| WO2023040079A1 (en) | High-precision grab bucket grabbing position control method having radar feedback | |
| CN115880252B (en) | Container sling detection method, device, computer equipment and storage medium | |
| CN108445514A (en) | A kind of container stacking alignment method based on global position system and laser ranging | |
| CN108681702B (en) | Method and system for determining loading and unloading stowage decibel information of container | |
| CN105469401A (en) | Ship groove positioning method based on computer vision | |
| CN115258959B (en) | Sling control method, equipment and storage medium | |
| CN115586552A (en) | Method for accurately secondarily positioning unmanned truck collection under port tyre crane or bridge crane | |
| Jung et al. | Advanced sensing system of crane spreader motion (for mobile harbor) | |
| CN115849192A (en) | Automatic outer container-placing method based on area array laser scanning | |
| CN113538566B (en) | Cargo ship hatch position acquisition method and system based on laser radar | |
| Jiang et al. | Real-time hatch detection during unloading of dry bulk carriers with side rolling hatchcover | |
| US20230382693A1 (en) | Apparatus and method for determining the spatial position of pick-up elements of a container | |
| CN215798022U (en) | Shore bridge scanning device and shore bridge equipment | |
| CN114677424B (en) | Point cloud data processing method for unattended screw ship unloader |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210309 |
|
| RJ01 | Rejection of invention patent application after publication |