CN109839109B - Development machine absolute pose detection method based on image recognition and multi-sensor fusion - Google Patents
Development machine absolute pose detection method based on image recognition and multi-sensor fusion Download PDFInfo
- Publication number
- CN109839109B CN109839109B CN201910139486.XA CN201910139486A CN109839109B CN 109839109 B CN109839109 B CN 109839109B CN 201910139486 A CN201910139486 A CN 201910139486A CN 109839109 B CN109839109 B CN 109839109B
- Authority
- CN
- China
- Prior art keywords
- gyroscope
- machine
- camera
- gyroscopes
- tunneling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011161 development Methods 0.000 title claims abstract description 19
- 230000004927 fusion Effects 0.000 title claims abstract description 8
- 238000001514 detection method Methods 0.000 title abstract description 7
- 230000005641 tunneling Effects 0.000 claims abstract description 24
- 238000006073 displacement reaction Methods 0.000 claims abstract description 22
- 238000003384 imaging method Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 12
- 238000005065 mining Methods 0.000 claims description 11
- 239000011159 matrix material Substances 0.000 claims description 9
- 238000005259 measurement Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 3
- 239000003245 coal Substances 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Landscapes
- Gyroscopes (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention provides a development machine absolute pose detection method based on image recognition and multi-sensor fusion, wherein a laser indicator lamp is arranged on a development machine body, and projects a target pattern on a working surface; cameras are arranged on the left and right tunneling arms and the machine body of the tunneling machine, and the three cameras simultaneously collect target patterns; the method comprises the following steps that gyroscopes are respectively installed at the left and right tunneling arms of the tunneling machine and the center of a machine body, and the gyroscopes are used for collecting the three-axis angular speed and the rotation amount under the coordinate system of the gyroscopes; installing a displacement sensor in an electric control box of the heading machine to acquire the displacement of the heading machine to a working face; and according to the acquired data, establishing an equivalent constraint equation of pixel coordinates of the same characteristic point in the left and right imaging planes of the stereo camera, and calculating the three-dimensional coordinates of the characteristic point in real time, thereby realizing the positioning of the body in a working surface space.
Description
Technical Field
The invention relates to the technical field of position and posture detection of underground mine cantilever type development machines, in particular to a development machine absolute position and posture detection method based on image recognition and multi-sensor fusion.
Background
With the improvement of the mechanization level of mines, the adoption of a cantilever type tunneling machine for mining becomes a standard production mode of related enterprises. Under the call of no humanization of a dangerous production environment in the current national construction of digital mines, in recent years, part of coal mines start to use vibration and temperature sensors to detect vibration and temperature parameters of the development machine, and use laser sensors and video pictures to guide ground workers to operate the underground development machine, so that the number of times that the workers enter the first-line operation can be reduced, but the automation degree is low. Some researches apply multiple sensors to acquire displacement and attitude information, an embedded system solves a complex equation to solve the pose, the automation degree is improved, and the accuracy is low due to the fact that sensor data have errors and the data volume is large. Therefore, it is necessary to design a set of intelligent positioning detection method for the heading machine suitable for the mine environment, and with the improvement of computer image recognition technology, unmanned working faces with large data volume increasingly adopt image recognition to obtain target information. In view of the complex environment of the working face of the coal mining heading machine, the accuracy and the stability can be improved by adopting the method of combining the image recognition and the multiple sensors, and the method meets the requirements of the digital development of modern mines.
Disclosure of Invention
The purpose of the invention is as follows: the invention aims to provide an intelligent positioning detection method of a heading machine, which is suitable for mine environment.
The technical scheme is as follows: in order to achieve the purpose, the invention provides the following technical scheme:
the method for detecting the absolute pose of the heading machine based on image recognition and multi-sensor fusion comprises the following steps:
(1) defining a surface to be excavated by the development machine as a working surface; installing a laser indicator lamp at the center of one surface of the development machine body facing to the working surface, wherein the laser indicator lamp projects a target pattern on the working surface; installing a left camera on one surface of a left tunneling arm of the tunneling machine, which faces a working surface, installing a right camera on one surface of a right tunneling arm, which faces the working surface, and simultaneously acquiring a target pattern on the working surface by the left camera and the right camera; a displacement sensor is arranged in an electric control box of the heading machine and is used for acquiring the displacement of the heading machine to a working surface in unit time; the method comprises the following steps that gyroscopes are respectively installed at the center of a left tunneling arm, the center of a right tunneling arm and the center of a machine body of the tunneling machine, and the three gyroscopes are used for respectively outputting three-axis angular speed and rotation amount under a coordinate system of the three gyroscopes;
(2) selecting a certain pixel point in the cross-shaped pattern as a characteristic point; respectively calculating pixel coordinates of the feature points in the imaging planes of the left camera and the right camera;
(3) establishing an equivalent constraint equation of the feature points in the left camera coordinate system and the right camera coordinate system:
AX＝0
X＝[x x′]
a is a homogeneous coordinate matrix of the characteristic point in a left camera imaging coordinate system, x is a pixel coordinate of the characteristic point in a left camera imaging plane, x 'is a pixel coordinate of the characteristic point in a camera right imaging plane, p is a projection matrix of the left camera, and p' is a projection matrix of the right camera;
(4) calculating according to equivalent constraint equations of the feature points in the left camera coordinate system and the right camera coordinate system:
b represents the coordinates of the characteristic points in a coordinate system of the machine body of the heading machine, and theta is a deflection angle between the axes of the left camera shooting lens and the right camera shooting lens;
(5) respectively calculating attitude equations of the heading machine at the positions of the three gyroscopes according to the measurement results of the displacement sensor and the three gyroscopes, wherein the attitude equation f at the ith gyroscope is as follows:
wherein, deltay is the displacement of the heading machine to the working face in unit time measured by the displacement sensor, miIs the measurement vector of the ith gyroscope, mi＝[m1，m2，m3，m4，m5，m6，m7]，m1Measuring angular velocity, m, of x-axis rotation for a gyroscope2Measuring angular velocity, m, of y-axis rotation for a gyroscope3Measuring angular velocity, m, of z-axis rotation for a gyroscope4The component of the vector part of the rotation in the x-axis, m, is measured for the gyroscope5Measuring the component of the vector part of the rotation in the y-axis, m, for the gyroscope6Measuring the component of the vector part of the rotation in the z-axis, m, for the gyroscope7Measuring a scalar quantity of rotation, τ, for the gyroscopeiIs the time constant of the ith gyroscope, TsIs a preset sampling period of the gyroscope.
Furthermore, the left camera and the right camera are mining intrinsic safety cameras.
Furthermore, the three gyroscopes are all contained in a mining explosion-proof box.
Has the advantages that: compared with the prior art, the invention has the following advantages:
1. the absolute pose of the coal mining machine is resolved in real time by combining the image, the displacement and the acceleration information, the accuracy and the stability are improved compared with the traditional sensor sensing, and the method meets the requirements of the modern mine digital development.
2. A camera visual angle estimation method based on a fitting polynomial is designed, and a relative deflection angle is selected to enable the camera to have the maximum coincident visual field, so that the three-dimensional reconstruction of the camera body has a large dynamic range.
Detailed Description
The present invention will be further described with reference to the following examples.
The present embodiment adopts the following hardware devices: the mining safety type video camera system comprises three mining safety type video cameras, a cross-shaped laser indicator lamp, a displacement sensor and three electronic gyroscopes, wherein hardware equipment has a wireless communication function and transmits acquired data to a background computer through a wireless network at regular time; the three mining intrinsic safety cameras are respectively arranged on a left digging arm, a right digging arm and a rear transportation port of the digging machine, and acquire images at the rate of 30 frames per second; the cross-shaped laser indicator lamp is arranged in the middle of the front part of the development machine; the displacement sensor is arranged in an electric control box of the heading machine and transmits displacement signals every 1 s. Three electronic gyroscopes are sealed in a mining explosion-proof box, wherein two electronic gyroscopes are respectively arranged on a left tunneling arm and a right tunneling arm of the tunneling machine, and the other electronic gyroscope is arranged at the geometric center of the tunneling machine body.
The working process of the invention is as follows:
(1) recording the surface to be excavated by the development machine as a working surface; a cross-shaped laser indicator lamp is arranged at the center of one surface of the tunneling machine body facing to the working surface, and projects a cross-shaped pattern on the working surface; cameras on the left arm and the right arm of the development machine simultaneously acquire cross-shaped patterns on a working surface; a displacement sensor in an electric control box of the development machine acquires the displacement of the development machine to the working face; acquiring three-axis acceleration and attitude quaternion under a self coordinate system respectively at two tunneling arms of the tunneling machine and a gyroscope at the center of a machine body;
(2) after receiving the collected data, the background computer executes the following steps:
(2-1) selecting a certain pixel point in the cross-shaped pattern as a characteristic point; respectively calculating pixel coordinates of the feature points in the imaging planes of the left camera and the right camera;
(2-2) establishing an equivalent constraint equation of the feature points in the left camera coordinate system and the right camera coordinate system:
AX＝0
x＝[x x′]
a is a homogeneous coordinate matrix of the characteristic point in a left camera imaging coordinate system, x is a pixel coordinate of the characteristic point in a left camera imaging plane, x 'is a pixel coordinate of the characteristic point in a camera right imaging plane, p is a projection matrix of the left camera, and p' is a projection matrix of the right camera;
(2-3) calculating according to equivalent constraint equations of the feature points in the left camera coordinate system and the right camera coordinate system:
and further calculating:
b represents the coordinates of the characteristic points in a coordinate system of the heading machine body, and theta is a deflection angle between the axes of the left camera shooting lens and the right camera shooting lens;
(2-4) calculating an attitude equation f of the heading machine according to the measurement results of the displacement sensor and the three gyroscopes:
wherein, deltay is the displacement of the heading machine to the working face in unit time measured by the displacement sensor, miIs the measurement vector of the ith gyroscope, mi＝[m1，m2，m3，m4，m5，m6，m7]，m1Measuring angular velocity, m, of x-axis rotation for a gyroscope2Measuring angular velocity, m, of y-axis rotation for a gyroscope3Measuring angular velocity, m, of z-axis rotation for a gyroscope4The component of the vector part of the rotation in the x-axis, m, is measured for the gyroscope5Measuring the component of the vector part of the rotation in the y-axis, m, for the gyroscope6Measuring the component of the vector part of the rotation in the z-axis, m, for the gyroscope7Is a topMeasuring a scalar quantity of rotation, τiIs the time constant of the ith gyroscope, TsIs a preset sampling period of the gyroscope.
The above description is only of the preferred embodiments of the present invention, and it should be noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.
Claims (3)
1. The method for detecting the absolute pose of the heading machine based on image recognition and multi-sensor fusion is characterized by comprising the following steps of:
(1) defining a surface to be excavated by the development machine as a working surface; installing a laser indicator lamp at the center of one surface of the development machine body facing to the working surface, wherein the laser indicator lamp projects a target pattern on the working surface; installing a left camera on one surface of a left tunneling arm of the tunneling machine, which faces a working surface, installing a right camera on one surface of a right tunneling arm, which faces the working surface, and simultaneously acquiring a target pattern on the working surface by the left camera and the right camera; a displacement sensor is arranged in an electric control box of the heading machine and is used for acquiring the displacement of the heading machine to a working surface in unit time; the method comprises the following steps that gyroscopes are respectively installed at the center of a left tunneling arm, the center of a right tunneling arm and the center of a machine body of the tunneling machine, and the three gyroscopes are used for respectively outputting three-axis angular speed and rotation amount under a coordinate system of the three gyroscopes;
(2) selecting a certain pixel point in the target pattern as a characteristic point; respectively calculating pixel coordinates of the feature points in the imaging planes of the left camera and the right camera;
(3) establishing an equivalent constraint equation of the feature points in the left camera coordinate system and the right camera coordinate system:
AX＝0
x＝[x x′]
a is a homogeneous coordinate matrix of the characteristic point in a left camera imaging coordinate system, x is a pixel coordinate of the characteristic point in a left camera imaging plane, x 'is a pixel coordinate of the characteristic point in a camera right imaging plane, p is a projection matrix of the left camera, and p' is a projection matrix of the right camera;
(4) calculating according to equivalent constraint equations of the feature points in the left camera coordinate system and the right camera coordinate system:
b represents the coordinates of the characteristic points in a coordinate system of the machine body of the heading machine, and theta is a deflection angle between the axes of the left camera shooting lens and the right camera shooting lens;
(5) respectively calculating attitude equations of the heading machine at the positions of the three gyroscopes according to the measurement results of the displacement sensor and the three gyroscopes, wherein the attitude equation f at the ith gyroscope is as follows:
wherein, deltay is the displacement of the heading machine to the working face in unit time measured by the displacement sensor, miIs the measurement vector of the ith gyroscope, mi＝[m1，m2，m3，m4，m5，m6，m7]，m1Measuring angular velocity, m, of x-axis rotation for a gyroscope2Measuring angular velocity, m, of y-axis rotation for a gyroscope3Measuring angular velocity, m, of z-axis rotation for a gyroscope4The component of the vector part of the rotation in the x-axis, m, is measured for the gyroscope5Measuring the component of the vector part of the rotation in the y-axis, m, for the gyroscope6Measuring the component of the vector part of the rotation in the z-axis, m, for the gyroscope7Measuring a scalar quantity of rotation, τ, for the gyroscopeiIs the time constant of the ith gyroscope, TsIs a preset sampling period of the gyroscope.
2. The method for detecting the absolute pose of the heading machine based on the image recognition and the multi-sensor fusion as claimed in claim 1, wherein the left camera and the right camera are mining intrinsic safety cameras.
3. The method for detecting the absolute pose of the heading machine based on the image recognition and the multi-sensor fusion as claimed in claim 1, wherein the three gyroscopes are all contained in a mining explosion-proof box.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910139486.XA CN109839109B (en) | 2019-02-25 | 2019-02-25 | Development machine absolute pose detection method based on image recognition and multi-sensor fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910139486.XA CN109839109B (en) | 2019-02-25 | 2019-02-25 | Development machine absolute pose detection method based on image recognition and multi-sensor fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109839109A CN109839109A (en) | 2019-06-04 |
| CN109839109B true CN109839109B (en) | 2020-11-24 |
Family
ID=66884961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910139486.XA Active CN109839109B (en) | 2019-02-25 | 2019-02-25 | Development machine absolute pose detection method based on image recognition and multi-sensor fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109839109B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111879314B (en) * | 2020-08-10 | 2022-08-02 | 中国铁建重工集团股份有限公司 | Multi-sensor fusion roadway driving equipment real-time positioning system and method |
| CN112050732B (en) * | 2020-08-21 | 2022-03-15 | 西安科技大学 | Method and system for automatically detecting spatial pose of cantilever type heading machine |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201273195Y (en) * | 2008-10-16 | 2009-07-15 | 中国矿业大学 | Position detection apparatus of fully-mechanized mining face coal mining machine |
| EP2189753A3 (en) * | 2008-11-24 | 2011-01-05 | Politechnika Slaska | Assembly stand for mineral cutting head |
| CN201568056U (en) * | 2009-12-17 | 2010-09-01 | 中国矿业大学 | Orientation laser intelligent receiving target |
| CN104729501A (en) * | 2015-03-19 | 2015-06-24 | 中国矿业大学(北京) | Rotating-sector-laser-based position and pose measurement method of cantilever excavator |
| CN105136134A (en) * | 2015-08-10 | 2015-12-09 | 兖州煤业股份有限公司 | Heading machine position and posture detection and adjustment method and system |
| CN105115499B (en) * | 2015-09-09 | 2018-07-06 | 中船重工（青岛）轨道交通装备有限公司 | Guidance system and localization method applied to Double shield TBM |
| CN107014379B (en) * | 2017-05-25 | 2019-09-20 | 中国矿业大学 | A kind of development machine absolute space apparatus for detecting position and posture and method |
| CN108731672B (en) * | 2018-05-30 | 2022-03-11 | 中国矿业大学 | Coal mining machine attitude detection system and method based on binocular vision and inertial navigation |
| CN108592951B (en) * | 2018-05-30 | 2019-08-02 | 中国矿业大学 | A kind of coalcutter inertial navigation Initial Alignment Systems and method based on optical flow method |
| CN109115172B (en) * | 2018-08-23 | 2020-09-08 | 西安科技大学 | Machine vision-based method for detecting pose of body of coal mining machine |
-
2019
- 2019-02-25 CN CN201910139486.XA patent/CN109839109B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109839109A (en) | 2019-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111156998B (en) | Mobile robot positioning method based on RGB-D camera and IMU information fusion | |
| CN110487267B (en) | Unmanned aerial vehicle navigation system and method based on VIO & UWB loose combination | |
| US20160260250A1 (en) | Method and system for 3d capture based on structure from motion with pose detection tool | |
| KR101768958B1 (en) | Hybird motion capture system for manufacturing high quality contents | |
| US20040176925A1 (en) | Position/orientation measurement method, and position/orientation measurement apparatus | |
| CN107289910B (en) | Optical flow positioning system based on TOF | |
| WO2015134795A2 (en) | Method and system for 3d capture based on structure from motion with pose detection tool | |
| CN109540135B (en) | Method and device for detecting pose and extracting yaw angle of paddy field tractor | |
| CN102798456B (en) | Method, device and system for measuring working range of engineering mechanical arm frame system | |
| CN113140040A (en) | Multi-sensor fusion coal mine underground space positioning and mapping method and device | |
| CN109839109B (en) | Development machine absolute pose detection method based on image recognition and multi-sensor fusion | |
| CN112116651B (en) | Ground target positioning method and system based on monocular vision of unmanned aerial vehicle | |
| CN112611380B (en) | Attitude detection method based on multi-IMU fusion and attitude detection device thereof | |
| CN115371665B (en) | Mobile robot positioning method based on depth camera and inertial fusion | |
| CN110132309A (en) | A kind of rocker arm of coal mining machine inertia/visual combination determines appearance device normalization method | |
| CN114608554B (en) | Handheld SLAM equipment and robot instant positioning and mapping method | |
| CN112697131A (en) | Underground mobile equipment positioning method and system based on vision and inertial navigation system | |
| CN110864692A (en) | Pose determination method of heading machine | |
| CN111025330B (en) | Target inclination angle detection method and device based on depth map | |
| CN106969721A (en) | A kind of method for three-dimensional measurement and its measurement apparatus | |
| CN108444468B (en) | Directional compass integrating downward vision and inertial navigation information | |
| CN113474819A (en) | Information processing apparatus, information processing method, and program | |
| CN110411443B (en) | Coal mining machine rocker arm inertia/vision combined attitude determination device and method | |
| CN113701750A (en) | Fusion positioning system of underground multi-sensor | |
| CN210268669U (en) | Inertial navigation positioning system for underground mining mobile equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |