CN111895931B - Coal mine operation area calibration method based on computer vision - Google Patents
Coal mine operation area calibration method based on computer vision Download PDFInfo
- Publication number
- CN111895931B CN111895931B CN202010694534.4A CN202010694534A CN111895931B CN 111895931 B CN111895931 B CN 111895931B CN 202010694534 A CN202010694534 A CN 202010694534A CN 111895931 B CN111895931 B CN 111895931B
- Authority
- CN
- China
- Prior art keywords
- excavator
- transport vehicle
- dimensional
- dimensional detection
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
- G01B21/02—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness
- G01B21/04—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness by measuring coordinates of points
- G01B21/042—Calibration or calibration artifacts
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Abstract
The invention discloses a coal mine operation area calibration method based on computer vision, which relates to the technical field of coal mine operation area calibration and comprises the following steps: building and training a deep learning neural network model for two-dimensional detection and identification of an object; building and training a deep learning neural network model for three-dimensional detection of an object; collecting RGB images as the input of the deep learning neural network model for the two-dimensional detection and identification of the object, and acquiring a two-dimensional detection frame and a corresponding model of the working vehicle; cutting the RGB picture content in the obtained two-dimensional detection frame as the input of the deep learning neural network model of the three-dimensional detection, and obtaining the three-dimensional detection frame of the base of the working vehicle; determining pixel specification characteristics of the work vehicle in the image; specification characteristics of the work vehicle in practice are determined. The method and the device for detecting the working area of the working vehicle obtain the three-dimensional working area of the working vehicle in the image, determine the area where the working vehicle should arrive, help the working vehicle to work in the specified working area and reduce potential safety hazards.
Description
Technical Field
The invention relates to the technical field of coal mine operation area calibration, in particular to a coal mine operation area calibration method based on computer vision.
Background
In coal mine operation, whether a vehicle is driven by a person or an unmanned vehicle, the vehicle needs to work in a designated working area, and once the vehicle is out of range, serious production accidents can be caused. The camera monitoring is widely applied in real life, and can be conveniently installed in various scenes without complicated configuration. At present, technologies such as automatic driving, factory automatic operation and the like are assisted by a computer vision method, and good effects are obtained. The method of the deep learning technology of the convolutional neural network can automatically extract the characteristics of the object and carry out accurate positioning and classification. However, the traditional two-dimensional detection of objects cannot accurately define the working area around the excavator according to the distance.
Therefore, the operation area needs to be calibrated, and the vehicle in operation needs to be tracked in real time, so that the vehicle can only work in the calibrated area. The current practice is to use UWB (ultra wide band) positioning devices with fixed positions outside the working area, to set mobile UWB positioning devices in the vehicle, and to calculate the position of the vehicle by measuring the time from when a pulse signal starts from the fixed UWB positioning device to when the mobile UWB positioning device returns to the fixed UWB positioning platform in the vehicle, in order to detect whether the vehicle crosses the boundary of the working area.
In practical application, position coordinates need to be calibrated between fixed UWB devices and between the fixed UWB devices and the vehicle-mounted mobile UWB device. However, in a shallow coal mine operation area, the position and the landform of an operation place are frequently changed, soil layers, coal layers and blasting layers are staggered, the combination of all the layers is not regular, and earthwork is repeatedly peeled off, so that the operation area is difficult to calibrate by the conventional technology, a common UWB positioning scheme needs to move the position in a large range along with the change of the operation area, and once the UWB positioning scheme is moved, a professional is required to arrive at the operation area to refresh a coordinate system again, so that the UWB positioning system is inconvenient to use.
An effective solution to the problems in the related art has not been proposed yet.
Disclosure of Invention
Aiming at the problems in the related art, the invention provides a coal mine operation area calibration method based on computer vision, so as to overcome the technical problems in the prior related art.
The technical scheme of the invention is realized as follows:
a coal mine operation area calibration method based on computer vision comprises the following steps:
step S1, building and training a deep learning neural network model for two-dimensional detection and identification of an object;
step S2, building and training a deep learning neural network model for three-dimensional detection of an object;
step S3, collecting RGB images as the input of the deep learning neural network model for the two-dimensional detection and identification of the object, and acquiring a two-dimensional detection frame and a corresponding model of the working vehicle;
step S4, cutting the content of the RGB image in the two-dimensional detection frame as the input of the deep learning neural network model of the three-dimensional detection, and obtaining the three-dimensional detection frame of the base of the working vehicle;
step S5, determining pixel specification characteristics of the work vehicle in the image;
step S6, determining the specification characteristics of the work vehicle in practice;
in step S7, a three-dimensional work area of the work vehicle in the image is determined.
Further, the deep learning neural network model for two-dimensional detection and recognition of the object is built and trained, and the deep learning neural network model comprises the following steps:
collecting RGB images of a mining operation area containing operation vehicles;
manually marking a two-dimensional marking frame of the operation vehicle in the collected RGB image, and marking the model of the operation vehicle to obtain a marked image;
and training to obtain a deep learning neural network model for two-dimensional detection and identification of the object by taking the collected RGB image as input and taking the two-dimensional marking frame and the marking model as labels.
Further, the deep learning neural network model for building and training the three-dimensional detection of the object comprises the following steps:
cutting the region of the working vehicle according to the obtained marked images, applying a new data set to the cut region of the working vehicle, and manually marking a three-dimensional marking frame of the chassis of the working vehicle;
and taking the cut working vehicle region as input, taking a three-dimensional labeling frame of the working vehicle region as a label, and training to obtain a deep learning neural network for three-dimensional detection of the object.
Further, the image resolution of the deep learning neural network model for inputting the two-dimensional detection and identification of the object is 512 multiplied by 512; the image resolution of the deep learning neural network for three-dimensional detection of an input object is 64 × 64.
Further, the captured RGB image includes work vehicle element characteristic information.
Further, the work vehicle elements include a transport vehicle and an excavator.
Further, the collected RGB image is a high-definition camera, and the distance between the camera and the working area is less than 200 meters.
Further, the method comprises the following steps:
step S8, instructing the work vehicle to drive into the work area,
and step S9, determining whether the three-dimensional detection frame of the working vehicle is matched with the acquired three-dimensional working area, and if so, executing a working instruction.
The invention has the beneficial effects that:
the invention relates to a coal mine operation area calibration method based on computer vision, which comprises the steps of acquiring RGB images as the input of a deep learning neural network model for object two-dimensional detection and identification, acquiring a two-dimensional detection frame and a corresponding model of an operation vehicle, cutting the RGB image content in the acquired two-dimensional detection frame as the input of the deep learning neural network model for three-dimensional detection, acquiring a three-dimensional detection frame of an operation vehicle base, determining the pixel specification characteristics of the operation vehicle in the image and the specification characteristics of the operation vehicle in practice, thereby acquiring a three-dimensional working area of the operation vehicle in the image, determining the area which the operation vehicle should reach, helping the operation vehicle to operate in the specified working area and reducing potential safety hazards.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic flow chart of a coal mine operation area calibration method based on computer vision according to an embodiment of the invention;
FIG. 2 is a schematic flow chart of three-dimensional operating area calibration of a coal mine operating area calibration method based on computer vision according to an embodiment of the invention;
FIG. 3 is a schematic diagram of deep convolutional neural network training for two-dimensional vehicle detection and identification based on a coal mine operation area calibration method based on computer vision according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of deep convolutional neural network training for vehicle three-dimensional detection based on a coal mine operation area calibration method based on computer vision according to an embodiment of the invention;
FIG. 5 is a schematic diagram of a deep convolutional neural network framework of a vehicle two-dimensional detection and recognition model based on a computer vision-based coal mine operation area calibration method according to an embodiment of the invention;
FIG. 6 is a schematic diagram of a deep convolutional neural network framework of a vehicle three-dimensional detection model of a coal mine operation area calibration method based on computer vision according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present invention.
According to the embodiment of the invention, a coal mine operation area calibration method based on computer vision is provided.
As shown in fig. 1-2, a method for calibrating a coal mine operation area based on computer vision according to an embodiment of the present invention includes the following steps: step S1, building and training a deep learning neural network model for two-dimensional detection and identification of an object;
step S2, building and training a deep learning neural network model for three-dimensional detection of an object;
step S3, collecting RGB images as the input of the deep learning neural network model for the two-dimensional detection and identification of the object, and acquiring a two-dimensional detection frame and a corresponding model of the working vehicle;
step S4, cutting the content of the RGB image in the two-dimensional detection frame as the input of the deep learning neural network model of the three-dimensional detection, and obtaining the three-dimensional detection frame of the base of the working vehicle;
step S5, determining pixel specification characteristics of the work vehicle in the image;
step S6, determining the specification characteristics of the work vehicle in practice;
in step S7, a three-dimensional work area of the work vehicle in the image is determined.
The method comprises the following steps of constructing and training a deep learning neural network model for two-dimensional detection and identification of an object, wherein the deep learning neural network model comprises the following steps:
collecting RGB images of a mining operation area containing operation vehicles;
manually marking a two-dimensional marking frame of the operation vehicle in the collected RGB image, and marking the model of the operation vehicle to obtain a marked image;
and training to obtain a deep learning neural network model for two-dimensional detection and identification of the object by taking the collected RGB image as input and taking the two-dimensional marking frame and the marking model as labels.
The deep learning neural network model for building and training the three-dimensional detection of the object comprises the following steps:
cutting the region of the working vehicle according to the obtained marked images, applying a new data set to the cut region of the working vehicle, and manually marking a three-dimensional marking frame of the chassis of the working vehicle;
and taking the cut working vehicle region as input, taking a three-dimensional labeling frame of the working vehicle region as a label, and training to obtain a deep learning neural network for three-dimensional detection of the object.
The image resolution of the deep learning neural network model for inputting the two-dimensional detection and identification of the object is 512 multiplied by 512; the image resolution of the deep learning neural network for three-dimensional detection of an input object is 64 × 64.
Wherein the captured RGB image includes work vehicle element feature information.
Wherein the work vehicle elements include a transport vehicle and an excavator.
The RGB image acquisition device comprises a high-definition camera, a working area and a control panel, wherein the RGB image acquisition device is a high-definition camera, and the distance between the camera and the working area is less than 200 meters.
The method comprises the following steps:
step S8, instructing the work vehicle to drive into the work area,
and step S9, determining whether the three-dimensional detection frame of the working vehicle is matched with the acquired three-dimensional working area, and if so, executing a working instruction.
By means of the technical scheme, the coal mine working area calibration method based on computer vision acquires RGB images as input of a deep learning neural network model for two-dimensional detection and identification of an object, acquires a two-dimensional detection frame and a corresponding model of a working vehicle, cuts the content of the RGB images in the acquired two-dimensional detection frame as input of the deep learning neural network model for three-dimensional detection, acquires a three-dimensional detection frame of a working vehicle base, determines pixel specification characteristics of the working vehicle in the images and specification characteristics of the working vehicle in practice, accordingly acquires a three-dimensional working area of the working vehicle in the images, determines an area where the working vehicle should arrive, helps the working vehicle to work in the specified working area, and reduces potential safety hazards.
In addition, specifically, in order to train the two-dimensional detection model and the three-dimensional detection model, 1000 videos of mine operation are collected, and twenty thousand pictures containing the excavator and the transport vehicle at the same time are selected from the videos, wherein the resolution ratio is 512 multiplied by 512. And taking the manually marked two-dimensional detection frames and model types of the excavators and the transport vehicles as labels of the two-dimensional object detection model. After marking the two-dimensional detection frame, cutting out the area of the vehicle and zooming the area into the size of 64 multiplied by 64 to be used as input data of the three-dimensional detection model, manually marking the base of the excavator and the three-dimensional marking frame of the transport vehicle, and using the three-dimensional marking frame as a label of the three-dimensional detection model.
The working flow of the method is shown in fig. 2, firstly, a video monitored by a monitoring camera is converted into a picture as input, and after the picture passes through a two-dimensional detection model, a two-dimensional detection frame and a corresponding model of a transport vehicle and an excavator in the picture are obtained. Then, the obtained areas of the transporter and the excavator are cut out, the sizes of the areas are scaled to 64 × 64, and the areas are input into a three-dimensional detection model to obtain a three-dimensional detection frame of the chassis of the transporter and the excavator. The actual length and width of the chassis of the excavator are found out according to the model of the excavator, the pixel length occupied by the length and width of the chassis is calculated according to the obtained three-dimensional detection frame of the chassis of the excavator, the relation between the actual length and width of the chassis of the excavator and the pixel length is obtained according to the actual length and width of the chassis of the excavator, and the working interval where the transport vehicle should reach is calculated by combining the arm length of the excavator. And sending an instruction to enable the transport vehicle to reach a specified interval, and sending the reached instruction when the three-dimensional detection frame of the transport vehicle is basically coincided with the working interval.
Specifically, in practical application, as shown in fig. 3 to 6, the following links are included:
firstly, acquiring and processing picture data:
select suitable angle in mining area and place the camera, the camera is not more than 200 meters apart from the job scene, lets the camera can be complete see the work flow between transport vechicle and the excavator. And selecting 1000 videos in different scenes, selecting one picture every 30 frames in order to prevent the data set from being overlarge, and discarding pictures which do not comprise the excavator and the transport vehicle. After the picture data set is obtained, manually marking the two-dimensional marking frames and the corresponding models of the transport vehicle and the excavator in the RGB image, and using the categories corresponding to the coordinates, the length, the height and the models of the center points of the two-dimensional marking frames as labels of the two-dimensional detection and identification model. In order to train the three-dimensional detection model, the marked two-dimensional frame picture is cut out and scaled to 64 × 64, and the three-dimensional frame picture is marked on the vehicle picture manually as the input of the three-dimensional frame detection model. Due to the view angle of the vehicle, only three vertexes of the lower bottom surface of the vehicle three-dimensional detection frame are generally visible in the picture, and in the manual labeling process, the confidence coefficient of the visible vertexes of the lower bottom surface of the three-dimensional labeling frame is set to be 1, and the confidence coefficient of the invisible vertexes of the lower bottom surface of the three-dimensional labeling frame is set to be 0. During training, the coordinates and confidence degrees of the 4 vertexes of the lower bottom surface and the height of the three-dimensional detection frame are used as labels. When predicting, only three vertexes with the highest confidence degree and a high-reconstruction three-dimensional detection frame are selected.
Secondly, constructing and training an object two-dimensional detection and recognition model:
the method comprises the steps of taking pictures containing a transport vehicle and an excavator as input, taking coordinates, length and width of a vehicle center point marked manually as labels of a two-dimensional marking frame, and taking the type of the vehicle as a classified label.
TABLE 1 types of anchor boxes
| Anchor box type | Size of |
| C1 | 66×54 |
| C2 | 130×100 |
| C3 | 177×243 |
| C4 | 166×130 |
| C5 | 260×200 |
The two-dimensional object detection model of the vehicle adopts a neural network based on two-dimensional convolution as a main detection network, and for an input picture with the size of 512 x 512, the size of a final feature layer is 16 x 16. For each feature point of the last layer, five vehicle detection frames are predicted, and the five vehicle detection frames correspond to 5 anchor frames (an anchor refers to a pixel point which may exist in the video object. The predicted value of each vehicle detection box contains the coordinates of the center point of the vehicle, the offset values for the length and width of the anchor point box, the probability of each type of model, and the probability of whether or not an object is contained.
The model is trained using two loss functions simultaneously. Identifying the vehicle model by using a binary cross entropy between a prediction category and a calibration category as a loss function; and detecting the position of the vehicle by adopting the mean square error loss between the detected center point coordinate, the detected length and the detected high offset value and the calibrated center point coordinate, the calibrated length and the calibrated high offset value as a loss function. Based on the actual vehicle size in the mine, 5 anchor blocks were designed, and their sizes are shown in table 1.
During training, firstly, the characteristics of the picture are extracted through a backbone network, five detection frames predicted by each characteristic point are output at the last layer, and the detection frame with the IOU (the intersection and union ratio of the predicted detection frame and the manual labeling frame) larger than a certain proportion is selected to calculate the loss function. The learning rate of the model is set to be 0.01-0.001, the sum of mean square errors is used as a loss function in coordinate regression, cross entropy is used as a loss function in vehicle model classification, the learning rate is attenuated by one half every 10 generations, and 60 batches are trained.
Thirdly, building and training a three-dimensional object detection model:
the vehicle and excavator area in the picture is cut off and then scaled to 64 × 64, and the size is input into the object three-dimensional detection model for training. The network for three-dimensional frame detection comprises four convolutional layers, a global pooling layer and two full-connection layers. The final output dimensionality of the full connection layer is 1 multiplied by 13, coordinates and corresponding confidence degrees of four vertexes of the lower bottom surface of the three-dimensional detection frame are sequentially represented, and the height of the three-dimensional detection frame is further represented. And during prediction, selecting three vertexes with the maximum confidence coefficient and a high-reconstruction three-dimensional labeling frame. The network structure of the three-dimensional inspection model is shown in table 2.
And (3) adopting the mean square error loss as a loss function for the coordinates and the confidence degrees of the four vertexes of the bottom side quadrangle of the last layer and the 13 values of the height of the three-dimensional labeling box (the two-dimensional coordinates of the four vertexes account for two values, the total value is 8 values, the confidence degrees of the 4 vertexes are 4 values, the height is one value, and the total value is 13 values).
During training, an Adam or SGD optimizer is adopted as with the two-dimensional detection model, the initial learning rate is set to 0.01, the attenuation is half per 10 generations, the minimum attenuation is 0.001, and 60 epochs are trained in total (the learning rate, Adam, SGD and Epoch are common terms in the field of deep learning).
TABLE 2 three-dimensional vehicle detection network architecture
Fourthly, calibrating a working area:
in the actual operation process of the method, mining area operation pictures obtained by the camera in real time sequentially pass through the two-dimensional detection and identification model and the three-dimensional detection model, so that the model of the excavator and the chassis three-dimensional detection frame can be obtained.
In order to calculate the working area of the transport vehicle, the relationship between the surrounding pixel points of the excavator in the picture and the actual distance needs to be known. And finding out the actual length and width of the excavator chassis according to the model, obtaining the pixel length in the picture according to the three-dimensional detection frame of the chassis, and establishing the corresponding relation between the pixel and the actual length. And finally, calculating the working interval where the transport vehicle needs to be located according to the arm length of the excavator and the position of the chassis.
Assuming that the number of pixel points occupied by the width of the chassis of the excavator in the picture is x, and the actual width of the chassis is y meters, and obtaining the ratio k of x to y. The arm length of the excavator is h meters according to the model of the excavator, and the optimal distance between the transport vehicle and the excavator is 2/3 arm lengths of the excavator. Therefore, the distance between the vehicle and the excavator in the picture is required to be 2hk/3 pixel points, and therefore the three-dimensional working area of the transport vehicle is parallelly arranged at 2hk/3 pixel points away from the left side surface and the right side surface of the three-dimensional frame of the excavator, namely, the position corresponding to 2/3h meters away from the left side surface or the right side surface of the excavator in the real world.
Fifthly, positioning of the transport vehicle:
and after the operation interval of the transport vehicle in the image is determined, the three-dimensional detection of the vehicle is carried out in the video frame acquired in real time. When the transport vehicle in the image is 2hk/3 pixel points away from the excavator (namely, the transport vehicle reaches the position of 2/3h meters away from the left side surface or the right side surface of the excavator in the real world), and the three-dimensional detection frame of the transport vehicle is approximately matched with the three-dimensional detection frame of the working interval, sending a reached instruction; and if not, continuing to send instructions to enable the transport vehicle to enter the calibrated station.
In summary, with the aid of the technical solutions of the present invention, through
The method comprises the steps of acquiring an RGB image as the input of a deep learning neural network model for two-dimensional detection and identification of an object, acquiring a two-dimensional detection frame and a corresponding model of the working vehicle, cutting the acquired two-dimensional detection frame as the input of the deep learning neural network model for three-dimensional detection, acquiring a three-dimensional detection frame of a base of the working vehicle, determining the pixel specification characteristics of the working vehicle in the image and the specification characteristics of the working vehicle in practice, thereby acquiring a three-dimensional working area of the working vehicle in the image, determining the area where the working vehicle should arrive, helping the working vehicle to work in a specified working area, and reducing potential safety hazards.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (3)
1. A coal mine operation area calibration method based on computer vision is characterized by comprising the following steps:
(1) acquiring RGB pictures of a transport vehicle and an excavator in a mining operation area;
(2) manually marking two-dimensional marking frames of the transport vehicle and the excavator in the RGB image, and meanwhile, marking the models of the transport vehicle and the excavator;
(3) cutting the areas of the transport vehicle and the excavator marked in the step (2), taking the image areas where the transport vehicle and the excavator are located as a new data set, and manually marking three-dimensional marking frames of chassis of the transport vehicle and the excavator;
(4) taking an RGB image containing a transport vehicle and an excavator as input, taking two-dimensional marking frames and models of the transport vehicle and the excavator as labels, and training a deep learning neural network model for two-dimensional object detection and identification;
(5) taking the cut excavator and transport vehicle pictures as input, taking the three-dimensional marking frames of the transport vehicle and the excavator base as labels, and training a deep learning neural network model for three-dimensional detection of an object;
(6) using RGB pictures acquired by a camera in real time as input of a trained object two-dimensional detection and recognition deep learning neural network model to detect two-dimensional detection frames and corresponding models of a transport vehicle and an excavator;
(7) cutting the vehicle area of the two-dimensional detection frame obtained in the step (6) as the input of a deep learning neural network model for three-dimensional detection of the object, and detecting the three-dimensional detection frames of the transport vehicle and the excavator base;
(8) calculating the length and width of pixels in a picture according to a three-dimensional detection frame of the chassis of the excavator;
(9) determining the actual length and width of the excavator chassis according to the identified excavator model;
(10) synthesizing the pixel length and the actual length and width of the chassis of the excavator in the picture to obtain a ratio, and determining the working interval of the transport vehicle according to the ratio and the safe distance between the excavator and the transport vehicle in the picture, wherein the safe distance is a multiple of the arm length of the excavator, and the arm length of the known excavator;
(11) and (3) after the working interval of the transport vehicle is determined, instructing the transport vehicle to drive into the working interval, and when the distance between the transport vehicle and the excavator in the image is a safe distance and the three-dimensional detection frame of the transport vehicle is matched with the working interval of the transport vehicle determined in the step (10), issuing an instruction of reaching the working interval to the transport vehicle.
2. The computer vision-based coal mine working area calibration method as claimed in claim 1, wherein the image resolution of the deep learning neural network model for two-dimensional detection and identification of the input object is 512 x 512; the image resolution of the deep learning neural network model for three-dimensional detection of an input object is 64 × 64.
3. The computer vision-based coal mine working area calibration method as claimed in claim 1, wherein the equipment for collecting RGB images is a high-definition camera, and the distance from the high-definition camera to the working area is less than 200 m.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010694534.4A CN111895931B (en) | 2020-07-17 | 2020-07-17 | Coal mine operation area calibration method based on computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010694534.4A CN111895931B (en) | 2020-07-17 | 2020-07-17 | Coal mine operation area calibration method based on computer vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111895931A CN111895931A (en) | 2020-11-06 |
| CN111895931B true CN111895931B (en) | 2021-11-26 |
Family
ID=73189475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010694534.4A Active CN111895931B (en) | 2020-07-17 | 2020-07-17 | Coal mine operation area calibration method based on computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111895931B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113740098A (en) * | 2021-08-25 | 2021-12-03 | 安阳屹星智能科技有限公司 | Bulk coal vehicle sampling method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6487303B1 (en) * | 1996-11-06 | 2002-11-26 | Komatsu Ltd. | Object detector |
| JP2016024685A (en) * | 2014-07-22 | 2016-02-08 | 日立建機株式会社 | Work vehicle for mine |
| CN105654067A (en) * | 2016-02-02 | 2016-06-08 | 北京格灵深瞳信息技术有限公司 | Vehicle detection method and device |
| CN107025642A (en) * | 2016-01-27 | 2017-08-08 | 百度在线网络技术（北京）有限公司 | Vehicle's contour detection method and device based on cloud data |
| CN108257139A (en) * | 2018-02-26 | 2018-07-06 | 中国科学院大学 | RGB-D three-dimension object detection methods based on deep learning |
| CN108875902A (en) * | 2017-12-04 | 2018-11-23 | 北京旷视科技有限公司 | Neural network training method and device, vehicle detection estimation method and device, storage medium |
| AU2018102037A4 (en) * | 2018-12-09 | 2019-01-17 | Ge, Jiahao Mr | A method of recognition of vehicle type based on deep learning |
| US10304191B1 (en) * | 2016-10-11 | 2019-05-28 | Zoox, Inc. | Three dimensional bounding box estimation from two dimensional images |
| CN110019914A (en) * | 2018-07-18 | 2019-07-16 | 王斌 | A kind of three-dimensional modeling data storehouse search method for supporting three-dimensional scenic interaction |
| CN110222593A (en) * | 2019-05-18 | 2019-09-10 | 四川弘和通讯有限公司 | A kind of vehicle real-time detection method based on small-scale neural network |
| CN110569792A (en) * | 2019-09-09 | 2019-12-13 | 吉林大学 | Method for detecting front object of automatic driving automobile based on convolutional neural network |
| EP3608840A1 (en) * | 2018-08-07 | 2020-02-12 | Accenture Global Solutions Limited | Image processing for automated object identification |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9487931B2 (en) * | 2014-09-12 | 2016-11-08 | Caterpillar Inc. | Excavation system providing machine cycle training |
| CN105117766A (en) * | 2015-09-24 | 2015-12-02 | 盐城工学院 | Engineering earthwork transport vehicle working amount counting system and method based on ultra-wideband ranging |
| US11441294B2 (en) * | 2016-10-31 | 2022-09-13 | Komatsu Ltd. | Measurement system, work machine, and measurement method |
| WO2018136889A1 (en) * | 2017-01-23 | 2018-07-26 | Built Robotics Inc. | Excavating earth from a dig site using an excavation vehicle |
| US10438371B2 (en) * | 2017-09-22 | 2019-10-08 | Zoox, Inc. | Three-dimensional bounding box from two-dimensional image and point cloud data |
| US10684137B2 (en) * | 2017-11-29 | 2020-06-16 | Deere & Company | Work site monitoring system and method |
| WO2020010517A1 (en) * | 2018-07-10 | 2020-01-16 | 深圳大学 | Trajectory prediction method and apparatus |
| JP7166108B2 (en) * | 2018-08-31 | 2022-11-07 | 株式会社小松製作所 | Image processing system, display device, image processing method, trained model generation method, and training data set |
| US20200117201A1 (en) * | 2018-10-15 | 2020-04-16 | Caterpillar Paving Products Inc. | Methods for defining work area of autonomous construction vehicle |
| CN110163836B (en) * | 2018-11-14 | 2021-04-06 | 宁波大学 | Excavator detection method used under high-altitude inspection based on deep learning |
| US11677930B2 (en) * | 2018-12-20 | 2023-06-13 | Here Global B.V. | Method, apparatus, and system for aligning a vehicle-mounted device |
-
2020
- 2020-07-17 CN CN202010694534.4A patent/CN111895931B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6487303B1 (en) * | 1996-11-06 | 2002-11-26 | Komatsu Ltd. | Object detector |
| JP2016024685A (en) * | 2014-07-22 | 2016-02-08 | 日立建機株式会社 | Work vehicle for mine |
| CN107025642A (en) * | 2016-01-27 | 2017-08-08 | 百度在线网络技术（北京）有限公司 | Vehicle's contour detection method and device based on cloud data |
| CN105654067A (en) * | 2016-02-02 | 2016-06-08 | 北京格灵深瞳信息技术有限公司 | Vehicle detection method and device |
| US10304191B1 (en) * | 2016-10-11 | 2019-05-28 | Zoox, Inc. | Three dimensional bounding box estimation from two dimensional images |
| CN108875902A (en) * | 2017-12-04 | 2018-11-23 | 北京旷视科技有限公司 | Neural network training method and device, vehicle detection estimation method and device, storage medium |
| CN108257139A (en) * | 2018-02-26 | 2018-07-06 | 中国科学院大学 | RGB-D three-dimension object detection methods based on deep learning |
| CN110019914A (en) * | 2018-07-18 | 2019-07-16 | 王斌 | A kind of three-dimensional modeling data storehouse search method for supporting three-dimensional scenic interaction |
| EP3608840A1 (en) * | 2018-08-07 | 2020-02-12 | Accenture Global Solutions Limited | Image processing for automated object identification |
| AU2018102037A4 (en) * | 2018-12-09 | 2019-01-17 | Ge, Jiahao Mr | A method of recognition of vehicle type based on deep learning |
| CN110222593A (en) * | 2019-05-18 | 2019-09-10 | 四川弘和通讯有限公司 | A kind of vehicle real-time detection method based on small-scale neural network |
| CN110569792A (en) * | 2019-09-09 | 2019-12-13 | 吉林大学 | Method for detecting front object of automatic driving automobile based on convolutional neural network |
Non-Patent Citations (1)
| Title |
|---|
| 基于机器视觉的调车机车辅助驾驶的研究;张驰;《中国博士学位论文全文数据库（电子期刊）》;20200131;第101-123 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111895931A (en) | 2020-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109934121B (en) | Orchard pedestrian detection method based on YOLOv3 algorithm | |
| CN109685066B (en) | Mine target detection and identification method based on deep convolutional neural network | |
| CN112793564B (en) | Autonomous parking auxiliary system based on panoramic aerial view and deep learning | |
| US10878288B2 (en) | Database construction system for machine-learning | |
| US10290219B2 (en) | Machine vision-based method and system for aircraft docking guidance and aircraft type identification | |
| CN111191485B (en) | Parking space detection method and system and automobile | |
| CN111462200A (en) | Cross-video pedestrian positioning and tracking method, system and equipment | |
| CN109791052A (en) | For generate and using locating reference datum method and system | |
| CN105069799A (en) | Angular point positioning method and apparatus | |
| CN110825101A (en) | Unmanned aerial vehicle autonomous landing method based on deep convolutional neural network | |
| CN113203409B (en) | Method for constructing navigation map of mobile robot in complex indoor environment | |
| CN109828267A (en) | The Intelligent Mobile Robot detection of obstacles and distance measuring method of Case-based Reasoning segmentation and depth camera | |
| CN110136186B (en) | Detection target matching method for mobile robot target ranging | |
| CN113033315A (en) | Rare earth mining high-resolution image identification and positioning method | |
| CN111895931B (en) | Coal mine operation area calibration method based on computer vision | |
| CN112580542A (en) | Steel bar counting method based on target detection | |
| AU2023202153A1 (en) | Apparatus for analyzing a payload being transported in a load carrying container of a vehicle | |
| CN114089330A (en) | Indoor mobile robot glass detection and map updating method based on depth image restoration | |
| CN113359692A (en) | Obstacle avoidance method and movable robot | |
| CN112800938B (en) | Method and device for detecting occurrence of side rockfall of unmanned vehicle | |
| CN112817006B (en) | Vehicle-mounted intelligent road disease detection method and system | |
| CN114445494A (en) | Image acquisition and processing method, image acquisition device and robot | |
| CN114022563A (en) | Dynamic obstacle detection method for automatic driving | |
| CN113963233A (en) | Target detection method and system based on double-stage convolutional neural network | |
| CN112598738A (en) | Figure positioning method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |