CN116935326A - Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium - Google Patents
Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116935326A CN116935326A CN202311024986.1A CN202311024986A CN116935326A CN 116935326 A CN116935326 A CN 116935326A CN 202311024986 A CN202311024986 A CN 202311024986A CN 116935326 A CN116935326 A CN 116935326A
- Authority
- CN
- China
- Prior art keywords
- image
- image information
- coal mine
- image acquisition
- coal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000003245 coal Substances 0.000 title claims abstract description 109
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012544 monitoring process Methods 0.000 title claims abstract description 42
- 238000003860 storage Methods 0.000 title claims abstract description 11
- 238000005065 mining Methods 0.000 claims abstract description 56
- 238000012549 training Methods 0.000 claims description 33
- 238000012545 processing Methods 0.000 claims description 19
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 7
- 238000012800 visualization Methods 0.000 claims description 6
- 238000009434 installation Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000008570 general process Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a coal mine monitoring method, a system, electronic equipment and a storage medium based on image recognition, wherein the coal mine monitoring method based on image recognition comprises the following steps: calibrating the image acquisition equipment; acquiring image information through the image acquisition equipment; detecting the image information and obtaining an identification result of target characteristics, wherein the target characteristics comprise any one or more of a coal mining machine, underground personnel, a side protection plate and a safety helmet; displaying the identification result; the condition that the comprehensive mining face cannot be observed in an omnibearing manner in the comprehensive mining face monitoring of the coal mine is solved, so that an underground coal cutter, a side protection plate, underground personnel and a safety helmet are effectively monitored, and early warning can be timely carried out when potential safety hazards occur.
Description
Technical Field
The application relates to the technical field of coal mining, in particular to a coal mine monitoring method, a system, electronic equipment and a storage medium based on image recognition.
Background
When the underground fully-mechanized coal mining face of the existing coal mine works, real-time video monitoring is needed to be carried out on the fully-mechanized coal mining face, and the safety condition of the fully-mechanized coal mining face is obtained. Most of the existing fully-mechanized mining face video monitoring technologies aim at dynamic scene monitoring collected by fixed cameras, and in order to monitor a large-range long-distance scene, a plurality of cameras are usually installed for data collection and analysis.
However, the existing underground fully-mechanized mining face monitoring system for the coal mine is single in presentation mode, and only is capable of directly displaying camera data, such as traditional nine-grid image data display, and is incapable of comprehensively observing the working state and fully-mechanized mining face condition of the coal mining machine due to the lack of continuity of videos acquired from different angles by a plurality of different cameras. The existing video monitoring system does not have the recognition and alarm of special information, such as the rolling state of the coal mining machine, the state of a side protection plate, personnel safety, safety helmets and the like, so that early warning cannot be timely performed when accidents happen underground.
Disclosure of Invention
The application aims to provide a coal mine monitoring method, a system, electronic equipment and a storage medium based on image recognition, which solve the problem that the comprehensive mining face cannot be observed in an omnibearing manner in the monitoring of the comprehensive mining face of a coal mine, so as to effectively monitor the state of a coal mining machine, a side protection plate, underground personnel and a safety helmet under the coal mine, and further timely early warning can be realized when potential safety hazards occur.
A first aspect of an embodiment of the present application provides a coal mine monitoring method based on image recognition, including:
calibrating the image acquisition equipment;
acquiring image information through the image acquisition equipment;
detecting the image information and obtaining an identification result of target characteristics, wherein the target characteristics comprise any one or more of a coal mining machine, underground personnel, a side protection plate and a safety helmet;
and displaying the identification result.
In one embodiment, the plurality of image capturing devices, the calibrating the image capturing device includes:
obtaining distortion parameters of the image acquisition equipment by adopting a Zhang calibration method;
and adopting a K center point clustering algorithm to fit and obtain target parameters as calibration parameters of a plurality of image acquisition devices.
In one embodiment, the acquiring, by the image acquisition device, image information includes:
analyzing the data packet acquired by the image acquisition equipment through a first callback function;
when the data packet of one frame of image completely arrives, decoding the data packet through a second callback function to obtain YUV original data;
and transcoding the YUV original data to obtain the image information, wherein the image information is JPG format information adopting RGB color space.
In one embodiment, the detecting the image information and obtaining the identification result of the target feature includes:
polling the image information;
identifying the coal mining machine through a training model, and obtaining position information of the coal mining machine;
the image information is polled again;
and identifying the coal mining machine through a training model so as to obtain the position information of the coal mining machine in real time.
In one embodiment, the detecting the image information and obtaining the identification result of the target feature further includes:
and identifying the image information through a detection model, and continuously identifying and detecting the underground personnel, the side protection plate and the safety helmet to obtain the identification result.
In one embodiment, the method further comprises:
judging whether the underground personnel, the side protection plate and the safety helmet are separated from a preset safety state according to the identification result;
if yes, outputting alarm information.
In one embodiment, the training model is obtained by training using a YOLO-V5 model frame, and the step of obtaining the training model includes:
acquiring original image information through the image acquisition equipment;
carrying out data annotation on the original image information to obtain a sample file, wherein the data annotation category comprises any one or more of the coal mining machine, the underground personnel, the side protection plate and the safety helmet;
and training by adopting the YOLO-V5 model framework based on the sample file to obtain the training model.
A second aspect of an embodiment of the present application provides a coal mine monitoring system based on image recognition, including: a plurality of image acquisition devices, a communication device, a central processing server and a visualization device;
the image acquisition devices are used for acquiring image information of target characteristics, wherein the target characteristics comprise any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet;
the image information is transmitted to the central processing server through the communication device;
the central processing server is used for identifying the image information to obtain an identification result of the target feature;
the visualization equipment is used for displaying the identification result;
the central processing server is also used for judging whether the target feature is separated from a preset installation state according to the identification result and outputting alarm information.
A third aspect of an embodiment of the present application provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the image recognition based coal mine monitoring method described above.
A fourth aspect of an embodiment of the present application provides a computer readable storage medium storing computer instructions for causing a processor to perform the above image recognition based coal mine monitoring method.
Compared with the prior art, the embodiment of the application has the beneficial effects that: the image acquisition equipment is calibrated, the image information is acquired through the image acquisition equipment, the image information is monitored, the identification result of the characteristic information of the coal mining machine, underground personnel, the side protection plate and the safety helmet is acquired, the identification result is displayed, the working state of the coal mining machine is observed in an omnibearing manner, the coal mine monitoring efficiency is improved, the potential safety risk can be found in time, and the occurrence probability of coal mine safety production accidents is effectively reduced.
Drawings
FIG. 1 is a flow chart of a coal mine monitoring method based on image recognition according to an embodiment of the present application;
FIG. 2 is a flowchart of calibrating an image capturing device according to an embodiment of the present application;
FIG. 3 is a flowchart of acquiring image information by an image acquisition device according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a data processing flow of image information of an image capturing device according to an embodiment of the present application;
FIG. 5 is a flowchart of detecting the image information and obtaining the recognition result of the target feature according to an embodiment of the present application;
FIG. 6 is a flowchart of detecting the image information and obtaining the recognition result of the target feature according to another embodiment of the present application;
FIG. 7 is a flowchart of a method for obtaining a training model according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a training flow of a training model according to an embodiment of the present application;
fig. 9 is a schematic diagram showing an effect of displaying a recognition result according to an embodiment of the present application;
FIG. 10 is a schematic diagram of a coal mine monitoring system based on image recognition according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the technical problems, technical schemes and beneficial effects to be solved more clear, the application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
It will be understood that when an element is referred to as being "mounted" or "disposed" on another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element.
It is to be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are merely for convenience in describing and simplifying the description based on the orientation or positional relationship shown in the drawings, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus are not to be construed as limiting the application.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present application, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Referring to fig. 1, fig. 1 shows a flowchart of a coal mine monitoring method based on image recognition according to the present application, and as shown in fig. 1, the coal mine monitoring method based on image recognition includes:
s110, calibrating the image acquisition equipment.
S120, acquiring image information through an image acquisition device.
In order to obtain the situation of the fully-mechanized coal mining face, the image acquisition devices are deployed on the coal cutter bracket, alternatively, one image acquisition device is deployed every three or two brackets, so that the view field of the image acquisition devices can cover the whole coal mine roadway space. In the process of monitoring a coal mine through a plurality of image acquisition devices, in order to determine the corresponding relation between three-dimensional positions of a space object and two-dimensional images of the space object, an imaging geometric model of the image acquisition devices needs to be established, namely the image acquisition devices are calibrated. By calibrating the image acquisition equipment, the acquired image information can be enabled to be more in line with the real shape of the three-dimensional world.
The image information is acquired through the image acquisition device, specifically, the camera performs data transmission through a TCP/IP protocol (Transmission Control Protocol/Internet Protocol, transmission control/interconnection protocol), the TCP is a connection-oriented network transmission protocol, multiple data flow operations are supported, flow control and error control are provided, and even a reordering function of out-of-order arrival messages is provided, so that the reliability of image information data transmission can be improved by adopting the TCP in the embodiment. Further, the general process of TCP transmission includes that after a client sends a connection request to a server, the server sends a connection confirmation to the client after receiving the connection request, and after the connection is achieved, both sides perform data transmission. In addition, the present embodiment may employ an IP multicast technology, which is a network technology that allows one or more senders to send data packets of a single or multiple senders to multiple receivers, so that a network bandwidth may be greatly saved.
S130, detecting the image information and obtaining a recognition result of target characteristics, wherein the target characteristics comprise any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet.
In the embodiment of the application, after the image information is acquired, the image information is detected, and the identification result of the target characteristic is acquired, wherein the target characteristic comprises any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet. The image information of the coal mining machine is, for example, position information of the coal mining machine, rolling information of the coal mining machine, and the like, and during the running process of the coal mining machine, abnormal conditions such as the situation that the upper roller collides with the side protection plate, the upper roller collides with the front cantilever, the lower roller does not prop in place, and the scraper conveyor is difficult to push, and the like, may cause the collision accident of the coal mining machine. The characteristic information of the safety helmet is, for example, the wearing information of the safety helmet of the underground personnel. The information is very key target characteristic information, and potential safety risks can be found in time through detection and identification of the characteristic information, so that the occurrence probability of coal mine safety production accidents is reduced.
And S140, displaying the identification result.
In the embodiment of the application, the recognition results of the characteristic information of the coal cutter, underground personnel, a side protection plate, a safety helmet and the like are displayed, so that the personnel can monitor and observe the underground condition of the coal mine in an omnibearing manner conveniently, for example, the recognition results are displayed through a display screen.
According to the coal mine monitoring method based on image recognition, provided by the embodiment of the application, the image acquisition equipment is calibrated, the image information is acquired through the image acquisition equipment, the image information is monitored, the recognition results of the characteristic information of the coal mining machine, underground personnel, the side protection plate and the safety helmet are acquired, the recognition results are displayed, the working state of the coal mining machine is comprehensively observed, the coal mine monitoring efficiency is improved, the potential safety risk can be found in time, and the occurrence probability of coal mine safety production accidents is effectively reduced.
Referring to fig. 2, in one embodiment, a plurality of image capturing devices are used to monitor underground conditions of a coal mine, where the step of calibrating the image capturing devices specifically includes:
s111, acquiring distortion parameters of the image acquisition equipment by adopting a Zhang calibration method.
S121, fitting and obtaining target parameters by adopting a K center point clustering algorithm, wherein the target parameters are used as calibration parameters of a plurality of image acquisition devices.
In the embodiment of the application, a plurality of image acquisition devices are used, the image acquisition devices are calibrated by adopting a Zhang calibration method (Zhang Zhengyou calibration method) to acquire distortion parameters of the image acquisition devices, and in order to enable data of the image acquisition devices to be more accurate, the image acquisition devices are calibrated, and the Zhang Zhengyou method is used for calibrating original image data.
Because there are multiple image acquisition devices, in order to obtain more accurate and more real image information, calibration needs to be performed for each image acquisition device, in some embodiments, 75 image acquisition devices are provided for coal mine monitoring, and data fitting can be performed through a linear data fitting method to obtain a comprehensive calibration parameter as the calibration parameter of all the image acquisition devices. For example, a clustering center algorithm is used for data fitting, the clustering center algorithm is a type of algorithm commonly used for clustering analysis, the main objective of the clustering center algorithm is to divide a data set into a plurality of clusters, the similarity of data points in each cluster is higher, the similarity between different clusters is lower, and the core principle of the clustering center algorithm is to realize data clustering by calculating the distance or the similarity between the data points.
Specifically, the embodiment adopts a K-center clustering algorithm, wherein the K-center clustering algorithm uses a certain data point in a cluster as a clustering center, firstly randomly selects K data points as initial center points, then distributes each data point to the center point closest to the data point to form K clusters, then calculates the average distance from all data points in each cluster to other data points, selects the smallest distance as a new center point, and repeats the steps until the center point is not changed any more or reaches a preset iteration number, and finally obtains K center points and corresponding clusters, namely, the clustering process of the data is completed. The position of the center point is continuously optimized through a K center clustering algorithm, so that the minimum value of the average distance in the cluster is fitted into an ideal calibration parameter to serve as the calibration parameter of all image acquisition equipment, the calibration of all image acquisition equipment is simplified, the calibration efficiency is improved, and more real image data can be obtained.
Referring to fig. 3, in one embodiment, the step of acquiring image information by the image acquisition device includes:
s121, analyzing the data packet acquired by the image acquisition equipment through the first callback function.
S122, when the data packet of one frame of image completely arrives, decoding the data packet through a second callback function to obtain YUV original data.
S123, transcoding the YUV original data to obtain image information, wherein the image information is JPG format information adopting RGB color space.
In this embodiment, referring to fig. 4, fig. 4 shows a data processing flow of image information of an image capturing device in the embodiment of the present application, first, real-time decompression processing is performed on video streams transmitted by a plurality of image capturing devices, so as to obtain original JPG image data.
Specifically, the data packet of the image capturing device is parsed by the first callback function, where the first callback function may be set at the net_dvr_setdata callback of the SDK (Software Development Kit ) console program of the image capturing device to perform the data packet parsing, and then, when the data packet of a frame of image arrives completely, the second callback function is called to perform decoding, where the playm4_setdeccallback of the SDK console program of the image capturing device may be used to perform decoding, where the data conversion is completed in the decoded callback function, and the data may be converted into a required format, where YUV original data is converted into JPG format information using RGB color space, i.e., JPG image. The YUV model defines a color space based on one luminance (Y component) and two chromaticities (UV components), in which there is one luminance signal Y for each color, and the two chrominance signals U and VRGB image have three channels R, G, B corresponding to three components of red, green, and blue, respectively, the colors being determined by the values of the three components, where the YUV raw data is restored to be converted into RGB format.
Referring to fig. 5, in one embodiment, the step of detecting image information and obtaining a recognition result of a target feature includes:
s131, the image information is polled.
S132, identifying the coal mining machine through a training model, and obtaining the position information of the coal mining machine.
In this embodiment, after original image data is obtained, image data of a plurality of image acquisition devices are detected and identified through the round image information, first, all the image data are detected and identified, and the position of the coal mining machine is detected in real time.
In some embodiments, with more than 2 high-power graphics cards on the server side, the multi-threaded computation is performed while processing logic is in progress, and the server has multiple GPUs, multiple pictures can be processed simultaneously on multiple cores using parallel computing techniques, such as multi-threading or multi-processing. Each core can independently process the detection task of one picture, so that the overall processing speed is accelerated.
S133, the partial images are polled again.
S134, identifying the coal mining machine through a training model so as to obtain the position information of the coal mining machine in real time.
In this embodiment, the position of the coal mining machine is obtained through the round of image information, and the round of operation is performed by adopting a slowly increasing strategy in the next round of operation, because the moving speed of the coal mining machine is very slow when the coal mining machine works, the calculating speed of each round of operation is about 10 frames, and the moving speed of the coal mining machine is generally 8m/min, and the moving speed is relatively slow. Therefore, after the coal machine is identified last time, the identification position of the next coal machine does not need to identify and judge the image information of all the image acquisition devices, only part of the image information needs to be polled, namely only the image information near the last result needs to be identified and judged. Because the coal mining machine may have left and right results, the image information of the recognition result of the last round of inspection needs to be added to the two sides symmetrically at the moment, so that the position of the current coal mining machine is determined. In this embodiment, by adopting the above method when the image information is polled again, the calculation amount of data can be reduced, and the recognition speed can be improved.
Referring to fig. 5, in one embodiment, the step of detecting the image information and obtaining the identification result of the target feature further includes:
s135, recognizing image information through the detection model, and continuously recognizing and detecting underground personnel, the side protection plate and the safety helmet to obtain a recognition result. According to the embodiment of the application, the position of the coal mining machine can be accurately positioned through the identification result of the image information, and the data results of other targets, namely underground personnel, the side protection plate and the safety helmet characteristic information, are required to be detected and identified again, so that the corresponding identification result is obtained.
Referring to fig. 6, in one embodiment, the method further includes the steps of:
s136, judging whether underground personnel, the side protection plate and the safety helmet are separated from a preset safety state according to the identification result.
And S137, if yes, outputting alarm information.
In the embodiment of the application, the data results of the underground personnel, the side protection plate and the safety helmet characteristic information are judged according to the identification results, for example, the behavior that whether the underground personnel deviate from a preset safety state is judged and predicted by detecting the identification results, for example, the falling behavior of the underground personnel is judged, at the moment, the underground personnel deviate from the preset safety state, so that the possible danger is predicted, and the alarm information is output. It will be appreciated that the side protection panel is out of a predetermined safety condition, such as a collision with the side protection panel, an out of position side protection panel, etc., and the helmet feature is out of a predetermined safety condition, such as an operation where the helmet is not worn by a person downhole, etc. By the method, when the potential safety hazards occur, the alarm can be given out in time, and the occurrence probability of coal mine production accidents is reduced.
Referring to fig. 7, in one embodiment, the training model in this embodiment is obtained by training using a YOLO-V5 model frame, and the step of obtaining the training model includes:
s1304, original image information is acquired by the image acquisition device.
S1302, carrying out data annotation on the original image information to obtain a sample file, wherein the data annotation category comprises any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet.
And S1303, training by using a YOLO-V5 model frame based on the sample file to obtain a training model.
In the embodiment of the application, the original image information is firstly obtained through the image acquisition equipment, then the original image information is subjected to data annotation to obtain a sample file, the category of the data annotation comprises any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet, and finally the model training is carried out by adopting a YOLO-V5 model frame based on the sample file to obtain a training model.
Specifically, referring to fig. 8 in combination, fig. 8 shows a training flow of the training model in this embodiment, where the YOLO-V5 model framework includes an input end, a skeleton feature extraction network, a neck network, and a detection head network that are sequentially connected, where the YOLO-V5 model framework changes the first layer of the network into a convolution layer with a size of 6x6, which is more efficient than some GPU devices (and corresponding optimization algorithms) that exist, and the YOLO-V5 model framework uses a convolution layer with a size of 6x 6.
In addition, the loss of the YOLO-V5 model framework consists mainly of three parts: classification loss (class loss), BCE loss, obj loss (Objectness loss) and BCE loss are used, all samples Obj loss and positioning loss (positioning loss) are used, CIoU loss is used, and only positive sample positioning loss is calculated, and loss=λ1, λ2 and λ3, loss=λ1×lcls+λ2×lobj+λ3 lc, corresponding to the balance coefficients of the three losses. Additionally, tags of various data labeling categories may be converted into a form suitable for the YOLO-V5 model framework, such as by single-hot encoding (one-hot encoding) to represent category tags for multi-category classification during recognition. The training model obtained by using the YOLO-V5 model frame realizes rapid identification of target characteristics, and the real-time performance of an algorithm is effectively improved.
The training model after training is output to generate a training model file, then model data can be randomly called for identification and detection when the training model file is called, image information input into the training model is respectively detected, identification results are output, and finally the identification results of a coal mining machine, underground personnel, a side protection plate and a safety helmet detected in real time can be output and displayed through a display screen, and the display screen is divided into three modules, wherein the three modules comprise a part of information output, coal mining machine identification, a part of personnel, a safety helmet and a side protection plate identification result. It can be understood that the data of the coal mining machine is obtained from the round inspection server, and the position of the coal mining machine is judged after a bounding box (boundbox) is identified and marked, so that the result of the identification of the coal mining machine is obtained. Elements such as underground personnel, a side protection plate, a safety helmet and the like are identified to obtain perception information, the perception information is processed and filtered at the rear end, and a part of content to be displayed is pushed to a platform for real-time display.
According to the coal mine monitoring method based on image recognition, provided by the embodiment of the application, the image acquisition equipment is calibrated, the image information is acquired through the image acquisition equipment, the image information is monitored, the recognition results of the characteristic information of the coal mining machine, underground personnel, the side protection plate and the safety helmet are acquired, the recognition results are displayed, the working state of the coal mining machine is comprehensively observed, the coal mine monitoring efficiency is improved, the potential safety risk can be found in time, and the occurrence probability of coal mine safety production accidents is effectively reduced.
Referring to fig. 10, a second aspect of the present application provides a coal mine monitoring system 100 based on image recognition, where the coal mine monitoring system 100 includes a plurality of image acquisition devices 110, a communication device 120, a central processing server 130, and a visualization device 140. The plurality of image capturing devices 110 are configured to capture image information of a target feature, where the target feature includes any one or more of a coal cutter, a downhole personnel, a side guard, and a helmet, and the image information is transmitted to the central processing server 130 through the communication device 120, where the communication device 120 may use wired communication or Wireless communication, including but not limited to ethernet, optical fiber, RS485, or other communication means, or the Wireless connection may include but not limited to a 3G/4G connection, a WiFi (Wireless-Fidelity) connection, a bluetooth connection, a wima× (Worldwide Interoperability for Microwave Access) connection, a Zigbee (low power local area network protocol, also known as a peak-to-peak protocol) connection, a UWB (ultra wideband) connection, and other now known or later developed Wireless connection means.
The central processing server 130 is used for identifying the image information to obtain the identification result of the target feature, the visualization device 140 is used for displaying the identification result, and further, the central processing server 130 is further used for judging whether the target feature deviates from the preset installation state according to the identification result and outputting alarm information.
Referring to fig. 11, a second aspect of the embodiment of the present application provides an electronic device 20, where the electronic device 20 includes at least one processor 21, and a memory 22 communicatively connected to the at least one processor 21, and the memory 22 stores a computer program executable by the at least one processor 21, and the computer program is executed by the at least one processor 21 to enable the at least one processor 21 to perform the coal mine monitoring method based on image recognition provided in the first aspect of the embodiment of the present application.
It should be appreciated that the processor described above may be a CPU, but may also be other general purpose processors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integratedcircuit, ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or any conventional processor or the like. It is noted that the processor may be a processor supporting an advanced reduced instruction set machine (advanced RISC machines, ARM) architecture.
Further, in an alternative embodiment, the memory may include read only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store information of the device type.
The memory may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random access memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic RAM (dynamic random access memory, DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous DRAM (SLDRAM), and direct memory bus RAM (DR RAM).
The fourth aspect of the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores computer instructions, wherein the computer instructions are used for enabling a processor to implement the coal mine monitoring method based on image recognition provided by the first aspect of the embodiment of the application when being executed.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. . Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. The coal mine monitoring method based on image recognition is characterized by comprising the following steps of:
calibrating the image acquisition equipment;
acquiring image information through the image acquisition equipment;
detecting the image information and obtaining an identification result of target characteristics, wherein the target characteristics comprise any one or more of a coal mining machine, underground personnel, a side protection plate and a safety helmet;
and displaying the identification result.
2. A method of coal mine monitoring as claimed in claim 1 wherein the plurality of image acquisition devices, the calibrating of the image acquisition devices, comprises:
obtaining distortion parameters of the image acquisition equipment by adopting a Zhang calibration method;
and adopting a K center point clustering algorithm to fit and obtain target parameters as calibration parameters of a plurality of image acquisition devices.
3. The coal mine monitoring method of claim 1, wherein acquiring image information by the image acquisition device comprises:
analyzing the data packet acquired by the image acquisition equipment through a first callback function;
when the data packet of one frame of image completely arrives, decoding the data packet through a second callback function to obtain YUV original data;
and transcoding the YUV original data to obtain the image information, wherein the image information is JPG format information adopting RGB color space.
4. The coal mine monitoring method of claim 1, wherein the detecting the image information and obtaining the recognition result of the target feature includes:
polling the image information;
identifying the coal mining machine through a training model, and obtaining position information of the coal mining machine;
the image information is polled again;
and identifying the coal mining machine through a training model so as to obtain the position information of the coal mining machine in real time.
5. The coal mine monitoring method of claim 1, wherein the detecting the image information and acquiring the recognition result of the target feature further comprises:
and identifying the image information through a detection model, and continuously identifying and detecting the underground personnel, the side protection plate and the safety helmet to obtain the identification result.
6. The coal mine monitoring method of claim 5, further comprising:
judging whether the underground personnel, the side protection plate and the safety helmet are separated from a preset safety state according to the identification result;
if yes, outputting alarm information.
7. A method of coal mine monitoring as claimed in any one of claims 4 to 6 wherein the training model is trained using a YOLO-V5 model framework, the step of obtaining the training model comprising:
acquiring original image information through the image acquisition equipment;
carrying out data annotation on the original image information to obtain a sample file, wherein the data annotation category comprises any one or more of the coal mining machine, the underground personnel, the side protection plate and the safety helmet;
and training by adopting the YOLO-V5 model framework based on the sample file to obtain the training model.
8. A coal mine monitoring system based on image recognition, comprising: a plurality of image acquisition devices, a communication device, a central processing server and a visualization device;
the image acquisition devices are used for acquiring image information of target characteristics, wherein the target characteristics comprise any one or more of a coal cutter, underground personnel, a side protection plate and a safety helmet;
the image information is transmitted to the central processing server through the communication device;
the central processing server is used for identifying the image information to obtain an identification result of the target feature;
the visualization equipment is used for displaying the identification result;
the central processing server is also used for judging whether the target feature is separated from a preset installation state according to the identification result and outputting alarm information.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the image recognition based coal mine monitoring method of any one of claims 1 to 7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the image recognition based coal mine monitoring method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311024986.1A CN116935326A (en) | 2023-08-15 | 2023-08-15 | Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311024986.1A CN116935326A (en) | 2023-08-15 | 2023-08-15 | Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116935326A true CN116935326A (en) | 2023-10-24 |
Family
ID=88384424
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311024986.1A Pending CN116935326A (en) | 2023-08-15 | 2023-08-15 | Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116935326A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117456708A (en) * | 2023-12-22 | 2024-01-26 | 山东省邱集煤矿有限公司 | Coal mine underground early warning method, system and equipment based on image key information identification |
-
2023
- 2023-08-15 CN CN202311024986.1A patent/CN116935326A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117456708A (en) * | 2023-12-22 | 2024-01-26 | 山东省邱集煤矿有限公司 | Coal mine underground early warning method, system and equipment based on image key information identification |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108476304B (en) | Discarded object monitoring device, discarded object monitoring system provided with same, and discarded object monitoring method | |
| KR102045871B1 (en) | System For Detecting Fire Based on Artificial Intelligence And Method For Detecting Fire Based on Artificial Intelligence | |
| CN106610987B (en) | Video image retrieval method, device and system | |
| US10057591B2 (en) | Method and device for processing a video stream in a video camera | |
| EP2903261B1 (en) | Apparatus and method for detecting event from plurality of photographed images | |
| CN109167997B (en) | Video quality diagnosis system and method | |
| CN116935326A (en) | Coal mine monitoring method and system based on image recognition, electronic equipment and storage medium | |
| KR101953342B1 (en) | Multi-sensor fire detection method and system | |
| KR20150035322A (en) | Image monitoring system | |
| KR20200058260A (en) | Apparatus for CCTV Video Analytics Based on Object-Image Recognition DCNN and Driving Method Thereof | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| JP2011087253A (en) | Imaging apparatus, information processing device and information processing method | |
| CN113660427A (en) | Image analysis system and method applied to video monitoring tester | |
| JP3486229B2 (en) | Image change detection device | |
| CN102025975A (en) | Automatic monitoring method and system | |
| KR102127276B1 (en) | The System and Method for Panoramic Video Surveillance with Multiple High-Resolution Video Cameras | |
| CN117612060A (en) | Video early warning system, method, equipment and medium based on artificial intelligent detection | |
| KR101712447B1 (en) | How to set up a video camera module with optical signal analysis funtion using the image sensor | |
| CN107609498B (en) | Data processing method of computer monitoring system | |
| CN203968273U (en) | A kind of video failure monitoring system for highway | |
| CN107959769B (en) | Video camera | |
| JP2017200098A (en) | Movement monitoring device | |
| CN109194981B (en) | Remote video tracking method, device and storage medium | |
| CN110944159A (en) | Information processing method, electronic equipment and information processing system | |
| JP2016099821A (en) | Monitoring device, monitoring system and monitoring method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |