CN116645645A

CN116645645A - Coal Mine Transportation Safety Determination Method and Coal Mine Transportation Safety Determination System

Info

Publication number: CN116645645A
Application number: CN202310594835.3A
Authority: CN
Inventors: 陈湘源; 张增誉; 饶天荣; 杨进; 惠树军; 李星; 于洋; 潘涛; 祝与淘
Original assignee: Guoneng Information Technology Co ltd; Guoneng Yulin Energy Co ltd
Current assignee: Guoneng Information Technology Co ltd; Guoneng Yulin Energy Co ltd
Priority date: 2023-05-19
Filing date: 2023-05-19
Publication date: 2023-08-25

Abstract

The application provides a coal mine transportation safety determination method and a coal mine transportation safety determination system. The method comprises the following steps: acquiring video information, wherein the video information is information of a video of a target area acquired by video acquisition equipment; acquiring positioning information, wherein the positioning information comprises information of the position of a vehicle and information of the position of a pedestrian acquired by a UWB positioning card; and respectively constructing a first model and a second model according to the video information and the positioning information, and calculating a weighted average value of the output result of the first model and the output result of the second model to obtain a target result. The safety of the driving process is determined in an automatic mode, and is determined in a multi-source data fusion mode, so that inaccuracy of single data judgment can be avoided, and the reliability of safety detection in the scheme is higher.

Description

Coal mine transportation safety determination method and coal mine transportation safety determination system

Technical Field

The application relates to the technical field of coal mine safety detection, in particular to a coal mine transportation safety determination method, a coal mine transportation safety determination device, a computer readable storage medium and a coal mine transportation safety determination system.

Background

Coal is the main energy source of China, and accounts for more than 60% of primary energy consumption, so that the coal mine is safe, healthy and efficient in production, and has very important significance for guaranteeing the energy safety of China and promoting the stable development of national economy. In recent years, with the wide application of advanced information technology in coal mines and the importance of nations on the construction of intelligent mines of the coal mines, the level of the intelligent mines of China is greatly improved. Along with continuous floor practice and upgrading development of mine informatization and automation technologies, more and more applications such as accurate positioning, wireless communication and vehicle-mounted video lay an information foundation for more comprehensively, accurately and real-timely sensing people, vehicles, roads and materials in an auxiliary transportation system of an underground coal mine. In order to avoid safety accidents such as collision and scratch of personnel and mine cars, the coal mine industry establishes a traffic-free and pedestrian-free transportation management system. However, the actual coal mine production still mainly depends on subjective activity of driving of drivers and passive manual monitoring and identification of safety personnel, and the subjective activity and the passive manual monitoring and identification of the safety personnel can lead to uncontrollable safety risks to a certain extent. Thus, in the current scheme, the reliability of safety monitoring for the auxiliary transportation system of the underground coal mine is low.

Disclosure of Invention

The application mainly aims to provide a coal mine transportation safety determination method, a coal mine transportation safety determination device, a computer-readable storage medium and a coal mine transportation safety determination system, so as to at least solve the problem of low reliability of safety monitoring of an underground coal mine auxiliary transportation system in the prior art.

In order to achieve the above object, according to one aspect of the present application, there is provided a coal mine transportation safety determination method comprising: acquiring video information, wherein the video information is information of a video of a target area acquired by video acquisition equipment, the target area is positioned in the traveling direction of a vehicle, and the minimum distance between the target area and the vehicle is kept unchanged; acquiring positioning information, wherein the positioning information comprises information of the position of the vehicle and information of the position of a pedestrian acquired by a UWB positioning card; respectively constructing a first model and a second model according to the video information and the positioning information, and calculating a weighted average value of an output result of the first model and an output result of the second model to obtain a target result, wherein the first model is used for preliminarily determining whether coal mine transportation is safe or not according to the video information, the second model is used for preliminarily determining whether coal mine transportation is safe or not according to the positioning information, and the target result is used for determining whether coal mine transportation is safe or not.

Optionally, before calculating a weighted average of the output result of the first model and the output result of the second model, the method further includes: constructing an initial first model, wherein the initial first model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical image information and historical area occupied by historical pedestrians in the historical image information, wherein the historical image information is any frame in historical video information; inputting the image information into the initial first model to obtain an initial area corresponding to the image information; constructing the second model, wherein the second model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical positioning information and a historical distance between a historical vehicle and a historical pedestrian corresponding to the historical positioning information; and inputting the positioning information into the second model to obtain the distance corresponding to the positioning information.

Optionally, the area range of the history area corresponds to the distance range of the history distance one by one, and before calculating a weighted average of the output result of the first model and the output result of the second model, the method further includes: acquiring a target area range of the initial area corresponding to the distance range of the distance under the condition that the area range of the initial area does not correspond to the distance range of the distance; and inputting the target area range into the initial first model, and performing model training on the initial first model again until the area range of the initial area corresponds to the distance range of the distance to obtain the first model.

Optionally, before constructing the second model, the method further comprises: acquiring a first time stamp of a ranging request frame sent by the UWB positioning card of the vehicle; acquiring a second time stamp of a ranging response frame sent by a UWB base station and received by the UWB positioning card of the vehicle; and calculating the distance between the vehicle and the UWB base station according to at least the first time stamp and the second time stamp.

Optionally, calculating a weighted average of the output result of the first model and the output result of the second model to obtain a target result, including: analyzing the video quality of the video information according to a PSNR avg.MSE algorithm to obtain a first evaluation score; analyzing the signal intensity of the positioning information according to an RSS algorithm to obtain a second evaluation score; determining a first weight coefficient of a first output result of the first model and a second weight coefficient of a second output result of the second model according to the first evaluation score and the second evaluation score; and calculating the sum of a first product and a second product, and obtaining the target result, wherein the first product is the product of the first output result of the first model and the first weight coefficient, and the second product is the product of the second output result of the second model and the second weight coefficient.

Optionally, determining a first weight coefficient of the first output result of the first model and a second weight coefficient of the second output result of the second model according to the first evaluation score and the second evaluation score respectively includes: determining the first weight coefficient according to the first evaluation score, determining the second weight coefficient according to the second evaluation score, wherein the first evaluation score and the first weight coefficient are positively correlated, the second evaluation score and the second weight coefficient are positively correlated, and the magnitude relation between the first weight coefficient and the second weight coefficient is related to a first difference value and a second difference value, wherein the first difference value is the difference value between the first evaluation score and a first score threshold value, and the second difference value is the value between the second evaluation score and a second score threshold value; and determining that the first weight coefficient is larger than the second weight coefficient when the first difference value is larger than the second difference value, determining that the first weight coefficient is smaller than the second weight coefficient when the first difference value is smaller than the second difference value, and determining that the first weight coefficient is equal to the second weight coefficient when the first difference value is equal to the second difference value.

Optionally, after calculating a weighted average of the output result of the first model and the output result of the second model, the method further includes: under the condition that the target result is greater than or equal to a preset threshold value, determining coal mine transportation safety; and under the condition that the target result is smaller than the preset threshold value, determining that coal mine transportation is unsafe, and generating alarm information.

According to another aspect of the present application, there is provided a coal mine transportation safety determination apparatus comprising: a first obtaining unit, configured to obtain video information, where the video information is information of a video of a target area collected by a video collecting device, the target area is located in a traveling direction of a vehicle, and a minimum distance between the target area and the vehicle is kept unchanged; the second acquisition unit is used for acquiring positioning information, wherein the positioning information comprises information of the position of the vehicle and information of the position of a pedestrian acquired by the UWB positioning card; the first determining unit is used for respectively constructing a first model and a second model according to the video information and the positioning information, calculating a weighted average value of an output result of the first model and an output result of the second model, and obtaining a target result, wherein the first model is used for preliminarily determining whether coal mine transportation is safe according to the video information, the second model is used for preliminarily determining whether coal mine transportation is safe according to the positioning information, and the target result is used for determining whether coal mine transportation is safe.

According to still another aspect of the present application, there is provided a computer readable storage medium including a stored program, wherein the program when run controls a device in which the computer readable storage medium is located to perform any one of the methods of determining safety of coal mine transportation.

According to yet another aspect of the present application there is provided a coal mine transportation safety determination system comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a safety determination method for performing any of the coal mine transportation.

By applying the technical scheme of the application, the safety of the driving process is determined in an automatic mode, and the safety of the driving process is determined in a multi-source data fusion mode, so that inaccuracy of single data judgment can be avoided, and the reliability of safety detection in the scheme is higher.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

Fig. 1 is a block diagram showing a hardware configuration of a mobile terminal for performing a coal mine transportation safety determination method according to an embodiment of the present application;

FIG. 2 shows a flow diagram of a coal mine transportation safety determination method provided in accordance with an embodiment of the present application;

FIG. 3 shows a flow diagram of a two-phase data fusion of the present approach;

FIG. 4 shows a training flow diagram of a machine learning based model;

FIG. 5 shows a schematic flow chart of the adaptive fusion algorithm of the present solution;

fig. 6 shows a block diagram of a coal mine transportation safety determination apparatus according to an embodiment of the present application.

Wherein the above figures include the following reference numerals:

102. a processor; 104. a memory; 106. a transmission device; 108. and an input/output device.

Detailed Description

It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For convenience of description, the following will describe some terms or terminology involved in the embodiments of the present application:

active learning: the learner picks untagged samples during the learning process and requests the outside to provide tagged information, with the goal of achieving good learning performance with as few queries as possible.

Semi-supervised learning: the method is a learning method combining supervised learning and unsupervised learning, and semi-supervised learning uses a large amount of unlabeled data and simultaneously uses the labeled data to perform pattern recognition work. When semi-supervised learning is used, fewer people are required to do work, and meanwhile, higher accuracy can be brought.

In the current informatization age taking data as a core, the traditional mode cannot reasonably utilize accumulation of the data to finish improvement of supervision quality, and cannot realize real-time monitoring and advanced early warning.

In recent years, deep Learning (DL) has achieved a lot of research results in various fields such as image recognition, machine translation, natural language processing (Natural Language Processing, NLP), and so on, and thus, more and more safety production scenes employ computer vision technology based on Deep Learning for anomaly detection. Compared with the defects of high efficiency, low efficiency, easiness in environmental influence, non-uniform detection standard and the like of manual detection, and the defects of poor anti-interference capability, low precision, difficulty in optimization and the like of the traditional machine vision, the intelligent inspection algorithm based on deep learning has stronger adaptability, higher precision, reusable model and easiness in iteration. However, the high-precision deep learning model generally needs a large amount of training data support, and in practical application, a large amount of personnel are needed for data labeling. In the underground coal mine environment, the safety monitoring model based on the deep learning image recognition cannot achieve ideal detection accuracy in practical application due to the reasons of complex environment, difficult data acquisition and the like, and the practical application of the method is greatly limited.

Meanwhile, based on the rising of Ultra Wide Band (UWB) positioning technology, the positioning precision is improved to a new height, and the technology has the advantages of good anti-interference performance, strong penetrating capacity, stronger anti-interference performance and the like, is more suitable for the complex positioning environment of a coal mine well, and simultaneously, a plurality of students and application manufacturers continuously develop wireless positioning algorithms in practice to realize the high-precision requirement of positioning. UWB technology has landed in a number of intelligent mine projects, providing good data conditions for personnel and vehicle positioning. However, in actual use, the UWB technology is prone to problems such as positioning drift and positioning accuracy degradation due to the signal being prone to interference, which affects the operation efficiency of the security detection system.

In practical application, the safety monitoring of the pedestrian driving of the auxiliary operation system cannot be effectively realized by using the two methods independently. The intelligent inspection algorithm based on deep learning has a larger difference from the algorithm principle of the safety monitoring method based on the UWB positioning technology, the effect of directly fusing the two methods is poor, and the defects of the two methods cannot be overcome. In summary, in order to better meet the safety management requirements of traffic and non-traffic in combination with the development of new generation information technology, information needs to be acquired from multiple dimensions such as positioning, images, videos and the like, and data in different dimensions are fused better and more fully through a multi-stage intelligent fusion algorithm, so that the robustness of a monitoring algorithm is improved. The ability of learning and fusing the multi-field signals is endowed to the machine so as to realize complementation of various heterogeneous information, and finally, the accuracy of the underground coal mine safety detection model is improved.

In the traditional ground traffic field, pedestrian detection is taken as an important branch of the target detection field, and through years of research of experts and scholars, key breakthrough in theory and application is achieved, so that very remarkable development is achieved. The existing pedestrian detection algorithms can be divided into two types, namely a pedestrian detection algorithm based on traditional artificial characteristics and a pedestrian detection algorithm based on deep learning. Pedestrian detection algorithms fall into two main categories, one based on traditional image processing and the other based on deep learning. With the great improvement of the computing speed of a computer in recent years, a method based on deep learning has higher and higher detection speed and detection precision, and is widely applied to the pedestrian detection field. The double-stage target detection algorithm represented by Fast RCNN and Fast RCNN lays a foundation for the subsequent derived network. In some schemes, a multi-scale self-adaptive joint discrimination network is constructed by improving a classical Fast RCNN detection framework, pedestrian scales are discriminated firstly, corresponding detection sub-networks are constructed according to different scales, and a large-scale pedestrian detection framework and a small-scale pedestrian detection framework are integrated, so that the detection capability of the network on multi-scale pedestrians is optimized.

The training set picture lighting condition adopted by the traditional road pedestrian detection or recently rapidly developed intelligent driving system in the process of model construction is good, and the sample size is extremely large. The underground coal mine production conditions are complex, underground illumination is weak, partial area water vapor is large, effective samples are few, data acquisition and labeling are difficult, and a large-scale data set for training a reliable model cannot be established, so that the application of a video analysis algorithm based on a deep learning algorithm in the underground coal mine environment is greatly limited, and the effect in practical application is not obvious.

In order to overcome the interference of the underground complex environment of the coal mine on the common visual model, the video image can be acquired and identified based on the high-definition infrared camera. However, the infrared camera can only be installed at a fixed position, the coverage area is small, the purchasing cost is high, the accurate positioning information and the vehicle-mounted video which are rapidly developed in recent years and continuously applied to underground coal mines are not effectively utilized, and the positioning algorithm and the deep learning are not used. If the technical scheme of the high-definition infrared camera is adopted to widely apply the requirements of better driving detection, larger user cost and construction period can be generated.

In some schemes, the underground vehicle is accurately positioned through data acquisition and data fusion, but the method needs to label the point cloud data in a large amount, consumes a large amount of labor cost and has higher labeling difficulty. Secondly, the method directly fuses information acquired by various sensors such as a sensor laser radar, a camera, a wheel speed odometer, an ultra-wideband positioning sensor and the like, the properties and the use scenes of different mode data are not further considered, the fusion effect is poor in the actual use process, the effect of compatibility and complementation of the multi-mode data cannot be truly achieved, and the application of the method in the underground production environment is limited.

As described in the background art, the reliability of safety monitoring of the auxiliary transportation system for the underground coal mine in the prior art is low, and in order to solve the problems, the embodiment of the application provides a method and a device for determining the safety of coal mine transportation, a computer readable storage medium and a system for determining the safety of coal mine transportation.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking the operation on the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of the mobile terminal of a coal mine transportation safety determination method according to an embodiment of the present application. As shown in fig. 1, a mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a display method of device information in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, to implement the above-described method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is configured to communicate with the internet wirelessly.

In this embodiment, a method of determining the safety of coal mine transportation operating on a mobile terminal, computer terminal or similar computing device is provided, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical sequence is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in a different order than that illustrated herein.

Fig. 2 is a flow chart of a coal mine transportation safety determination method according to an embodiment of the present application. As shown in fig. 2, the method comprises the steps of:

step S201, obtaining video information, wherein the video information is information of a video of a target area acquired by video acquisition equipment, the target area is positioned in the traveling direction of a vehicle, and the minimum distance between the target area and the vehicle is kept unchanged;

specifically, the video capturing apparatus may be mounted on a vehicle such that the video capturing apparatus may capture video of a target area, and thus may acquire video information in a traveling direction of the vehicle.

In particular, the video capture device may be a video camera, video recorder, or the like.

Step S202, positioning information is obtained, wherein the positioning information comprises information of the position of the vehicle and information of the position of a pedestrian acquired by a UWB positioning card;

specifically, UWB devices are installed on the vehicles, UWB identification cards are worn by pedestrians, and positioning information of the vehicles and positioning information of the pedestrians can be determined through UWB accurate positioning technology. In particular, the positioning information may be determined by transmitting a signal like a base station.

Step S203, respectively constructing a first model and a second model according to the video information and the positioning information, and calculating a weighted average of an output result of the first model and an output result of the second model to obtain a target result, wherein the first model is used for preliminarily determining whether coal mine transportation is safe according to the video information, the second model is used for preliminarily determining whether coal mine transportation is safe according to the positioning information, and the target result is used for determining whether coal mine transportation is safe.

Specifically, a first model can be built according to video information, a second model can be built according to positioning information, whether coal mine transportation is safe or not is preliminarily determined according to the first model, a first preliminary detection result is obtained, whether coal mine transportation is safe or not is preliminarily determined according to the second model, a second preliminary detection result is obtained, and whether coal mine transportation is safe or not is finally determined by fusing the two preliminary detection results.

Specifically, if the first preliminary detection result indicates that coal mine transportation is unsafe, and the second preliminary detection result indicates that coal mine transportation is unsafe, the obtained target result is also unsafe. If the first preliminary detection result represents unsafe coal mine transportation and the second preliminary detection result represents safe coal mine transportation, the obtained target result is safe coal mine transportation. And the coal mine transportation safety can be determined by comparing the target results obtained by the two models with a preset threshold value. If the first preliminary detection result represents coal mine transportation safety and the second preliminary detection result represents coal mine transportation safety, the obtained target result is also coal mine transportation safety. If the first preliminary detection result represents coal mine transportation safety and the second preliminary detection result represents coal mine transportation safety, the obtained target result is also coal mine transportation safety.

Through the embodiment, the safety of the driving process is determined in an automatic mode, and the safety of the driving process is determined in a multi-source data fusion mode, so that inaccuracy of single data judgment can be avoided, and the reliability of safety detection in the scheme is higher.

In a specific implementation process, before calculating a weighted average of the output result of the first model and the output result of the second model to obtain a target result, the method further includes the following steps: constructing an initial first model, wherein the initial first model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical image information and historical areas occupied by historical pedestrians in the historical image information, wherein the historical image information is any frame in historical video information; inputting the image information into the initial first model to obtain an initial area corresponding to the image information; constructing the second model, wherein the second model is obtained by training by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical positioning information, and a historical distance between a historical vehicle and a historical pedestrian corresponding to the historical positioning information; and inputting the positioning information into the second model to obtain the distance corresponding to the positioning information.

In the scheme, an initial first model and a second model can be firstly constructed, the occupied area of a pedestrian in an image of a target area can be determined according to the initial first model, the occupied area is larger as the pedestrian is closer to a vehicle, but the initial first model can be trained again subsequently due to the underground environment interference of a coal mine, the distance between the vehicle and the pedestrian can be determined according to the second model, and the accuracy of the second model is higher due to the fact that the second model is obtained based on UWB accurate positioning training.

Specifically, a first model constructed by a deep learning algorithm can be adopted to detect a personnel target, after personnel are identified, the real-time running speed of the vehicle and the image size of the personnel are combined to analyze, and when the preset threshold value is exceeded, an audible and visual alarm mode can be adopted to remind a driver (namely, whether coal mine transportation is safe or not is primarily determined). The first model is constructed by using a deep learning algorithm such as ResNet-34, and the like, can be well compatible with precision and speed, and can realize detection of different scales by adding a Deep Layer Aggregation (DLA) structure on a backbone network, and the first model can be an existing artificial intelligent model, such as a neural network model or a convolutional network model.

For the second model, the distance between the vehicle and the person can be determined according to the positioning information of the vehicle and the positioning information of the person, the UWB device is arranged on the vehicle, the person also carries the UWB identification card, and the distance between the vehicle and the person is determined according to the position of the UWB device and the position of the UWB identification card.

Specifically, the scheme is actually divided into two intelligent fusion stages, wherein the two stages respectively utilize positioning information auxiliary labeling and reliability intelligent fusion to fuse the information of two different modes of video information and positioning information. By analyzing the different properties and the applicable environments of the visual analysis technology and the accurate positioning technology, the conventional fusion technology cannot effectively fuse information of two different modes. The two-stage intelligent fusion technology can perform more accurate information fusion according to different properties among different information, meets the requirements of safety monitoring result accuracy and robustness under actual use conditions, and reduces the cost of relevant model training and deployment. The following describes two intelligent fusion phases respectively.

In order to improve the accuracy of the first model and improve the accuracy of the subsequent safety detection, the area range of the history area corresponds to the distance range of the history distance one by one, and before calculating the weighted average value of the output result of the first model and the output result of the second model to obtain the target result, the method further comprises the following steps: when the area range of the initial area does not correspond to the distance range of the distance, acquiring a target area range of the initial area corresponding to the distance range of the distance; and inputting the target area range into the initial first model, and performing model training on the initial first model again until the area range of the initial area corresponds to the distance range of the distance to obtain the first model.

In the scheme, the first model can be adjusted by using the second model, and the initial first model can be influenced by the underground environment of a coal mine during detection, for example, exposure or dust can lead to inaccurate detection, the second model is used for positioning detection based on UWB, and the positioning detection is a real-time detection process, so that the accuracy of the detection of the second model is higher, the initial first model can be updated by using the second model until the accurate first model is obtained, the accuracy of the first model can be ensured to be higher, the accuracy of the result output by adopting the first model is also higher, and the accuracy of the subsequently obtained target result is further ensured to be higher.

Specifically, as shown in fig. 3, the one-stage fusion is mainly a fusion training process for vision and positioning based on active learning and semi-supervised learning, and involves two systems, namely an intelligent vision system and an accurate positioning system, wherein the intelligent vision system comprises a first model (a personnel vehicle detection model) (also comprises a video acquisition device for shooting videos), the accurate positioning system comprises a second model (a distance calculation model based on positioning data) (also comprises a positioning base station), and from the software perspective, the one-stage fusion further comprises a data reflux module and a model training module.

The positioning information and the video information can be fused in the data reflow module for selecting training videos. The main difficulty of the underground coal mine driving safety detection modeling is that the abnormal data size is small, and the training process of the deep learning algorithm is greatly dependent on the accumulation of data. The traditional method mainly carries out manual marking by marking personnel, which is labor-consuming and difficult, and can not guarantee marking precision. Therefore, the scheme designs the data reflux module based on positioning data fusion aiming at the situation, so that automatic abnormal data selection and accumulation can be performed, the cost is reduced, positioning information and video information can be primarily fused, and the model training can be performed. The module adopts a strategy of double thresholds, namely, a high threshold and a low threshold are designed based on a first model and a second model (the probability of detecting pedestrians is higher than 97% and is higher than 90% and is lower than the high threshold), wherein the high threshold ensures that the false alarm rate is less than or equal to 3%, the false alarm rate is used for online operation, the false alarm is reduced as much as possible on the premise of ensuring the detection rate, the high threshold is used for two-stage fusion, and self-adaptive data fusion is carried out, so that a safety detection result is obtained; the low threshold ensures that the false alarm rate is less than or equal to 10 percent, and when the first model or the second model monitors the video clips exceeding the low threshold, the system can acquire suspected abnormal data (data acquisition) for iteration of the subsequent model.

Based on the data accumulation and reflow technology of fusing accurate positioning information, the conventional method cannot effectively accumulate a large amount of data for training due to low occurrence frequency of abnormal data. The data accumulation and reflow technology based on the fused accurate positioning information can automatically accumulate related data, integrate the positioning information into the first model training, and realize the primary fusion of the data

The data acquired through the data reflow module has less data with obvious abnormality, and the rest of the data are difficult to distinguish and need to be marked. The labeling data are different in requirements under different scenes, and the cost is high. In order to save the labeling cost, reasonably utilize the backflow data to carry out data iteration, intercept videos from a video library, carry out fusion learning training with positioning information, introduce active learning and semi-supervised learning in the model training process, adopt defogging algorithm training with single scale, reduce interference caused by water vapor, promote model robustness, and the model training flow is shown in fig. 4:

a. active learning: in order to solve the problem of labeling mass data, an active learning idea is introduced. Before labeling, a batch of data which is relatively large in help of subsequent model iteration is screened out from the massive data in an active learning mode, and then the batch of data is labeled (a labeling sample is obtained by manual labeling) so as to reduce cost and save time. Several common strategies were first subjected to comparative experiments using small amounts of data, such as: uncertainty sampled queries (Uncertainty Sampling), committee-based queries (Query-By-Committee), model change expected queries (Expected Model Change), etc., after confirmation of appropriate strategies, training is performed;

b. Semi-supervised learning: and (3) utilizing a small number of marked samples after active learning selection, combining a large number of unmarked samples to improve the algorithm capability, and continuing model training, such as Soft teacher. After the model effect is improved, the active learning can screen out important data with a larger probability, so that the model is iterated more efficiently, a closed loop is formed, and the first model can be used online.

By adopting a mode of combining active learning and semi-supervised learning, the detection precision of the first model can be greatly improved only by carrying out a small amount of labeling on the image information of the scene to be distinguished by safety management personnel of coal mine enterprises, and the deployment difficulty of the related model is reduced.

Based on the fusion positioning of active learning and semi-supervised learning, the first model can be adjusted according to the real-time change of the underground coal mine environment, the positioning information of the second model is primarily fused, the relevant information of data acquisition and labeling is provided for deep learning, the cost of deep learning development and deployment is reduced, and the wide application of the model in underground coal mine scenes is ensured.

The underground coal mine visual monitoring training technology based on the combination of active learning and semi-supervised learning aims at the problems that underground coal mine environmental data are difficult to collect and labeling difficulty is high, related data can be automatically collected by combining the active learning and the semi-supervised learning, and a reliable first model and a reliable second model are obtained according to a very small amount of manual labeling results.

The above-described manner of determining the distance between the vehicle and the UWB base station may also be achieved by other means, for example, the above-described method further comprises the steps of, before constructing the above-described second model: acquiring a first time stamp of a ranging request frame sent by the UWB positioning card of the vehicle; acquiring a second time stamp of a ranging response frame sent by a UWB base station and received by the UWB positioning card of the vehicle; and calculating a distance between the vehicle and the UWB base station based at least on the first time stamp and the second time stamp.

According to the scheme, the distance between the vehicle and the base station can be determined according to the unidirectional flight time of the signal, the flight time of the signal can be judged according to the time stamp of the data sent by the UWB positioning card and the time stamp of the data received by the UWB positioning card, and then the flight time is multiplied by the speed of light propagation (or the signal propagation speed), so that the distance between the vehicle and the base station can be obtained, and the distance between the vehicle and the base station can be simply and directly determined.

Specifically, for the distance between the pedestrian and the base station, the distance between the pedestrian and the base station can be determined according to the flight time of signals between the UWB positioning card of the pedestrian and the base station, and then after the distance between the pedestrian and the base station and the distance between the vehicle and the base station time are obtained, the distance of a triangular relationship is obtained, and then the distance between the vehicle and the pedestrian is determined.

Specifically, the distance between the vehicle and the UWB base station may be determined from the difference between the first time stamp and the second time stamp and the speed of light propagation.

Specifically, the above scheme adopts a unidirectional flight time model to perform distance calculation, and of course, a bidirectional flight time model can also be used to perform distance calculation on the position information uploaded by the UWB accurate positioning system. Wherein T is defined respectively _SP Transmitting a time stamp of a ranging request frame for a vehicle-mounted identification card (UWB positioning card), T _RP Time stamp, T, of receiving ranging request frame for UWB base station _SR Transmitting a time stamp, T, of a ranging response frame for a UWB base station _RR Receiving a time stamp of a ranging response frame for the vehicle-mounted identification card, T _SF Transmitting a ranging data frame (T) for a vehicle-mounted identification card _SP ，T _RP ，T _SR ) Time stamp, T of _RF Receiving a ranging data frame (T _SP ，T _RP ，T _SF ) Is used for the time stamp of (a),is T _SR And T is _RP Time difference of->Is T _SF And T _RR After receiving the ranging data frame sent by the vehicle-mounted identification card, the UWB base station calculates the distance according to the received data according to the following procedures:

(1) Calculating the time difference between the sending and receiving of the ranging frames by the vehicle-mounted identification card: t (T) _TRT ＝T _RR -T _SP ；

(2) Calculating the time difference between the UWB base station receiving and transmitting the ranging frame:

(3) Calculating the time difference between the transmission of the ranging frame and the reception of the data frame by the UWB base station: t (T) _ART ＝T _RF -T _SR ；

(4) Calculating the time difference between the data frame sent by the vehicle-mounted identification card and the ranging frame received by the vehicle-mounted identification card:

(5) According toAnd->Respectively calculating the flight time if the flight time is equal to each other:

(6) Calculating the distance between the vehicle-mounted identification card and the UWB base station: l=t _TOF gC, L is distance, C is speed of light.

The first model based on deep learning and the second model based on accurate positioning have different reliability in different scenes under the coal mine, and in the specific implementation process, the weighted average of the output result of the first model and the output result of the second model is calculated to obtain a target result, and the method can be realized by the following steps: analyzing the video quality of the video information according to a PSNR avg.MSE algorithm to obtain a first evaluation score; analyzing the signal intensity of the positioning information according to an RSS algorithm to obtain a second evaluation score; determining a first weight coefficient of a first output result of the first model and a second weight coefficient of a second output result of the second model according to the first evaluation score and the second evaluation score; and calculating a sum of a first product, which is a product of the first output result of the first model and the first weight coefficient, and a second product, which is a product of the second output result of the second model and the second weight coefficient, to obtain the target result.

In the scheme, the reliability of different models can be calculated in real time, the weights of the different models are determined by analyzing the quality of video and the strength of signals, and then the output results of the two models are adaptively fused, so that the accuracy and the reliability of the whole detection of the scheme are improved.

Specifically, the above embodiment is actually a two-stage fusion process, the two-stage fusion is an adaptive intelligent fusion process based on reliability analysis, and in the running process of a single vehicle, the second model is based on the real-time position of the vehicle and personnel in the three-dimensional space of the coal mine, which is acquired by the UWB accurate positioning technology, and the actual distance between the vehicle and the personnel in the adjacent area is calculated according to the acquired roadway topological relation, and when the actual distance is smaller than a safety threshold value, an alarm is given. As shown in fig. 5, after the result of the safety inspection analysis of the first model and the result of the safety inspection analysis of the second model are obtained, fusion analysis of the two results is also required. Considering that the condition that the video quality such as dust, water mist, dim light and the like is low based on the first model is that the detection accuracy is greatly reduced, and the second model based on UWB accurate positioning cannot provide accurate results when the signal is poor. The scheme uses the self-adaptive result fusion method to better fuse the safety monitoring results obtained by the two models, so that the overall accuracy is improved, and meanwhile, the robustness is improved.

Specifically, as shown in fig. 5, a video quality analysis algorithm may be used to analyze video quality of video information, and a signal strength detection algorithm may be used to analyze signal strength of positioning information, where the video quality analysis algorithm and the signal strength detection algorithm are described as follows:

the video quality analysis uses the PSNR avg.mse algorithm (image evaluation algorithm) to analyze the quality of the surveillance video. PSNR (peak signal to noise ratio) is the ratio of the energy of the peak signal to the average energy of the noise, essentially comparing the difference in pixel values of two images. The unit of PSNR is dB, the larger the value, the smaller the distortion, the formula:

MSE represents the mean square error of two m n monochrome images I and K, I represents the reference image, K represents the image to be evaluated, and x and y represent the coordinates of the pixels in the image. For PSNR avg.mse, when aggregating the frame-by-frame scores of the entire video, the arithmetic mean of MSE is first calculated, and then the logarithm is taken to obtain a first evaluation score, with the specific formula:

where V represents the original video frame,table referring to the frame, n represents the total number of frames of the video, MAX represents the maximum value of the color of the image point, and 255 if each sample point is represented by 8 bits.

The signal strength analysis adopts an RSS algorithm (signal strength analysis algorithm) to analyze the signal strength, and when the RSS algorithm is used, the distance between a target and a receiver and the signal strength can be estimated by measuring the energy between nodes, and because the strength of a received signal is inversely proportional to the propagation distance, when a positioning signal is far away from a positioning base station, the strength of the positioning signal is weakened, and meanwhile, the accuracy and the reliability of positioning data are also greatly reduced.

In order to improve the accuracy of the target result, further determining the reliability and accuracy of the output result of the first model and the output result of the second model, and determining the first weight coefficient of the first output result of the first model and the second weight coefficient of the second output result of the second model according to the first evaluation score and the second evaluation score, respectively, specifically may be implemented by the following steps: determining the first weight coefficient according to the first evaluation score, and determining the second weight coefficient according to the second evaluation score, wherein the first evaluation score and the first weight coefficient are positively correlated, the second evaluation score and the second weight coefficient are positively correlated, and the magnitude relation between the first weight coefficient and the second weight coefficient is related to a first difference value and a second difference value, wherein the first difference value is the difference value between the first evaluation score and a first score threshold value, and the second difference value is the value between the second evaluation score and a second score threshold value; and determining that the first weight coefficient is larger than the second weight coefficient when the first difference is larger than the second difference, determining that the first weight coefficient is smaller than the second weight coefficient when the first difference is smaller than the second difference, and determining that the first weight coefficient is equal to the second weight coefficient when the first difference is equal to the second difference.

In the scheme, the self-adaptive result fusion technology is used, so that the reliability of the first output result of the first model and the reliability of the second output result of the second model are further determined, different weight coefficients are determined through different output results, and then the self-adaptive result fusion can be performed subsequently, the fusion algorithm can be dynamically adjusted according to the real-time environment under the coal mine, the accuracy and the reliability of the whole scheme are ensured, and the scheme is more suitable for the practical environment under the coal mine.

Specifically, in the adaptive result fusion method, for a first video-based model, the quality of video may be evaluated by a video quality detection algorithm. For the second model based on UWB accurate positioning, the signal strength of the positioning information of UWB accurate positioning can be acquired. And respectively carrying out normalization processing on the video quality parameters and the signal intensity to obtain weight values (a first weight coefficient and a second weight coefficient) of monitoring results of different models, as shown in fig. 5, and fusing the monitoring results according to the weight values. The fusion formula is:

wherein R represents the result after fusion, W _V A first weight coefficient representing the first model, V (x) representing a first input of the first model As a result, W _D And a second weight coefficient representing a second model, and D (x) represents a second output result of the second model. And (3) carrying out normalization treatment on the R to obtain a final target result. The weight coefficient can be analyzed, and when the weight coefficients of the two models are lower than a safety threshold (which can be 0.5), an alarm can be given to remind safety monitoring personnel to perform manual rechecking, so that the overall detection accuracy and precision are guaranteed to be good.

Specifically, in this scheme, through the degree of depth intelligence fusion to colliery current video intelligent analysis system in pit and accurate positioning system reach accurate reliable safety monitoring pedestrian and do not drive a vehicle's function, and then solved the supplementary transportation safety monitoring system construction in the pit of colliery in the past and need expend a large amount of manpower and materials, the accuracy is lower and the relatively poor problem of reliability.

In addition, the prompt voice can be distinguished under different conditions of two-stage fusion, and when the reliability of the first model and the second model is high, a pedestrian in front is prompted; and when the reliability of the second model is high and the reliability of the first model is low, prompting that the second model detects pedestrians, otherwise prompting that the video detects pedestrians, and prompting that the manual review is performed when the reliability of the two models is low.

Specifically, in the scheme, aiming at complex scenes of safety monitoring of underground coal mine driving and pedestrians, single information sources easily cause the condition of low accuracy of driving safety monitoring, the accurate positioning technology and the deep learning technology are fused more comprehensively through the two-stage intelligent fusion technology, and the deployment of a high-precision safety monitoring model can be realized on the basis of the existing accurate positioning system and driving video recording system (video intelligent analysis system) of the coal mine.

The two-stage intelligent fusion model fully fuses and complements the two different modes of the visual data and the positioning data from the aspects of data accumulation, model training and reliability-based self-adaptive fusion. The accurate positioning technology and the visual detection technology based on deep learning are fused to perform safety detection on driving and pedestrians, double prevention and guarantee can be achieved, the driving and pedestrians can be effectively recorded and controlled through mutual combination, safety risks are reduced, monitoring robustness is improved, and intelligent model application is provided for construction of an overall business system. Meanwhile, the scheme does not need to use special visual sensors such as infrared cameras and the like aiming at complex environments in the coal mine, can be deployed and applied on the existing system in the coal mine, and reduces the deployment cost of the system.

In some embodiments, after calculating a weighted average of the output result of the first model and the output result of the second model to obtain the target result, the method further includes the steps of: under the condition that the target result is greater than or equal to a preset threshold value, determining the coal mine transportation safety; and under the condition that the target result is smaller than the preset threshold value, determining that the coal mine transportation is unsafe, and generating alarm information.

In the scheme, the target result can be directly compared with the preset threshold value, whether coal mine transportation is safe or not is simply and directly determined according to the size relation between the target result and the preset threshold value, a complex calculation process is not needed, and further, alarm information is generated to prompt a driver in time under the condition that coal mine transportation is unsafe.

In addition, in the scheme, the UWB-based accurate positioning technology can be used for safety monitoring of underground coal mine traveling vehicles and pedestrians. However, due to the fact that the underground roadway structure of the coal mine is complex, in order to avoid the situation that the positioning accuracy is greatly reduced due to the fact that UWB accurate positioning is easy to generate due to weak signals, the safety monitoring accuracy of the whole scheme is improved, the whole operation efficiency of the coal mine is improved, a visual safety monitoring technology based on an infrared camera can be used, the infrared camera is more suitable for the environment of insufficient illumination of the underground coal mine, and interference of dust, water mist and the like on the accuracy of a visual sensor is achieved. But the deployment cost of the infrared camera is higher, additional hardware purchase and deployment are needed, and the overall deployment cost is increased.

The embodiment of the application also provides a coal mine transportation safety determination device, and the coal mine transportation safety determination device can be used for executing the coal mine transportation safety determination method. The device is used for realizing the above embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The safety determination device for coal mine transportation provided by the embodiment of the application is described below.

Fig. 6 is a block diagram of a coal mine transportation safety determination apparatus according to an embodiment of the present application. As shown in fig. 6, the apparatus includes:

a first obtaining unit 10, configured to obtain video information, where the video information is information of a video of a target area collected by a video collecting device, the target area is located in a traveling direction of a vehicle, and a minimum distance between the target area and the vehicle is kept unchanged;

A second obtaining unit 20, configured to obtain positioning information, where the positioning information includes information of a position of the vehicle and information of a position of a pedestrian acquired by a UWB positioning card;

a first determining unit 30, configured to construct a first model and a second model according to the video information and the positioning information, respectively, and calculate a weighted average of an output result of the first model and an output result of the second model, to obtain a target result, where the first model is used for primarily determining whether coal mine transportation is safe according to the video information, the second model is used for primarily determining whether coal mine transportation is safe according to the positioning information, and the target result is used for determining whether coal mine transportation is safe.

In a specific implementation process, the device further includes a first construction unit, a first processing unit, a second construction unit and a second processing unit, where the first construction unit is configured to construct an initial first model before calculating a weighted average of an output result of the first model and an output result of the second model to obtain a target result, where the initial first model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired in a historical time period: historical image information and historical areas occupied by historical pedestrians in the historical image information, wherein the historical image information is any frame in historical video information; the first processing unit is used for inputting the image information into the initial first model to obtain an initial area corresponding to the image information; the second building unit is configured to build the second model, where the second model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes training data acquired in a historical period: historical positioning information, and a historical distance between a historical vehicle and a historical pedestrian corresponding to the historical positioning information; the second processing unit is used for inputting the positioning information into the second model to obtain the distance corresponding to the positioning information.

In order to improve the accuracy of the first model and improve the accuracy of the subsequent safety detection, the area range of the historical area corresponds to the distance range of the historical distance one by one, the device further comprises a third acquisition unit and a third processing unit, wherein the third acquisition unit is used for acquiring the target area range of the initial area corresponding to the distance range of the distance before calculating the weighted average value of the output result of the first model and the output result of the second model to obtain the target result; and the third processing unit is used for inputting the target area range into the initial first model, and performing model training on the initial first model again until the area range of the initial area corresponds to the distance range of the distance to obtain the first model.

The above manner of determining the distance between the vehicle and the UWB base station may also be implemented in other manners, for example, the apparatus further includes a fourth obtaining unit, a fifth obtaining unit, and a calculating unit, where the fourth obtaining unit is configured to obtain, before the second model is constructed, a first timestamp of the ranging request frame sent by the UWB locator card of the vehicle; the fifth acquisition unit is used for acquiring a second timestamp of a ranging response frame sent by the UWB base station and received by the UWB positioning card of the vehicle; the calculating unit is used for calculating the distance between the vehicle and the UWB base station according to at least the first time stamp and the second time stamp.

The first model based on deep learning and the second model based on accurate positioning have different reliability in different scenes under the coal mine, and in the specific implementation process, the first determining unit comprises a first processing module, a second processing module, a determining module and a calculating module, wherein the first processing module is used for analyzing the video quality of the video information according to a PSNR avg.MSE algorithm to obtain a first evaluation score; the second processing module is used for analyzing the signal intensity of the positioning information according to an RSS algorithm to obtain a second evaluation score; the determining module is used for determining a first weight coefficient of a first output result of the first model and a second weight coefficient of a second output result of the second model according to the first evaluation score and the second evaluation score respectively; the calculation module is configured to calculate a sum of a first product and a second product, and obtain the target result, where the first product is a product of the first output result of the first model and the first weight coefficient, and the second product is a product of the second output result of the second model and the second weight coefficient.

In order to improve the accuracy of the target result, further determining the reliability and accuracy of the output result of the first model and the output result of the second model, wherein the determining module comprises a first determining submodule and a second determining submodule, the first determining submodule is used for determining the first weight coefficient according to the first evaluation score and determining the second weight coefficient according to the second evaluation score, the first evaluation score and the first weight coefficient are in positive correlation, the second evaluation score and the second weight coefficient are in positive correlation, and the magnitude relation between the first weight coefficient and the second weight coefficient is related to a first difference value and a second difference value, wherein the first difference value is the difference value between the first evaluation score and a first score threshold, and the second difference value is the value between the second evaluation score and a second score threshold; the second determining submodule is configured to determine that the first weight coefficient is greater than the second weight coefficient when the first difference is greater than the second difference, determine that the first weight coefficient is less than the second weight coefficient when the first difference is less than the second difference, and determine that the first weight coefficient is equal to the second weight coefficient when the first difference is equal to the second difference.

In some embodiments, the apparatus further includes a second determining unit and a third determining unit, where the second determining unit is configured to determine coal mine transportation safety when the target result is greater than or equal to a preset threshold after calculating a weighted average of the output result of the first model and the output result of the second model to obtain the target result; and the third determining unit is used for determining unsafe transportation of the coal mine and generating alarm information under the condition that the target result is smaller than the preset threshold value.

The coal mine transportation safety determination device comprises a processor and a memory, wherein the first acquisition unit, the second acquisition unit, the first determination unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions. The modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The inner core can be provided with one or more than one, and the problem of low reliability of safety monitoring of the underground coal mine auxiliary transportation system in the prior art is solved by adjusting the parameters of the inner core.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

The embodiment of the invention provides a computer readable storage medium, which comprises a stored program, wherein the program is used for controlling equipment where the computer readable storage medium is located to execute the coal mine transportation safety determination method.

The embodiment of the application provides a processor which is used for running a program, wherein the safety determination method for coal mine transportation is executed when the program runs.

The application also provides a coal mine transportation safety determination system comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise a safety determination method for executing any one of the coal mine transportation safety determination methods.

The embodiment of the application provides equipment, which comprises a processor, a memory and a program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize at least the following coal mine transportation safety determination method steps. The device herein may be a server, PC, PAD, cell phone, etc.

The application also provides a computer program product adapted to perform a program initialized with at least the following safety determination method steps of coal mine transportation when executed on a data processing device.

It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) According to the coal mine transportation safety determination method, the safety of the driving process is determined in an automatic mode, and the safety of the driving process is determined in a multi-source data fusion mode, so that inaccuracy of single data judgment can be avoided, and therefore the reliability of safety detection in the scheme is high.

2) The coal mine transportation safety determination device determines the safety of the driving process in an automatic mode, and determines the safety of the driving process in a multi-source data fusion mode, so that inaccuracy of single data judgment can be avoided, and the safety detection reliability is higher in the scheme.

The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for determining the safety of coal mine transportation, comprising:

Acquiring video information, wherein the video information is information of a video of a target area acquired by video acquisition equipment, the target area is positioned in the traveling direction of a vehicle, and the minimum distance between the target area and the vehicle is kept unchanged;

acquiring positioning information, wherein the positioning information comprises information of the position of the vehicle and information of the position of a pedestrian acquired by a UWB positioning card;

respectively constructing a first model and a second model according to the video information and the positioning information, and calculating a weighted average value of an output result of the first model and an output result of the second model to obtain a target result, wherein the first model is used for preliminarily determining whether coal mine transportation is safe or not according to the video information, the second model is used for preliminarily determining whether coal mine transportation is safe or not according to the positioning information, and the target result is used for determining whether coal mine transportation is safe or not.

2. The method of claim 1, wherein prior to calculating a weighted average of the output results of the first model and the output results of the second model to obtain a target result, the method further comprises:

Constructing an initial first model, wherein the initial first model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical image information and historical area occupied by historical pedestrians in the historical image information, wherein the historical image information is any frame in historical video information;

inputting the image information into the initial first model to obtain an initial area corresponding to the image information;

constructing the second model, wherein the second model is trained by using a plurality of sets of training data, and each set of training data in the plurality of sets of training data comprises data acquired in a historical time period: historical positioning information and a historical distance between a historical vehicle and a historical pedestrian corresponding to the historical positioning information;

and inputting the positioning information into the second model to obtain the distance corresponding to the positioning information.

3. The method of claim 2, wherein the area range of the historical area and the distance range of the historical distance correspond one-to-one, and wherein prior to calculating a weighted average of the output result of the first model and the output result of the second model to obtain the target result, the method further comprises:

Acquiring a target area range of the initial area corresponding to the distance range of the distance under the condition that the area range of the initial area does not correspond to the distance range of the distance;

and inputting the target area range into the initial first model, and performing model training on the initial first model again until the area range of the initial area corresponds to the distance range of the distance to obtain the first model.

4. The method of claim 2, wherein prior to constructing the second model, the method further comprises:

acquiring a first time stamp of a ranging request frame sent by the UWB positioning card of the vehicle;

acquiring a second time stamp of a ranging response frame sent by a UWB base station and received by the UWB positioning card of the vehicle;

and calculating the distance between the vehicle and the UWB base station according to at least the first time stamp and the second time stamp.

5. The method of claim 2, wherein calculating a weighted average of the output results of the first model and the output results of the second model to obtain a target result comprises:

analyzing the video quality of the video information according to a PSNR avg.MSE algorithm to obtain a first evaluation score;

Analyzing the signal intensity of the positioning information according to an RSS algorithm to obtain a second evaluation score;

determining a first weight coefficient of a first output result of the first model and a second weight coefficient of a second output result of the second model according to the first evaluation score and the second evaluation score;

and calculating the sum of a first product and a second product, and obtaining the target result, wherein the first product is the product of the first output result of the first model and the first weight coefficient, and the second product is the product of the second output result of the second model and the second weight coefficient.

6. The method of claim 5, wherein determining a first weight coefficient of a first output result of the first model and a second weight coefficient of a second output result of the second model based on the first and second evaluation scores, respectively, comprises:

determining the first weight coefficient according to the first evaluation score, determining the second weight coefficient according to the second evaluation score, wherein the first evaluation score and the first weight coefficient are positively correlated, the second evaluation score and the second weight coefficient are positively correlated, and the magnitude relation between the first weight coefficient and the second weight coefficient is related to a first difference value and a second difference value, wherein the first difference value is the difference value between the first evaluation score and a first score threshold value, and the second difference value is the value between the second evaluation score and a second score threshold value;

And determining that the first weight coefficient is larger than the second weight coefficient when the first difference value is larger than the second difference value, determining that the first weight coefficient is smaller than the second weight coefficient when the first difference value is smaller than the second difference value, and determining that the first weight coefficient is equal to the second weight coefficient when the first difference value is equal to the second difference value.

7. The method of claim 5, wherein after calculating a weighted average of the output results of the first model and the output results of the second model to obtain a target result, the method further comprises:

under the condition that the target result is greater than or equal to a preset threshold value, determining coal mine transportation safety;

and under the condition that the target result is smaller than the preset threshold value, determining that coal mine transportation is unsafe, and generating alarm information.

8. A coal mine transportation safety determination apparatus, comprising:

a first obtaining unit, configured to obtain video information, where the video information is information of a video of a target area collected by a video collecting device, the target area is located in a traveling direction of a vehicle, and a minimum distance between the target area and the vehicle is kept unchanged;

The second acquisition unit is used for acquiring positioning information, wherein the positioning information comprises information of the position of the vehicle and information of the position of a pedestrian acquired by the UWB positioning card;

the first determining unit is used for respectively constructing a first model and a second model according to the video information and the positioning information, calculating a weighted average value of an output result of the first model and an output result of the second model, and obtaining a target result, wherein the first model is used for preliminarily determining whether coal mine transportation is safe according to the video information, the second model is used for preliminarily determining whether coal mine transportation is safe according to the positioning information, and the target result is used for determining whether coal mine transportation is safe.

9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run controls a device in which the computer readable storage medium is located to perform the method of determining safety of coal mine transportation according to any one of claims 1 to 7.

10. A coal mine transportation safety determination system, comprising: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a security determination method for performing the coal mine transportation of any of claims 1 to 7.