CN116486340A

CN116486340A - Foreign matter detection method and device for conveyor belt, storage medium and processor

Info

Publication number: CN116486340A
Application number: CN202310461725.XA
Authority: CN
Inventors: 雷晓树; 刘利明; 高旺; 乔治忠; 胡金良; 苏兴龙
Original assignee: Shenhua Zhungeer Energy Co Ltd
Current assignee: Shenhua Zhungeer Energy Co Ltd
Priority date: 2023-04-25
Filing date: 2023-04-25
Publication date: 2023-07-25

Abstract

The application provides a foreign matter detection method, a device, a storage medium and a processor of a conveyor belt, wherein the method comprises the following steps: comprising the following steps: acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment; inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model and is used for restoring the image with the first definition into the image with the second definition, and the second definition is higher than the first definition; and inputting the second image into a foreign matter detection model, determining whether the second image contains foreign matters, wherein the foreign matter detection model is a model obtained by training the improved YOLOX model, and is at least used for determining whether the second image contains foreign matters. The method solves the problem that the deep learning model in the prior art has poor recognition capability on the image foreign matters with motion blur.

Description

Foreign matter detection method and device for conveyor belt, storage medium and processor

Technical Field

The present application relates to the field of image processing technology, and in particular, to a method for detecting a foreign object on a conveyor belt, a device for detecting a foreign object on a conveyor belt, a storage medium, a processor, and an electronic apparatus.

Background

The coal mine safety problem is one of the important concerns in China, and large foreign matters such as iron sheets and steel bars are mixed into a conveyor belt in the coal conveying process, so that faults such as tearing of the conveyor belt and the like are easy to cause, and the coal mine safety problem is one of factors for causing accidents.

The prior main means for detecting the foreign matters of the conveyor belt are human eyes, rays and video identification, the former two methods have high cost, and the human eyes identification has higher concentration requirements on workers and larger limitations, so that the video identification becomes the most widely applied foreign matters detection method of the conveyor belt at present by virtue of low cost, higher identification rate and simpler deployment mode.

The conventional foreign matter identification method based on video images can be summarized as follows: 1. collecting a conveyor belt foreign matter image as training data; 2. building a deep learning network; 3. training the built deep learning network through training data; 4. and extracting images in the video according to the frames, and identifying whether foreign matters exist in the extracted images in the video through a trained network.

However, when the motion speed of the conveyor belt is too high, the problem that the images extracted from the real-time monitoring video have motion blur exists, and the trained deep learning model has the problem that the foreign matter recognition capability is poor and the detection performance is reduced for the motion blurred images.

Disclosure of Invention

The main object of the present application is to provide a method for detecting a foreign object on a conveyor belt, a device for detecting a foreign object on a conveyor belt, a storage medium, a processor and an electronic device, so as to at least solve the problem that a deep learning model in the prior art has poor recognition capability on a motion blurred image foreign object.

In order to achieve the above object, according to one aspect of the present application, there is provided a foreign matter detection method of a conveyor belt, the method including: acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment; inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model and is used for reducing the image with the first definition into an image with a second definition, and the second definition is higher than the first definition; and inputting the second image into a foreign matter detection model, and determining whether the foreign matter exists in the second image, wherein the foreign matter detection model is a model obtained by training an improved YOLOX model, and is at least used for determining whether the foreign matter exists in the image.

Optionally, the foreign object detection model is further configured to label a bounding box of the foreign object in an image, and further configured to determine a type of the foreign object in the image, and after the second image is input into the foreign object detection model, determine whether the foreign object exists in the second image, the method further includes: determining, in a pixel coordinate system, whether the bounding box of the foreign object is within a preset region, in the case that the foreign object is present in the second image; acquiring the type of the foreign matter under the condition that the boundary box is in the preset area; generating first alarm information under the condition that the type of the foreign matter is a first preset type, wherein the first alarm information is used for representing information that the foreign matter of the first preset type threatens the safety of the conveyor belt, and the first preset type at least comprises one of the following: anchor rods and channel steel.

Optionally, after the type of the foreign matter is acquired, the method further includes: determining a foreign object area, which is an area of the bounding box of the foreign object, in case the type of the foreign object is a second preset type, the second preset type comprising at least one of: gangue; and under the condition that the foreign matter area is larger than a preset area, generating second alarm information, wherein the second alarm information is used for representing the information of the second preset type that the foreign matter threatens the safety of the conveyor belt.

Optionally, the method further comprises: acquiring a plurality of third images and a plurality of fourth images, wherein the third images are images of the second definition of the conveyor belt at a first historical moment, the conveyor belt is in a static state at the first historical moment, the fourth images are images of the first definition of the conveyor belt at a second historical moment, and the conveyor belt is in the motion state at the second historical moment; adding noise to each third image respectively to obtain various normal distribution noises, wherein the third images are in one-to-one correspondence with the normal distribution noises; training the U-Net network model using a plurality of sets of first training data, each set of first training data comprising: the normal distribution noise and a target image, wherein the target image is any one of all the fourth images; and stopping training the U-Net network model until the loss function of the U-Net network model is converged, and obtaining the removal model.

Optionally, the method further comprises: and training the improved YOLOX model by using a plurality of sets of second training data to obtain the foreign object detection model, wherein each set of second training data in the plurality of sets of second training data comprises: the device comprises a fifth image and a foreign matter type corresponding to the fifth image, wherein the fifth image is an image with the second definition of the conveyor belt at a third historical moment, the conveyor belt is in the motion state at the third historical moment, the fifth image corresponds to the foreign matter type one by one, and the foreign matter type is the type of the foreign matter in the fifth image corresponding to the foreign matter type.

Optionally, the improved YOLOX model is a model obtained by adding a hole convolution layer between an up-sampling layer and a cross-stage layer in the YOLOX model.

According to another aspect of the present application, there is provided a foreign matter detection device of a conveyor belt, the device including: the first acquisition unit is used for acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment; the removing unit is used for inputting the first image into a removing model to obtain a second image, the removing model is a model obtained by training a U-Net network model, the removing model is used for reducing the image with the first definition into the image with the second definition, and the second definition is higher than the first definition; the first determining unit inputs the second image into a foreign object detection model to determine whether the foreign object exists in the second image, wherein the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least used for determining whether the foreign object exists in the image.

According to still another aspect of the present application, there is provided a computer-readable storage medium including a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform any one of the foreign matter detection methods of the conveyor belt.

According to still another aspect of the present application, there is provided a processor for running a program, wherein the program runs while executing any one of the foreign matter detection methods of the conveyor belt.

According to one aspect of the present application, there is provided an electronic device comprising: the apparatus includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a foreign matter detection method for executing any one of the conveyor belts.

By applying the technical scheme, the method firstly reduces the low-definition image of the conveyor belt into the high-definition image through the removal model, then adopts the foreign matter detection model obtained by training the improved YOLOX model to identify whether the foreign matters exist in the high-definition image, thereby improving the accuracy of detecting the foreign matters of the conveyor belt.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

fig. 1 is a block diagram showing a hardware configuration of a mobile terminal performing a foreign matter detection method of a conveyor belt according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a method for detecting foreign matters in a conveyor belt according to an embodiment of the present application;

FIG. 3 illustrates a schematic diagram of a removal model training process provided in accordance with an embodiment of the present application;

FIG. 4 shows a schematic diagram of a backbone feature extraction network architecture provided in accordance with an embodiment of the present application;

FIG. 5 illustrates a schematic diagram of a hole convolution layer structure provided in accordance with an embodiment of the present disclosure;

FIG. 6 is a flow chart illustrating another method of detecting foreign matter on a conveyor belt according to an embodiment of the present application;

fig. 7 shows a block diagram of a foreign matter detection device of a conveyor belt according to an embodiment of the present application.

Detailed Description

It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

As described in the background art, in order to solve the problem that the deep learning model in the prior art has poor recognition capability for the motion blurred image foreign matter, embodiments of the present application provide a foreign matter detection method for a conveyor belt, a foreign matter detection device for a conveyor belt, a storage medium, a processor, and an electronic device.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking a mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of a mobile terminal according to a method for detecting foreign matters on a conveyor belt according to an embodiment of the present invention. As shown in fig. 1, a mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a display method of device information in an embodiment of the present invention, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, that is, to implement the above-described method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.

In the present embodiment, there is provided a foreign matter detection method of a conveyor belt operating on a mobile terminal, a computer terminal, or the like, it is to be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown or described herein.

Fig. 2 is a flowchart of a foreign matter detection method of a conveyor belt according to an embodiment of the present application. As shown in fig. 2, the method comprises the steps of:

step S201, acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment;

specifically, in order to determine whether a foreign object exists on the conveyor belt in real time during the movement of the conveyor belt, it is necessary to acquire an image of the conveyor belt in real time, that is, acquire a first image in real time, and the current time is any time during the movement of the conveyor belt.

Step S202, inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model, and is used for restoring the image with the first definition into the image with the second definition, and the second definition is higher than the first definition;

Specifically, when the belt moves at too high a speed during the movement of the belt, the first image may have a problem of blurring, that is, the sharpness of the first image may be low, the first image may be input to the removal model, and the first image may be restored to a high-sharpness image (second image).

In order to improve accuracy of detection of foreign matters on the conveyor belt, in an alternative scheme, the method further comprises:

acquiring a plurality of third images and a plurality of fourth images, wherein the third images are images of the second definition of the conveyor belt at a first history time, the conveyor belt is in a stationary state at the first history time, the fourth images are images of the first definition of the conveyor belt at a second history time, and the conveyor belt is in the moving state at the second history time;

specifically, the application constructs a data set, the data set includes a training set, the training set includes a subset 1 and a subset 2, the subset 1 is 2000 images (third image) acquired by a camera in a static state of a conveyor belt, a fourth image may be an image of a first definition of the conveyor belt acquired in a moving state of the conveyor belt, and also may process an image of a second definition of the conveyor belt to obtain an image of the first definition, and program codes of image processing are as follows:

Adding noise to each third image respectively to obtain multiple normal distribution noises, wherein the third images are in one-to-one correspondence with the normal distribution noises;

specifically, based on the idea of the diffusion model, in the diffusion phase, the third image x is presented with ₀ Adding noise to sequentially generate x ₁ ，x ₂ ，…，x _t-1 ，x _t ，…，x _T-1 ，x _T Which satisfies the formulaWherein x is _T Is normally distributed noise, x _t To a third image x ₀ Adding the t-th noise to obtain noise, z _t Is the t noise, beta _t Is the weight, beta _t Increasing with increasing t, beta ranges from 10 ^-4 Up to 2X 10 ^-2 And satisfies the linear change, the value of T is 1000, and the 1-beta _t ＝α _t Formula->Can be expressed asWherein->z is the sample value of the standard normal distribution.

Training the U-Net network model using a plurality of sets of first training data, each set of first training data comprising: the normal distribution noise and a target image, wherein the target image is any one of all the fourth images;

specifically, based on the idea of the diffusion model, in the denoising stage, as shown in fig. 3, the training process of the U-Net network model is as follows: the U-Net network model distributes noise x normally _T Splicing with the target image to obtain a restored image x _T-1 Then, the U-Net network model will restore the image x _T-1 Splicing with the target image, and continuously restoring until the U-Net network model restores the image x ₁ And splicing the target image to obtain a clear image, wherein the target image is one fourth image randomly selected from all the fourth images.

And stopping training the U-Net network model until the loss function of the U-Net network model is converged, and obtaining the removal model.

Specifically, the U-net network model of the present application is similar to the conventional U-net network model structure, and is different in that the input of the conventional U-net network model is 3 channels, the input of the conventional U-net network model structure of the present application is 6 channels, that is, the conventional U-net network model is input 3 channels of noise, the output is 3 channels of clear images, the present application is a spliced image of input 6 channels, and the output is a 6 channel image in which two clear RGB images are spliced together.

In this embodiment, noise is added to the third image to obtain normal distribution noise, the normal distribution noise and the fourth image form first training data to train the U-Net network model until the loss function of the U-Net network model converges to obtain a removal model, and compared with the idea of a traditional diffusion model, the method and the device adopt the fourth image, namely a low-definition image of the conveyor belt, as constraint conditions in the denoising stage, so that the obtained removal model has higher deblurring capability, and can restore the low-definition image to a higher-definition image, thereby further improving the accuracy of foreign matter detection of the conveyor belt.

Step S203, inputting the second image into a foreign object detection model, determining whether a foreign object exists in the second image, where the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least used for determining whether the foreign object exists in the image.

Specifically, after the first image is restored to the high-definition image (second image), the improved YOLOX model is used to identify whether foreign matter exists in the high-definition image, so that the accuracy of detecting the foreign matter of the conveyor belt is improved.

In order to improve accuracy of multi-scale foreign matter detection of the conveyor belt, in an alternative scheme, the improved YOLOX model is a model obtained by adding a cavity convolution layer between an up-sampling layer and a cross-stage layer in the YOLOX model.

In this embodiment, an image input to the YOLOX model is firstly subjected to feature extraction in a trunk feature extraction network in the YOLOX model, the extracted features can be called feature layers, and are feature sets of the image input to the YOLOX model, as shown in fig. 4, the trunk feature extraction network mainly comprises a Focus layer, a CSP (Cross Stage Partial) 1X layer, a CSP2X layer (the above-mentioned cross-stage layer), an upsampled layer (the above-mentioned upsampling layer) and the like, the Focus layer is used for taking out every other pixel of the input image in the spatial dimension, and then stitching is performed, so that the information of the image width is reduced, the channel number is increased, the parameter quantity is reduced under the condition that the original information is less lost, the effects of the CSP1X layer and the CSP2X layer are the same, and the purpose of the Up Sample layer is to alleviate the problem that the size of the deep layer is too small.

Specifically, as shown in fig. 5, the hole convolution layer is divided into two parts: the multi-branch convolution layer provides different sizes of receptive fields for an input feature map through expansion convolution, the multi-branch convolution layer is used for fusing traffic information from three branch receptive fields, multi-scale precision prediction is improved, the multi-branch convolution layer comprises expansion convolution layers, BN (Batch Normalization) layers and ReLU (Rectified Linear Unit) activation layers, the purpose of the expansion convolution is to learn features of different scales, the purpose of the BN layers is to prevent gradient disappearance and overfitting, the purpose of the ReLU activation layer is to enable linear mapping to be converted into nonlinear mapping, the problem that a linear model cannot solve is solved, expansion convolution in three parallel branches has the same kernel size but different expansion rates, specifically, each expansion convolution kernel is 3×3, the expansion rates of different branches are respectively 1, 3 and 5, expansion convolution supports the exponential expansion receptive fields without losing resolution, and in the expansion convolution operation of the expansion convolution, elements of the convolution kernels are spaced, the size of the space depends on the expansion rate, and are different from that of elements of convolution kernels in standard convolution operation.

In order to improve accuracy of detection of foreign matters on the conveyor belt, in another alternative, the method further includes:

training the improved YOLOX model by using a plurality of sets of second training data to obtain the foreign object detection model, wherein each set of second training data in the plurality of sets of second training data comprises: a fifth image and a foreign object type corresponding to the fifth image, wherein the fifth image is an image of the second definition of the conveyor belt at a third history time, the conveyor belt is in the moving state at the third history time, the fifth image is in one-to-one correspondence with the foreign object type, and the foreign object type is a type of the foreign object in the fifth image corresponding to the foreign object type.

Specifically, the training set in the data set constructed by the application further comprises a subset 2, wherein the subset 2 is 1400 images (fifth images) of the conveyor belt containing the foreign matters, which are acquired through the camera under the movement state of the conveyor belt (the speed of the conveyor belt is lower), the fifth images are required to be marked by using a Labelimg image marking tool, and the marking format adopts a YOLO format.

In this embodiment, the fifth image is a high-definition image of the conveyor belt containing the foreign matters collected by the camera under the motion state of the conveyor belt, and the improved YOLOX model is trained by using a huge amount of fifth images, so that a foreign matter detection model with higher foreign matter recognition capability is obtained, and the accuracy of detecting the foreign matters of the conveyor belt is improved.

In order to determine whether the foreign object threatens the transportation safety of the conveyor belt, in an alternative solution, the foreign object detection model is further used to mark a bounding box of the foreign object in the image, and is further used to determine a type of the foreign object in the image, and after step S203, the method further includes:

step S301, when the foreign object exists in the second image, determining whether the boundary frame of the foreign object is in a preset area or not under a pixel coordinate system;

step S302, acquiring the type of the foreign matter under the condition that the boundary box is in the preset area;

specifically, coordinates of four points of upper left, upper right, lower left and lower right in a preset area are manually marked in the second image, and a preset area frame is selected in the second image.

Step S303, when the type of the foreign object is a first preset type, generating first alarm information, where the first alarm information is used to represent information that the foreign object of the first preset type threatens the safety of the conveyor belt, and the first preset type at least includes one of the following: anchor rods and channel steel.

In this embodiment, as shown in fig. 6, when the foreign matter detection model determines that the conveyor belt does not have a foreign matter, the conveyor belt normally operates without warning, when the foreign matter detection model determines that the conveyor belt has a foreign matter, it determines whether a bounding box of the foreign matter in the second image is in a preset area (conveyor belt area), if not, it indicates that the conveyor belt is safe, the conveyor belt normally operates without warning, if so, it determines that the foreign matter is on the conveyor belt, and at this time, it determines whether the foreign matter is dangerous, wherein it determines whether the foreign matter is dangerous (determines whether the foreign matter is dangerous for conveying the conveyor belt), firstly obtains the type of the foreign matter, the conveyor belt for conveying coal mainly includes gangue, anchor rods, channel steel, and the like, wherein the anchor rods and the channel steel have high dangerousness, if the type of the foreign matter is the anchor rods and the channel steel, then generates first warning information, and timely informs a worker to interrupt the operation of the conveyor belt immediately.

In order to determine whether the foreign matter threatens the conveyor belt transportation safety, in another alternative, after step S302, the method further includes:

determining a foreign object area, which is an area of the bounding box of the foreign object, when the type of the foreign object is a second predetermined type, the second predetermined type including at least one of: gangue;

and generating second alarm information under the condition that the foreign matter area is larger than a preset area, wherein the second alarm information is used for representing the information of the second preset type of foreign matters threatening the safety of the conveyor belt.

In this embodiment, the conveyor belt for transporting coal mainly includes gangue, anchor rod, channel steel, etc., wherein the risk of gangue type foreign matter is affected by its size, as shown in fig. 6, in determining whether the foreign matter is dangerous, if the type of foreign matter is gangue, it is determined whether the foreign matter area is greater than the preset area, when the foreign matter area is greater than the preset area, it indicates that the size of gangue on the conveyor belt is greater, and the threat to the transportation safety of the conveyor belt can be generated, then a second alarm message is generated, and the staff is informed in time to interrupt the operation of the conveyor belt immediately.

According to the embodiment, the method comprises the steps of firstly restoring the low-definition image of the conveyor belt into the high-definition image through the removal model, then adopting the foreign matter detection model obtained by training the improved YOLOX model to identify whether the foreign matter exists in the high-definition image, so that the accuracy of detecting the foreign matter of the conveyor belt is improved, and the problem that the deep learning model in the prior art has poor recognition capability on the motion-blurred image foreign matter is solved because the low-definition image of the conveyor belt is subjected to the restoration definition treatment before the foreign matter recognition.

In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, the implementation procedure of the foreign matter detection method of the conveyor belt of the present application will be described in detail below with reference to specific embodiments.

The embodiment relates to a specific foreign matter detection method of a conveyor belt, which comprises the following steps:

step S1: acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment;

step S2: inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model and is used for restoring the image with the first definition into an image with a second definition, and the second definition is higher than the first definition;

Step S3: inputting the second image into a foreign object detection model, and determining whether a foreign object exists in the second image, wherein the foreign object detection model is a model obtained by training an improved YOLOX model, and is at least used for determining whether the foreign object exists in the image;

step S4: determining whether the bounding box of the foreign object is within a preset area under a pixel coordinate system when the foreign object exists in the second image;

step S5: acquiring the type of the foreign matter under the condition that the boundary box is in the preset area;

step S6: generating first alarm information under the condition that the type of the foreign matter is a first preset type, wherein the first alarm information is used for representing information that the foreign matter of the first preset type threatens the safety of the conveyor belt, and the first preset type at least comprises one of the following: the anchor rod and the channel steel; determining a foreign object area, which is an area of the bounding box of the foreign object, when the type of the foreign object is a second predetermined type, the second predetermined type including at least one of: gangue; and generating second alarm information under the condition that the foreign matter area is larger than a preset area, wherein the second alarm information is used for representing the information of the second preset type of foreign matters threatening the safety of the conveyor belt.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

The embodiment of the application also provides a foreign matter detection device of the conveyor belt, and it should be noted that the foreign matter detection device of the conveyor belt of the embodiment of the application can be used for executing the foreign matter detection method for the conveyor belt provided by the embodiment of the application. The device is used for realizing the above embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The following describes a foreign matter detection device of a conveyor belt provided in an embodiment of the present application.

Fig. 7 is a schematic view of a foreign matter detection device of a conveyor belt according to an embodiment of the present application. As shown in fig. 7, the apparatus includes:

A first acquiring unit 10, configured to acquire a first image, where the first image is an image of a first definition of a conveyor belt at a current time, and the conveyor belt is in a motion state at the current time;

A removal unit 20, configured to input the first image into a removal model to obtain a second image, where the removal model is a model obtained by training a U-Net network model, and the removal model is configured to restore the image with the first definition to an image with a second definition, where the second definition is higher than the first definition;

In order to improve accuracy of detection of foreign matters in the conveyor belt, in an alternative scheme, the device further comprises a second acquisition unit, a processing unit and a first training unit:

The second acquiring unit is configured to acquire a plurality of third images and a plurality of fourth images, the third images being images of the second definition of the conveyor belt at a first history time, the conveyor belt being in a stationary state at the first history time, the fourth images being images of the first definition of the conveyor belt at a second history time, the conveyor belt being in the moving state at the second history time;

in particular, the method comprises the steps of,

the processing unit is used for adding noise to each third image respectively to obtain various normal distribution noises, and the third images are in one-to-one correspondence with the normal distribution noises;

The first training unit is configured to train the U-Net network model using a plurality of sets of first training data, where each set of first training data in the plurality of sets of first training data includes: the normal distribution noise and a target image, wherein the target image is any one of all the fourth images;

specifically, based on the idea of the diffusion model, in the denoising stage, as shown in fig. 3, the training process of the U-Net network model is as follows: the U-Net network model distributes noise x normally _T And a target imageSplicing to obtain a restored image x _T-1 Then, the U-Net network model will restore the image x _T-1 Splicing with the target image, and continuously restoring until the U-Net network model restores the image x ₁ And splicing the target image to obtain a clear image, wherein the target image is one fourth image randomly selected from all the fourth images.

And the first training unit is further configured to stop training the U-Net network model until the loss function of the U-Net network model converges, and obtain the removal model.

A first determining unit 30, configured to input the second image into a foreign object detection model, determine whether a foreign object exists in the second image, where the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least configured to determine whether the foreign object exists in the image.

In order to improve accuracy of detection of foreign matters on the conveyor belt, in another alternative, the apparatus further includes:

a second training unit, configured to train the improved YOLOX model with a plurality of sets of second training data to obtain the foreign object detection model, where each set of second training data in the plurality of sets of second training data includes: a fifth image and a foreign object type corresponding to the fifth image, wherein the fifth image is an image of the second definition of the conveyor belt at a third history time, the conveyor belt is in the moving state at the third history time, the fifth image is in one-to-one correspondence with the foreign object type, and the foreign object type is a type of the foreign object in the fifth image corresponding to the foreign object type.

In order to determine whether the foreign matter threatens the transportation safety of the conveyor belt, in an alternative solution, the foreign matter detection model is further used for marking a bounding box of the foreign matter in the image, and is further used for determining the type of the foreign matter in the image, and the apparatus further comprises a second determining unit, a third acquiring unit and a first generating unit:

the second determining unit is configured to determine, in a pixel coordinate system, whether the bounding box of the foreign object is within a preset area when the foreign object is present in the second image;

the third obtaining unit is configured to obtain the type of the foreign object when the bounding box is in the preset area;

The first generating unit is configured to generate first alarm information when the type of the foreign object is a first preset type, where the first alarm information is used to characterize information that the foreign object of the first preset type threatens the safety of the conveyor belt, and the first preset type at least includes one of the following: anchor rods and channel steel.

In order to determine whether the foreign matter threatens the conveyor belt transportation safety, in another alternative, the above apparatus further includes a third determining unit and a second generating unit:

the third determining unit is configured to determine a foreign object area when the type of the foreign object is a second predetermined type, where the foreign object area is an area of the bounding box of the foreign object, and the second predetermined type includes at least one of: gangue;

The second generation unit is configured to generate second alarm information when the foreign object area is greater than a preset area, where the second alarm information is used to represent information that the foreign object of the second preset type threatens the safety of the conveyor belt.

Through the embodiment, the device firstly restores the low-definition image of the conveyor belt into the high-definition image through the removal model, then adopts the foreign matter detection model obtained by training the improved YOLOX model to identify whether the foreign matter exists in the high-definition image, thereby improving the accuracy of detecting the foreign matter of the conveyor belt.

The foreign matter detection device of the conveyor belt includes a processor and a memory, the first acquisition unit, the removal unit, the first determination unit, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions. The modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. One or more kernels can be arranged, and the problem that the deep learning model in the prior art has poor recognition capability on the image foreign matters with motion blur is solved by adjusting kernel parameters.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

The embodiment of the invention provides a computer readable storage medium, which comprises a stored program, wherein the program is controlled to control equipment where the computer readable storage medium is located to execute a foreign matter detection method of a conveyor belt.

Specifically, the foreign matter detection method of the conveyor belt includes:

Optionally, the foreign object detection model is further configured to label a bounding box of the foreign object in an image, and further configured to determine a type of the foreign object in the image, and after the second image is input into the foreign object detection model to determine whether the foreign object exists in the second image, the method further includes: determining whether the bounding box of the foreign object is within a preset area under a pixel coordinate system when the foreign object exists in the second image; acquiring the type of the foreign matter under the condition that the boundary box is in the preset area; generating first alarm information under the condition that the type of the foreign matter is a first preset type, wherein the first alarm information is used for representing information that the foreign matter of the first preset type threatens the safety of the conveyor belt, and the first preset type at least comprises one of the following: anchor rods and channel steel.

Optionally, after the type of the foreign matter is acquired, the method further includes: determining a foreign object area, which is an area of the bounding box of the foreign object, when the type of the foreign object is a second predetermined type, the second predetermined type including at least one of: gangue; and generating second alarm information under the condition that the foreign matter area is larger than a preset area, wherein the second alarm information is used for representing the information of the second preset type of foreign matters threatening the safety of the conveyor belt.

Optionally, the method further comprises: acquiring a plurality of third images and fourth images, wherein the third images are images of the second definition of the conveyor belt at a first history time, the conveyor belt is in a static state at the first history time, the fourth images are images of the first definition of the conveyor belt at a second history time, and the conveyor belt is in the moving state at the second history time; adding noise to each third image respectively to obtain multiple normal distribution noises, wherein the third images are in one-to-one correspondence with the normal distribution noises; training the U-Net network model using a plurality of sets of first training data, each set of first training data comprising: the normal distributed noise and the fourth image; and stopping training the U-Net network model until the loss function of the U-Net network model is converged, and obtaining the removal model.

Optionally, the method further comprises: training the improved YOLOX model by using a plurality of sets of second training data to obtain the foreign object detection model, wherein each set of second training data in the plurality of sets of second training data comprises: a fifth image and a foreign object type corresponding to the fifth image, wherein the fifth image is an image of the second definition of the conveyor belt at a third history time, the conveyor belt is in the moving state at the third history time, the fifth image is in one-to-one correspondence with the foreign object type, and the foreign object type is a type of the foreign object in the fifth image corresponding to the foreign object type.

The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute the foreign matter detection method of the conveyor belt.

The embodiment of the invention provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes at least the following steps when executing the program:

The device herein may be a server, PC, PAD, cell phone, etc.

The present application also provides a computer program product adapted to perform a program initialized with at least the following method steps when executed on a data processing device:

It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) The foreign matter detection method of the conveyor belt of the application comprises the following steps: acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment; inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model and is used for restoring the image with the first definition into an image with a second definition, and the second definition is higher than the first definition; and inputting the second image into a foreign object detection model, and determining whether the foreign object exists in the second image, wherein the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least used for determining whether the foreign object exists in the image. According to the method, firstly, a low-definition image of a conveyor belt is restored to a high-definition image through a removal model, then, a foreign matter detection model obtained by training an improved YOLOX model is adopted to identify whether foreign matters exist in the high-definition image, so that the accuracy of detecting the foreign matters of the conveyor belt is improved, and the problem that the deep learning model in the prior art is poor in recognition capability of the foreign matters of the image with motion blur is solved because the low-definition image of the conveyor belt is subjected to restoration definition treatment before the foreign matters are identified.

2) The foreign matter detection device of the conveyor belt of the present application includes: a first acquiring unit, configured to acquire a first image, where the first image is an image of a first definition of a conveyor belt at a current time, and the conveyor belt is in a motion state at the current time; a removal unit, configured to input the first image into a removal model to obtain a second image, where the removal model is a model obtained by training a U-Net network model, and the removal model is configured to restore the image with the first definition to an image with a second definition, where the second definition is higher than the first definition; and a first determination unit configured to input the second image into a foreign object detection model, determine whether or not a foreign object exists in the second image, wherein the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least used for determining whether or not the foreign object exists in the image. The device firstly restores the low-definition image of the conveyor belt into the high-definition image through the removal model, then adopts the foreign matter detection model obtained by training the improved YOLOX model to identify whether the foreign matters exist in the high-definition image, thereby improving the accuracy of the foreign matter detection of the conveyor belt.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of detecting foreign matter in a conveyor belt, the method comprising:

acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment;

inputting the first image into a removal model to obtain a second image, wherein the removal model is a model obtained by training a U-Net network model and is used for reducing the image with the first definition into an image with a second definition, and the second definition is higher than the first definition;

and inputting the second image into a foreign matter detection model, and determining whether the foreign matter exists in the second image, wherein the foreign matter detection model is a model obtained by training an improved YOLOX model, and is at least used for determining whether the foreign matter exists in the image.

2. The method according to claim 1, wherein the foreign object detection model is further used for marking a bounding box of the foreign object in an image, and is further used for determining a type of the foreign object in an image, and after inputting the second image into the foreign object detection model, determining whether the foreign object is present in the second image, the method further comprises:

determining, in a pixel coordinate system, whether the bounding box of the foreign object is within a preset region, in the case that the foreign object is present in the second image;

acquiring the type of the foreign matter under the condition that the boundary box is in the preset area;

generating first alarm information under the condition that the type of the foreign matter is a first preset type, wherein the first alarm information is used for representing information that the foreign matter of the first preset type threatens the safety of the conveyor belt, and the first preset type at least comprises one of the following: anchor rods and channel steel.

3. The method according to claim 2, wherein after the type of the foreign matter is acquired, the method further comprises:

determining a foreign object area, which is an area of the bounding box of the foreign object, in case the type of the foreign object is a second preset type, the second preset type comprising at least one of: gangue;

And under the condition that the foreign matter area is larger than a preset area, generating second alarm information, wherein the second alarm information is used for representing the information of the second preset type that the foreign matter threatens the safety of the conveyor belt.

4. The method according to claim 1, wherein the method further comprises:

acquiring a plurality of third images and a plurality of fourth images, wherein the third images are images of the second definition of the conveyor belt at a first historical moment, the conveyor belt is in a static state at the first historical moment, the fourth images are images of the first definition of the conveyor belt at a second historical moment, and the conveyor belt is in the motion state at the second historical moment;

adding noise to each third image respectively to obtain various normal distribution noises, wherein the third images are in one-to-one correspondence with the normal distribution noises;

5. The method according to claim 1, wherein the method further comprises:

and training the improved YOLOX model by using a plurality of sets of second training data to obtain the foreign object detection model, wherein each set of second training data in the plurality of sets of second training data comprises: the device comprises a fifth image and a foreign matter type corresponding to the fifth image, wherein the fifth image is an image with the second definition of the conveyor belt at a third historical moment, the conveyor belt is in the motion state at the third historical moment, the fifth image corresponds to the foreign matter type one by one, and the foreign matter type is the type of the foreign matter in the fifth image corresponding to the foreign matter type.

6. The method according to claim 1 or 5, wherein the YOLOX model after modification is a model obtained by adding a hole convolution layer between an up-sampling layer and a cross-stage layer in the YOLOX model.

7. A foreign matter detection device of a conveyor belt, characterized by comprising:

The first acquisition unit is used for acquiring a first image, wherein the first image is an image with first definition of a conveyor belt at the current moment, and the conveyor belt is in a motion state at the current moment;

the removing unit is used for inputting the first image into a removing model to obtain a second image, the removing model is a model obtained by training a U-Net network model, the removing model is used for reducing the image with the first definition into the image with the second definition, and the second definition is higher than the first definition;

the first determining unit inputs the second image into a foreign object detection model to determine whether the foreign object exists in the second image, wherein the foreign object detection model is a model obtained by training an improved YOLOX model, and the foreign object detection model is at least used for determining whether the foreign object exists in the image.

8. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program, when run, controls an apparatus in which the computer-readable storage medium is located to execute the foreign matter detection method of the conveyor belt according to any one of claims 1 to 6.

9. A processor for running a program, wherein the program runs to execute the foreign matter detection method of the conveyor belt according to any one of claims 1 to 6.

10. An electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a method for performing the foreign object detection of the conveyor belt of any one of claims 1 to 6.