CN115761642A - Image processing-based crushing operation monitoring method and system - Google Patents
Image processing-based crushing operation monitoring method and system Download PDFInfo
- Publication number
- CN115761642A CN115761642A CN202211470253.6A CN202211470253A CN115761642A CN 115761642 A CN115761642 A CN 115761642A CN 202211470253 A CN202211470253 A CN 202211470253A CN 115761642 A CN115761642 A CN 115761642A
- Authority
- CN
- China
- Prior art keywords
- feature map
- training
- crushing
- feature
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The application relates to the field of intelligent monitoring, and particularly discloses a crushing operation monitoring method and a system based on image processing. Further, whether large raw coal blocks different in size from the small raw coal blocks exist or not is detected based on global difference feature distribution, and then intelligent monitoring of the raw coal crushing effect is conducted. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Description
Technical Field
The present application relates to the field of intelligent monitoring, and more particularly, to a method and system for monitoring a crushing operation based on image processing.
Background
Coal is an important energy source related to the countryside, is generally buried in the ground deeply, and needs to be excavated when in use, and the raw coal excavated from the ground is mostly in the shape of blocks with irregular surfaces, which brings great inconvenience to subsequent transportation, storage and use.
The existing raw coal crusher generally adopts modes of striking, impacting, shearing, grinding and the like to crush raw coal, but the raw coal has different sizes, part of the raw coal has larger volume, and the crushed raw coal has different sizes from large raw coal to small raw coal after entering the raw coal crusher for crushing operation, so that the crushed raw coal needs to be secondarily crushed. At present, the actual size of raw coal is different in the mining process, and the crushing effect is different in each time, which brings difficulty to the operation control of crushing.
Therefore, an optimized crushing operation monitoring solution is desired.
Disclosure of Invention
The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides a crushing operation monitoring method and system based on image processing, and the method and system adopt an artificial intelligence monitoring technology based on machine learning to extract multi-scale implicit feature distribution information in a high-dimensional feature space of an image after raw coal crushing operation after dust removal processing, and express the crushing effect features of raw coal after crushing operation based on the difference features among the multi-scale raw coal crushed image features. Further, whether large raw coal blocks different in size from the small raw coal blocks exist or not is detected based on global difference feature distribution, and then intelligent monitoring of the raw coal crushing effect is conducted. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
According to one aspect of the present application, there is provided a method for weaving a high abrasion resistant textile fabric, comprising:
acquiring an image acquired by a camera after crushing operation;
passing the post-crushing image through a dust removal generator based on a countermeasure generation network to obtain a post-crushing image;
enabling the generated image after the crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a first scale feature map and a second scale feature map, wherein the first convolution neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolution neural network model uses a second cavity convolution kernel with a second cavity rate;
calculating a difference feature map between the first scale feature map and the second scale feature map;
expanding each feature matrix of the difference feature map along the channel dimension into a feature vector to obtain a plurality of local difference feature vectors;
passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector; and
and passing the global difference characteristic vector through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
In the image processing-based crushing job monitoring method, the countermeasure generation network includes a discriminator and a generator, wherein the passing the crushing job image through a dust removal generator of the countermeasure generation network to obtain a generated crushing job image includes: inputting the post-crushing-job image into the dust removal generator based on the countermeasure generation network to generate the post-crushing-job image by the generator of the countermeasure generation network through deconvolution coding.
In the method for monitoring a crushing operation based on image processing, the step of passing the generated image after the crushing operation through a dual-flow detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first scale feature map and a second scale feature map includes: performing convolution processing of a first convolution kernel on input data in forward transfer of layers using layers of the first convolution neural network: performing convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network is the first scale feature map, and the input of the first layer of the first convolutional neural network is the image after the generating of the crushing operation; and performing convolution processing of a second convolution kernel on the input data in forward pass of the layers using layers of the second convolutional neural network: performing convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; and the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the generated image after the crushing operation.
In the image processing-based crushing operation monitoring method, the calculating a difference feature map between the first scale feature map and the second scale feature map includes: calculating a difference characteristic diagram between the first scale characteristic diagram and the second scale characteristic diagram according to the following formula;
wherein the formula is:
wherein, F 1 Representing said first scale feature map, F 2 Representing said second scale feature map, F c A graph of the difference signature is represented,
indicating a difference by position.
In the image processing-based crushing operation monitoring method, the expanding each feature matrix of the difference feature map along the channel dimension into a feature vector to obtain a plurality of local difference feature vectors includes: and expanding each feature matrix of the differential feature map along the channel dimension into a feature vector along a row vector or a column vector to obtain the plurality of local differential feature vectors.
In the method for monitoring a crushing operation based on image processing, the passing the plurality of local differential feature vectors through a context encoder based on a converter to obtain a global differential feature vector includes: performing one-dimensional arrangement on the local differential feature vectors to obtain a global differential feature vector; calculating a product between the global differential feature vector and a transposed vector of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention correlation matrices; respectively carrying out standardization processing on each self-attention correlation matrix in the self-attention correlation matrixes to obtain a plurality of standardized self-attention correlation matrixes; obtaining a plurality of probability values by passing each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices through a Softmax classification function; weighting each differential feature vector in the local differential feature vectors by taking each probability value in the probability values as weight to obtain a plurality of context semantic differential feature vectors; and cascading the plurality of context semantic difference feature vectors to obtain the global difference feature vector.
In the method for monitoring a crushing operation based on image processing, the passing the global differential feature vector through a classifier to obtain a classification result includes: processing the global difference feature vector using the classifier to obtain a classification result with the following formula:
softmax{(M c ，B c )}，softmax{(M c ，B c ) | X }, wherein M c Weight matrix being a fully connected layer, B c And representing the bias vector of the fully connected layer, wherein X is the global differential feature vector.
In the image processing-based crushing operation monitoring method, the method further comprises the training step of: training the dual-stream detection network, the transformer-based context encoder, and the classifier; wherein the training step comprises: acquiring training data, wherein the training data comprises images after training crushing operation and a classification result of whether the raw coal crushing effect meets a preset standard; enabling the image after the training crushing operation to pass through a dust removal generator based on the countermeasure generation network to obtain an image after the training crushing operation; enabling the image after the training generation crushing operation to pass through a double-current detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a training first scale feature map and a training second scale feature map, wherein the first convolutional neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolutional neural network model uses a second cavity convolution kernel with a second cavity rate; calculating a training difference feature map between the training first scale feature map and the training second scale feature map; expanding each feature matrix of the training difference feature map along the channel dimension into a feature vector to obtain a plurality of training local difference feature vectors; passing the plurality of training local differential feature vectors through the converter-based context encoder to obtain training global differential feature vectors; and passing the training global difference feature vector through the classifier to obtain a classification loss function value; calculating a sequence pair sequence response rule intrinsic learning loss function value based on a distance between the training first scale feature map and the training second scale feature map; and computing a weighted sum of the classification loss function values and the sequence versus sequence response rule intrinsic learning loss function values as loss function values to train the dual-stream detection network, the converter-based context encoder, and the classifier.
In the image processing-based crushing operation monitoring method, the calculating a sequence pair sequence response rule intrinsic learning loss function value based on the distance between the training first scale feature map and the training second scale feature map includes: calculating a learning loss function value in the sequence-to-sequence response rule based on the distance between the training first scale feature map and the training second scale feature map according to the following formula; wherein the formula is:
wherein, V 1 And V 2 Respectively, the feature vectors obtained after the training first scale feature diagram and the training second scale feature diagram are expanded, and W is 1 And W 2 Respectively, the classifier is used for developing the training first scale feature map and the training second scale feature map to obtain a weight matrix of a feature vector, reLU (-) represents a ReLU activation function, sigmoid (-) represents a Sigmoid activation function,
representing matrix multiplication, d (·,) represents the euclidean distance between two vectors,
the values of the learned loss functions inherent to the sequence response rules are expressed.
According to another aspect of the present application, there is provided an image processing-based crushing operation monitoring system, comprising:
the image acquisition module is used for acquiring images acquired by the camera after the crushing operation;
a dust removal module for passing the post-crushing image through a dust removal generator based on a countermeasure generation network to obtain a post-crushing image;
the multi-scale convolution module is used for enabling the generated images after the crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a first scale feature map and a second scale feature map, wherein the first convolution neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolution neural network model uses a second cavity convolution kernel with a second cavity rate;
a difference module, configured to calculate a difference feature map between the first scale feature map and the second scale feature map;
the feature vector generation module is used for expanding each feature matrix of the differential feature map along the channel dimension into feature vectors to obtain a plurality of local differential feature vectors;
a context encoding module for passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector; and
and the classification result generation module is used for enabling the global difference characteristic vector to pass through a classifier to obtain a classification result, and the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
According to still another aspect of the present application, there is provided an electronic apparatus including: a processor; and a memory in which computer program instructions are stored, which, when executed by the processor, cause the processor to perform the image processing based crushing job monitoring method as described above.
According to yet another aspect of the present application, a computer readable medium is provided, having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the image processing based crushing job monitoring method as described above.
Compared with the prior art, the crushing operation monitoring method and the crushing operation monitoring system based on the image processing have the advantages that the artificial intelligence monitoring technology based on machine learning is adopted, the image after the raw coal crushing operation is subjected to dust removal processing, the multi-scale implicit feature distribution information of the image in the high-dimensional feature space is extracted, and the crushing effect features of the raw coal after the crushing operation are expressed based on the difference features among the multi-scale raw coal crushed image features. And further, detecting whether large blocks of raw coal with sizes different from those of the small blocks of raw coal exist or not based on global difference feature distribution, and then intelligently monitoring the crushing effect of the raw coal. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control is performed when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 is an application scenario diagram of a crushing operation monitoring method based on image processing according to an embodiment of the present application;
FIG. 2 is a flow chart of a method for monitoring a crushing operation based on image processing according to an embodiment of the present application;
FIG. 3 is a flow chart of a training phase in a method for monitoring a crushing operation based on image processing according to an embodiment of the present application;
fig. 4 is a schematic diagram of an architecture of a crushing operation monitoring method based on image processing according to an embodiment of the present application;
FIG. 5 is a schematic diagram of an architecture of a training phase in a method for monitoring a crushing operation based on image processing according to an embodiment of the present application;
fig. 6 is a flowchart of a context encoding process in the image processing-based crushing operation monitoring method according to the embodiment of the present application;
FIG. 7 is a block diagram of an image processing based crushing job monitoring system according to an embodiment of the present application;
fig. 8 is a block diagram of an electronic device according to an embodiment of the application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.
Overview of a scene
As mentioned in the background of the invention, coal is an important energy source related to the national civilian life, and is generally buried deep in the ground, and when in use, the coal needs to be excavated first, and the raw coal excavated from the ground is mostly in the form of irregular blocks, which brings great inconvenience to subsequent transportation, storage and use.
The existing raw coal crusher generally adopts modes of striking, impacting, shearing, grinding and the like to crush raw coal, but the raw coal has different sizes, part of the raw coal has larger volume, and the crushed raw coal has different sizes from large raw coal to small raw coal after entering the raw coal crusher for crushing operation, so that the crushed raw coal needs to be secondarily crushed. At present, the actual size of raw coal is different in the process of mining, and the crushing effect is different from time to time, which brings difficulty to the operation control of crushing. Therefore, an optimized crushing operation monitoring solution is desired.
At present, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, speech signal processing, and the like. In addition, deep learning and neural networks also exhibit a level close to or even exceeding that of humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.
In recent years, deep learning and development of neural networks provide new solutions and schemes for intelligent monitoring of crushing operation.
Correspondingly, the monitoring of the crushing condition of the raw coal can be carried out by monitoring the raw coal image after the crushing operation, namely, a remote centralized control terminal is arranged in a centralized control center, and the transferring and crushing operation conditions of the whole system are monitored in real time through image data acquired by a camera. However, it is considered that the acquired image data may be blurred due to interference of environmental dust due to the fact that information in the image acquired by the camera after the crushing operation is more and is influenced by the raw coal mining environment, and thus, monitoring of the raw coal crushing effect is difficult.
Based on this, in the technical scheme of this application, adopt the artificial intelligence monitoring technology based on machine learning, remove the dust processing with the image after the raw coal crushing operation and extract its multi-scale implicit feature distribution information in high dimension feature space to show the crushing effect characteristic of the raw coal after the crushing operation based on the difference characteristic between the broken image characteristic of multi-scale raw coal. Further, considering that there may be large raw coal in a partial region in the image, whether there is large raw coal having a size different from that of the small raw coal is detected based on global differential feature distribution, and intelligent monitoring of the raw coal crushing effect is performed. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Specifically, in the technical scheme of the application, firstly, images after crushing operation are collected through a camera. Then, considering that in the image after the crushing operation, because a large amount of environmental small particles such as dust exist in a mining place of raw coal and the like affect the definition of the image after the crushing operation, the implicit characteristics of the image after the crushing operation become fuzzy due to the interference of external environmental factors during characteristic extraction, and the accuracy of judging the crushing effect of the raw coal is further reduced. Therefore, in the technical scheme of the application, the image definition is enhanced by the dust removing generator based on the countermeasure generation network before the feature extraction. Specifically, the post-crushing-job image is input to the dust removal generator based on the countermeasure generation network to generate the post-crushing-job image by deconvolution coding by the generator of the countermeasure generation network. In particular, the generation network based on the countermeasure comprises a discriminator and a generator, wherein the generator is used for generating the image after dust removal, the discriminator is used for calculating the difference between the image after dust removal and the real image, and the network parameters of the generator are updated through a gradient descent direction propagation algorithm to obtain the generator with the dust removal function.
Then, the convolutional neural network model having excellent performance in the aspect of implicit feature extraction of the image is used for carrying out feature mining on the image after the fragmentation operation is generated. In particular, when the raw coal crushing effect is judged through the generated image after the crushing operation, since the crushed size and the crushing uniformity distribution information of the raw coal in the generated image after the crushing operation are the key for detecting the crushing effect, the generated image after the crushing operation is processed through a double-current detection network comprising a first convolution neural network model and a second convolution neural network model to extract the local multi-scale implicit feature distribution information in the generated image after the crushing operation, so that a first scale feature map and a second scale feature map are obtained. It is worth mentioning that here, the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate.
Then, considering that similar feature distribution information exists in the image feature information of different scales if the crushing effect of the raw coal meets a preset standard in the hidden feature information of the crushed raw coal image. Therefore, in the technical solution of the present application, a difference feature map between the first scale feature map and the second scale feature map is further calculated to represent the crushing effect feature of the raw coal after the crushing operation based on the difference feature between the multi-scale raw coal crushed image features.
Further, it is considered that there may be local features in the image feature distribution after the raw coal is crushed, and feature information of large raw coal exists, which may not be able to detect the crushing effect of the raw coal well in performing local feature mining comparison. Therefore, each feature matrix of the differential feature map along the channel dimension is further expanded into feature vectors along a row vector or a column vector to obtain a plurality of local differential feature vectors, and the local differential feature vectors are encoded in a context encoder based on a converter to extract the relevance features of the local raw coal crushing effect features on the channel dimension based on global feature distribution, so as to obtain a global differential feature vector.
Then, the global differential feature vector with the global raw coal crushing effect feature is classified in a classifier to obtain a classification result used for indicating whether the raw coal crushing effect meets a preset standard or not, so that the intelligent monitoring of the raw coal crushing effect is carried out.
Particularly, in the technical solution of the present application, when calculating the difference feature map between the first scale feature map and the second scale feature map, since the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate, it is desirable to improve the intrinsic response relationship between feature distributions between the first scale feature map and the second scale feature map under feature extraction of hole convolution kernels having different hole rates, so as to improve the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map.
Therefore, the applicant of the present application regards the first scale feature map and the second scale feature map as a feature value sequence, and calculates a sequence-to-sequence response rule internalization learning loss function, expressed as:
V 1 and V 2 Respectively are the feature vectors obtained after the first scale feature map and the second scale feature map are unfolded, and W is 1 And W 2 Classifier pair V respectively 1 And V 2 The weight matrix of (a).
That is, for the feature vector V by the classifier 1 And a feature vector V 2 To obtain enhanced discriminative power between sequences of feature vectors. By training the network with this loss function, restoration of causal features with better discrimination between sequences with responsiveness can be achieved to the feature vector V 1 And a feature vector V 2 The cause-result response rule of each other is internalized and learned, thereby enhancing the feature vector V as a feature sequence 1 And a feature vector V 2 The internal response relation between the first scale feature map and the second scale feature map improves the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map, and further improves the classification accuracy. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control is performed when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Based on this, the present application provides a crushing operation monitoring method based on image processing, which includes: acquiring an image acquired by a camera after crushing operation; passing the post-crushing image through a dust removal generator based on a countermeasure generation network to obtain a post-crushing image; enabling the generated image after the crushing operation to pass through a double-current detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a first scale characteristic diagram and a second scale characteristic diagram, wherein the first convolutional neural network model uses a first hole convolution kernel with a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel with a second hole rate; calculating a difference feature map between the first scale feature map and the second scale feature map; expanding each feature matrix of the difference feature map along the channel dimension into a feature vector to obtain a plurality of local difference feature vectors; passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector; and enabling the global differential feature vector to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
Fig. 1 is an application scenario diagram of a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 1, in this application scenario, a post-crushing operation image is acquired by a camera (e.g., C as illustrated in fig. 1). Then, the image is input into a server (for example, S in fig. 1) deployed with a crushing job monitoring algorithm based on image processing, wherein the server can process the input image by the crushing job monitoring algorithm based on image processing to generate a classification result for indicating whether the raw coal crushing effect meets a predetermined standard.
Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.
Exemplary method
Fig. 2 is a flowchart of a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 2, the method for monitoring a crushing operation based on image processing according to the embodiment of the present application includes: s110, acquiring an image after crushing operation collected by a camera; s120, enabling the image after the crushing operation to pass through a dust removal generator based on a countermeasure generation network to obtain an image after the crushing operation; s130, enabling the generated image after the crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a first scale feature map and a second scale feature map, wherein the first convolution neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolution neural network model uses a second cavity convolution kernel with a second cavity rate; s140, calculating a difference feature map between the first scale feature map and the second scale feature map; s150, expanding each feature matrix of the difference feature map along the channel dimension into feature vectors to obtain a plurality of local difference feature vectors; s160, passing the local differential feature vectors through a context encoder based on a converter to obtain a global differential feature vector; and S170, enabling the global difference characteristic vector to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
Fig. 4 is a schematic configuration diagram of a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 4, in the network architecture, in the inference phase, a camera is used to collect an image after a crushing operation; then the image after the crushing operation is processed through a dust removing generator based on a countermeasure generation network to obtain an image after the crushing operation is generated; then, enabling the generated image after the crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a first scale feature map and a second scale feature map, wherein the first convolution neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolution neural network model uses a second cavity convolution kernel with a second cavity rate; then, calculating a difference feature map between the first scale feature map and the second scale feature map; expanding each feature matrix of the difference feature map along the channel dimension into a feature vector to obtain a plurality of local difference feature vectors; then, passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector; and then, the global difference feature vector is processed by a classifier to obtain a classification result, and the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
More specifically, in step S110, a post-crushing operation image captured by a camera is acquired. The monitoring of the crushing condition of the raw coal can be carried out by monitoring the raw coal image after the crushing operation, so that in the technical scheme of the application, the remote centralized control terminal is arranged in the centralized control center, and the transferring and crushing operation conditions of the whole system are monitored in real time through image data acquired by the camera.
More specifically, in step S120, the post-crushing-job image is passed through a dust removal generator based on a countermeasure generation network to obtain a post-crushing-job image. Considering that in the image after the crushing operation, because a large amount of environmental small particles such as dust exist in a mining place of raw coal and influence the definition of the image after the crushing operation, the implicit characteristics of the image after the crushing operation become fuzzy due to the interference of external environmental factors during characteristic extraction, and the accuracy of judging the crushing effect of the raw coal is further reduced. Therefore, in the technical scheme of the application, before feature extraction, the image definition is enhanced through a dust removing generator based on a confrontation generation network. Specifically, the post-crushing-job image is input to the dust removal generator based on the countermeasure generation network to generate the post-crushing-job image by deconvolution coding by the generator of the countermeasure generation network. In particular, the generation network based on the countermeasure comprises a discriminator and a generator, wherein the generator is used for generating an image after dust removal, the discriminator is used for calculating the difference between the image after dust removal and a real image, and network parameters of the generator are updated through a gradient descent direction propagation algorithm to obtain the generator with the dust removal function.
More specifically, in step S130, the generated post-crushing-operation image is passed through a dual-flow detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first scale feature map and a second scale feature map, where the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate. And performing feature mining on the image after the fragmentation operation by using a convolution neural network model with excellent performance in the aspect of implicit feature extraction of the image. In particular, when the crushing effect of the raw coal is judged through the generated image after the crushing operation, since the information about the crushed size and the crushing uniformity distribution of the raw coal in the generated image after the crushing operation is a key for detecting the crushing effect, the generated image after the crushing operation is processed through a dual-flow detection network comprising a first convolutional neural network model and a second convolutional neural network model to extract the local multi-scale implicit feature distribution information in the generated image after the crushing operation, so as to obtain a first-scale feature map and a second-scale feature map. It is worth mentioning that here, the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate. More specifically, the step of passing the generated image after the crushing operation through a dual-flow detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first scale feature map and a second scale feature map includes: performing convolution processing of a first convolution kernel on input data in forward transfer of layers using layers of the first convolution neural network: carrying out convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network is the first scale feature map, and the input of the first layer of the first convolutional neural network is the generated image after the crushing operation; and performing convolution processing of a second convolution kernel on the input data in forward transfer of layers using layers of the second convolutional neural network: performing convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; and the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the generated image after the crushing operation. Wherein the first convolutional neural network and the second convolutional neural network comprise a plurality of neural network layers cascaded with each other, wherein each neural network layer comprises a convolutional layer, a pooling layer, and an activation layer.
More specifically, in step S140, a difference feature map between the first scale feature map and the second scale feature map is calculated. Considering that in the hidden feature information of the crushed raw coal image, if the crushing effect of the raw coal meets a preset standard, similar feature distribution information exists in the image feature information of different scales. Therefore, in the technical solution of the present application, a difference feature map between the first scale feature map and the second scale feature map is further calculated to represent the crushing effect feature of the raw coal after the crushing operation based on the difference feature between the multi-scale raw coal crushed image features. In a specific example of the present application, a difference feature map between the first scale feature map and the second scale feature map is calculated in the following formula; wherein the formula is:
wherein, F 1 Representing said first scale feature map, F 2 Representing said second scale feature map, F c A graph of the difference signature is represented,
indicating a difference by position.
More specifically, in step S150, each feature matrix along the channel dimension of the differential feature map is expanded into a feature vector to obtain a plurality of local differential feature vectors. Considering that there may be local features in the image feature distribution after the raw coal is broken, feature information of large raw coal may exist, which may not be able to detect the breaking effect of the raw coal well in performing local feature mining comparison. Therefore, in a specific example of the present application, each feature matrix along the channel dimension of the differential feature map is further expanded as a feature vector along a row vector or a column vector to obtain the plurality of local differential feature vectors.
More specifically, in step S160, the plurality of local differential feature vectors are passed through a converter-based context encoder to obtain a global differential feature vector. Namely, the local differential feature vectors are encoded in a context encoder based on a converter, so as to extract the relevance feature of the local raw coal crushing effect features on the channel dimension based on the global feature distribution, thereby obtaining the global differential feature vector. More specifically, the passing the plurality of local differential feature vectors through a context encoder based on a converter to obtain a global differential feature vector includes: performing one-dimensional arrangement on the local differential feature vectors to obtain a global differential feature vector; calculating a product between the global differential feature vector and a transposed vector of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention correlation matrices; respectively carrying out standardization processing on each self-attention correlation matrix in the self-attention correlation matrixes to obtain a plurality of standardized self-attention correlation matrixes; obtaining a plurality of probability values by passing each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices through a Softmax classification function; weighting each differential feature vector in the local differential feature vectors by taking each probability value in the probability values as a weight to obtain a plurality of context semantic differential feature vectors; and cascading the plurality of context semantic difference feature vectors to obtain the global difference feature vector.
Fig. 6 is a flowchart of a context encoding process in a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 6, in the context coding process, the following steps are included: s310, performing one-dimensional arrangement on the local differential feature vectors to obtain a global differential feature vector; s320, calculating the product between the global differential feature vector and the transposed vector of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention correlation matrixes; s330, respectively carrying out standardization processing on each self-attention correlation matrix in the self-attention correlation matrixes to obtain a plurality of standardized self-attention correlation matrixes; s340, obtaining a plurality of probability values by passing each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices through a Softmax classification function; s350, weighting each differential feature vector in the local differential feature vectors by taking each probability value in the probability values as a weight to obtain a plurality of context semantic differential feature vectors; s360, cascading the plurality of context semantic difference feature vectors to obtain the global difference feature vector.
More specifically, in step S170, the global difference feature vector is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether the raw coal crushing effect meets a predetermined criterion. It should be understood that the global differential feature vector with global raw coal crushing effect features is classified by a classifier to obtain a classification result indicating whether the raw coal crushing effect meets a predetermined standard, so as to perform intelligent monitoring of the raw coal crushing effect. In one particular example of the present application, the classifier includes a plurality of fully-connected layers and a Softmax layer cascaded with a last fully-connected layer of the plurality of fully-connected layers. In the classification processing of the classifier, performing multiple times of full-connection coding on the global differential feature vector by using multiple full-connection layers of the classifier to obtain a coded classification feature vector; further, inputting the encoded classification feature vector into a Softmax layer of the classifier, namely, classifying the encoded classification feature vector by using the Softmax classification function to obtain a classification result indicating whether the raw coal crushing effect meets a predetermined standard, more specifically, passing the global differential feature vector through the classifier to obtain the classification result, comprises: processing the global difference feature vector using the classifier to obtain a classification result with the following formula:
softmax{(M c ，B c )}，softmax{(M c ，B c ) I X, wherein M c Is a weight matrix of the fully-connected layer,B c and representing the bias vector of the fully connected layer, wherein X is the global differential feature vector.
It will be appreciated that the dual stream detection network, the transformer based context encoder and the classifier need to be trained prior to encoding using the neural network described above. That is, in the image processing-based crushing operation monitoring method of the present application, a training module is included for training the dual-stream detection network, the converter-based context encoder, and the classifier.
Fig. 3 is a flowchart of a training phase in a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 3, the method for monitoring a crushing operation based on image processing according to the embodiment of the present application includes: s210, acquiring training data, wherein the training data comprises images after training crushing operation and a classification result of whether the raw coal crushing effect meets a preset standard; s220, enabling the image after the training crushing operation to pass through the dust removal generator based on the countermeasure generation network to obtain an image after the training generation crushing operation; s230, enabling the images after the training and generating crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a training first scale feature map and a training second scale feature map, wherein the first convolution neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolution neural network model uses a second cavity convolution kernel with a second cavity rate; s240, calculating a training difference feature map between the training first scale feature map and the training second scale feature map; s250, expanding each feature matrix of the training differential feature map along the channel dimension into feature vectors to obtain a plurality of training local differential feature vectors; s260, passing the training local differential feature vectors through the context encoder based on the converter to obtain a training global differential feature vector; and S270, passing the training global difference feature vector through the classifier to obtain a classification loss function value; s280, calculating a sequence pair sequence response rule intrinsic learning loss function value based on the distance between the training first scale feature map and the training second scale feature map; and, S290, computing a weighted sum of the classification loss function values and the sequence-to-sequence response rule intrinsic learning loss function values as loss function values to train the dual-stream detection network, the converter-based context encoder, and the classifier.
Fig. 5 is a schematic diagram of an architecture of a training phase in a crushing operation monitoring method based on image processing according to an embodiment of the present application. As shown in fig. 5, in the network architecture, in the training phase, training data is first obtained, where the training data includes images after training crushing operation and a classification result indicating whether the raw coal crushing effect meets a predetermined criterion; then the image after the training crushing operation passes through the dust removing generator based on the countermeasure generation network to obtain an image after the training crushing operation; enabling the image after the training generation crushing operation to pass through a double-current detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a training first scale feature map and a training second scale feature map, wherein the first convolutional neural network model uses a first cavity convolution kernel with a first cavity rate, and the second convolutional neural network model uses a second cavity convolution kernel with a second cavity rate; then, calculating a training difference feature map between the training first scale feature map and the training second scale feature map; expanding each feature matrix of the training differential feature map along the channel dimension into a feature vector to obtain a plurality of training local differential feature vectors; then, the training local differential feature vectors are processed by the context encoder based on the converter to obtain a training global differential feature vector; passing the training global differential feature vector through the classifier to obtain a classification loss function value; further, calculating a sequence pair sequence response rule intrinsic learning loss function value based on a distance between the training first scale feature map and the training second scale feature map; computing a weighted sum of the classification loss function values and the sequence versus sequence response rules intrinsic learning loss function values as loss function values to train the dual-stream detection network, the converter-based context encoder, and the classifier.
In particular, in the technical solution of the present application, when calculating the difference feature map between the first scale feature map and the second scale feature map, since the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate, it is desirable to improve the intrinsic response relationship between the feature distributions between the first scale feature map and the second scale feature map under the feature extraction of the hole convolution kernels having different hole rates, so as to improve the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map.
Therefore, the applicant of the present application regards the first scale feature map and the second scale feature map as a feature value sequence, and calculates a sequence-to-sequence response rule internalization learning loss function, expressed as:
wherein, V 1 And V 2 Respectively are the feature vectors obtained after the training first scale feature map and the training second scale feature map are unfolded, and W is 1 And W 2 The weight matrixes of the feature vectors obtained by the classifier after the training first scale feature map and the training second scale feature map are expanded respectively, reLU (-) represents a ReLU activation function, sigmoid (-) represents a Sigmoid activation function,
representing matrix multiplication, d (·,) represents the euclidean distance between two vectors,
and (3) a learning loss function value intrinsic to the sequence response rule is represented. That is, for the feature vector V by the classifier 1 And a feature vector V 2 To obtain enhanced discriminative power between sequences of feature vectors. By training the network with this loss function, restoration of causal features with better distinctiveness between sequences with responsiveness can be achieved to the feature vector V 1 And a feature vector V 2 Is subjected to internalized learning, thereby enhancing the feature vector V as a feature sequence 1 And a feature vector V 2 The internal response relation between the first scale feature map and the second scale feature map improves the calculation accuracy of the difference feature map between the first scale feature map and the second scale feature map, and further improves the classification accuracy. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
In summary, the image processing-based crushing operation monitoring method based on the embodiment of the application is clarified, and by adopting the machine learning-based artificial intelligence monitoring technology, the image after the raw coal crushing operation is subjected to dust removal processing, multi-scale implicit feature distribution information of the image in a high-dimensional feature space is extracted, and the crushing effect features of the raw coal after the crushing operation are expressed based on the difference features among the multi-scale raw coal crushed image features. And further, detecting whether large blocks of raw coal with sizes different from those of the small blocks of raw coal exist or not based on global difference feature distribution, and then intelligently monitoring the crushing effect of the raw coal. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Exemplary System
Fig. 7 is a block diagram of an image processing based crushing operation monitoring system according to an embodiment of the present application. As shown in fig. 7, the image processing-based crushing work monitoring system 300 according to the embodiment of the present application includes: an image acquisition module 310; a dust removal module 320; a multi-scale convolution module 330; a difference module 340; a feature vector generation module 350; a context encoding module 360; and a classification result generation module 370.
The image obtaining module 310 is configured to obtain an image after the crushing operation, which is collected by a camera; the dust removing module 320 is used for enabling the image after the crushing operation to pass through a dust removing generator based on a countermeasure generation network to obtain an image after the crushing operation is generated; the multi-scale convolution module 330 is configured to pass the generated image after the crushing operation through a dual-flow detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first scale feature map and a second scale feature map, where the first convolutional neural network model uses a first hole convolution kernel having a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel having a second hole rate; the difference module 340 is configured to calculate a difference feature map between the first scale feature map and the second scale feature map; the feature vector generation module 350 is configured to expand each feature matrix of the differential feature map along the channel dimension into a feature vector to obtain a plurality of local differential feature vectors; the context encoding module 360 is configured to pass the local differential feature vectors through a context encoder based on a converter to obtain a global differential feature vector; and the classification result generating module 370 is configured to pass the global difference feature vector through a classifier to obtain a classification result, where the classification result is used to indicate whether the raw coal crushing effect meets a predetermined criterion.
In one example, in the image processing-based crushing operation monitoring system 300, the dust removing module 320 includes: the countermeasure generation network comprises a discriminator and a generator, wherein the step of passing the post-crushing-operation image through a dust removal generator of the countermeasure generation network to obtain a post-crushing-operation image comprises the following steps: inputting the post-crushing-operation image into the dust removal generator based on the countermeasure generation network to generate the post-crushing-operation image through deconvolution coding by the generator of the countermeasure generation network.
In one example, in the image processing-based crushing operation monitoring system 300, the multi-scale convolution module 330 includes: performing convolution processing of a first convolution kernel on input data in forward transfer of layers using layers of the first convolution neural network: performing convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network is the first scale feature map, and the input of the first layer of the first convolutional neural network is the generated image after the crushing operation; and performing convolution processing of a second convolution kernel on the input data in forward pass of the layers using layers of the second convolutional neural network: carrying out convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; and the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the generated image after the crushing operation.
In one example, in the image processing-based crushing operation monitoring system 300, the difference module 340 includes: calculating a difference feature map between the first scale feature map and the second scale feature map according to the following formula; wherein the formula is:
wherein, F 1 Representing said first scale feature map, F 2 Representing said second scale feature map, F c A graph of the difference signature is represented,
indicating a difference by position.
In one example, in the image processing-based crushing operation monitoring system 300, the feature vector generating module 350 includes: and expanding each feature matrix of the differential feature map along the channel dimension into a feature vector along a row vector or a column vector to obtain the plurality of local differential feature vectors.
In one example, in the image processing-based crushing job monitoring system 300, the context encoding module 360 includes: performing one-dimensional arrangement on the local differential feature vectors to obtain a global differential feature vector; calculating a product between the global differential feature vector and a transposed vector of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention correlation matrices; respectively carrying out standardization processing on each self-attention correlation matrix in the self-attention correlation matrixes to obtain a plurality of standardized self-attention correlation matrixes; obtaining a plurality of probability values by passing each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices through a Softmax classification function; weighting each differential feature vector in the local differential feature vectors by taking each probability value in the probability values as weight to obtain a plurality of context semantic differential feature vectors; and cascading the plurality of context semantic difference feature vectors to obtain the global difference feature vector.
In one example, in the image processing-based crushing work monitoring system 300, the classification result generating module 370 includes: processing the global difference feature vector using the classifier to obtain a classification result with the following formula:
softmax{(M c ，B c )}，softmax{(M c ，B c ) | X }, wherein M c Weight matrix being a fully connected layer, B c And representing the bias vector of the fully connected layer, wherein X is the global differential feature vector.
In summary, the image processing-based crushing operation monitoring system based on the embodiment of the application is illustrated, and by adopting the machine learning-based artificial intelligence monitoring technology, the image after the raw coal crushing operation is subjected to dust removal processing, multi-scale implicit feature distribution information of the image in a high-dimensional feature space is extracted, and the crushing effect feature of the raw coal after the crushing operation is represented based on the difference feature between the features of the multi-scale raw coal crushed image. Further, whether large raw coal blocks different in size from the small raw coal blocks exist or not is detected based on global difference feature distribution, and then intelligent monitoring of the raw coal crushing effect is conducted. Therefore, the crushing effect of the raw coal can be monitored in real time according to actual conditions, so that crushing control can be carried out when the crushing effect of the raw coal does not meet the preset standard, and the effect of the raw coal crushing operation is improved.
Exemplary electronic device
Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 8.
FIG. 8 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.
As shown in fig. 8, the electronic device 10 includes one or more processors 11 and memory 12.
The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.
In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
The input device 13 may include, for example, a keyboard, a mouse, and the like.
The output device 14 can output various information including the classification result to the outside. The output devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.
Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 8, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the functions of the image processing based crushing job monitoring method according to the various embodiments of the present application described in the "exemplary methods" section of this specification, above.
The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present application may also be a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps in the functions in the image processing based crushing job monitoring method according to various embodiments of the present application described in the "exemplary methods" section above in this specification.
The computer readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The basic principles of the present application have been described above with reference to specific embodiments, but it should be noted that advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and the advantages, effects, etc. must not be considered to be possessed by various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.
The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. As used herein, the words "or" and "refer to, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It should also be noted that in the devices, apparatuses, and methods of the present application, each component or step can be decomposed and/or re-combined. These decompositions and/or recombinations should be considered as equivalents of the present application.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (10)
1. A crushing operation monitoring method based on image processing is characterized by comprising the following steps:
acquiring an image acquired by a camera after crushing operation;
passing the post-crushing work image through a dust removal generator based on a countermeasure generation network to obtain a post-crushing work image;
enabling the generated image after the crushing operation to pass through a double-current detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a first scale characteristic diagram and a second scale characteristic diagram, wherein the first convolutional neural network model uses a first hole convolution kernel with a first hole rate, and the second convolutional neural network model uses a second hole convolution kernel with a second hole rate;
calculating a difference feature map between the first scale feature map and the second scale feature map;
expanding each feature matrix of the difference feature map along the channel dimension into a feature vector to obtain a plurality of local difference feature vectors;
passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector; and
and passing the global differential feature vector through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
2. The image-processing-based crushing job monitoring method according to claim 1, wherein the countermeasure generation network comprises a discriminator and a generator, wherein the passing the crushing job image through a dust removal generator of the countermeasure generation network to obtain a generated crushing job image comprises: inputting the post-crushing-operation image into the dust removal generator based on the countermeasure generation network to generate the post-crushing-operation image through deconvolution coding by the generator of the countermeasure generation network.
3. The image-processing-based crushing operation monitoring method according to claim 2, wherein the step of passing the generated crushing operation image through a dual-flow detection network including a first convolutional neural network model and a second convolutional neural network model to obtain a first scale feature map and a second scale feature map comprises:
performing convolution processing of a first convolution kernel on input data in forward transfer of layers using layers of the first convolution neural network: performing convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network is the first scale feature map, and the input of the first layer of the first convolutional neural network is the generated image after the crushing operation; and
performing convolution processing of a second convolution kernel on input data in forward pass of layers using layers of the second convolutional neural network: carrying out convolution processing on input data to obtain a convolution characteristic diagram; performing pooling processing based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; and the output of the last layer of the second convolutional neural network is the second scale feature map, and the input of the first layer of the second convolutional neural network is the generated image after the crushing operation.
4. The image processing-based crushing operation monitoring method according to claim 3, wherein the calculating of the differential feature map between the first scale feature map and the second scale feature map comprises: calculating a difference feature map between the first scale feature map and the second scale feature map according to the following formula;
wherein the formula is:
wherein, F 1 Representing said first scale feature map, F 2 Representing said second scale feature map, F c Presentation instrumentThe difference characteristic diagram is described above in detail,
indicating a difference by position.
5. The image processing-based crushing operation monitoring method according to claim 4, wherein the expanding each feature matrix along the channel dimension of the differential feature map into feature vectors to obtain a plurality of local differential feature vectors comprises:
and expanding each feature matrix of the differential feature map along the channel dimension into a feature vector along a row vector or a column vector to obtain the plurality of local differential feature vectors.
6. The image-processing-based crushing job monitoring method according to claim 5, wherein passing the plurality of local differential feature vectors through a converter-based context encoder to obtain a global differential feature vector comprises:
performing one-dimensional arrangement on the local differential feature vectors to obtain a global differential feature vector;
calculating a product between the global differential feature vector and a transposed vector of each differential feature vector in the plurality of local differential feature vectors to obtain a plurality of self-attention correlation matrices;
respectively carrying out standardization processing on each self-attention correlation matrix in the self-attention correlation matrixes to obtain a plurality of standardized self-attention correlation matrixes;
obtaining a plurality of probability values by passing each normalized self-attention correlation matrix in the plurality of normalized self-attention correlation matrices through a Softmax classification function;
weighting each differential feature vector in the local differential feature vectors by taking each probability value in the probability values as a weight to obtain a plurality of context semantic differential feature vectors;
and cascading the plurality of context semantic difference feature vectors to obtain the global difference feature vector.
7. The image processing-based crushing operation monitoring method according to claim 6, wherein the passing the global differential feature vector through a classifier to obtain a classification result comprises: processing the global difference feature vector using the classifier to obtain a classification result with the following formula:
softmax{(M c ,B c )}，softmax{(M c ,B c ) I X, wherein M c Weight matrix being a fully connected layer, B c And representing the bias vector of the fully connected layer, wherein X is the global differential feature vector.
8. The image processing-based crushing operation monitoring method according to claim 7, further comprising the training step of: training the dual-stream detection network, the transformer-based context encoder, and the classifier;
wherein the training step comprises:
acquiring training data, wherein the training data comprise images after training crushing operation and a classification result of whether the raw coal crushing effect meets a preset standard;
enabling the image after the training crushing operation to pass through a dust removal generator based on the confrontation generation network to obtain an image after the training crushing operation;
enabling the images after the training and generating crushing operation to pass through a double-current detection network comprising a first convolutional neural network model and a second convolutional neural network model to obtain a training first scale feature map and a training second scale feature map, wherein the first convolutional neural network model uses a first hole convolutional kernel with a first hole rate, and the second convolutional neural network model uses a second hole convolutional kernel with a second hole rate;
calculating a training difference feature map between the training first scale feature map and the training second scale feature map;
expanding each feature matrix of the training differential feature map along the channel dimension into a feature vector to obtain a plurality of training local differential feature vectors;
passing the plurality of training local differential feature vectors through the converter-based context encoder to obtain training global differential feature vectors; and
passing the training global differential feature vector through the classifier to obtain a classification loss function value;
calculating a sequence pair sequence response rule intrinsic learning loss function value based on a distance between the training first scale feature map and the training second scale feature map; and
computing a weighted sum of the classification loss function values and the sequence versus sequence response rules intrinsic learning loss function values as loss function values to train the dual-stream detection network, the converter-based context encoder, and the classifier.
9. The image processing-based crushing job monitoring method according to claim 8, wherein the calculating a sequence pair sequence response rule intrinsic learning loss function value based on the distance between the training first scale feature map and the training second scale feature map comprises:
calculating a learning loss function value in the sequence-to-sequence response rule based on the distance between the training first scale feature map and the training second scale feature map according to the following formula;
wherein the formula is:
wherein, V 1 And V 2 Respectively are the feature vectors obtained after the training first scale feature map and the training second scale feature map are unfolded, and W is 1 And W 2 Respectively, the classifier is used for developing the training first scale feature map and the training second scale feature map to obtain a weight matrix of a feature vector, reLU (-) represents a ReLU activation function, sigmoid (-) represents a Sigmoid activation function,
representing matrix multiplication, d (·,) represents the euclidean distance between two vectors,
the values of the learned loss functions inherent to the sequence response rules are expressed.
10. A crushing operation monitoring system based on image processing is characterized by comprising:
the image acquisition module is used for acquiring images acquired by the camera after the crushing operation;
a dust removal module for passing the post-crushing image through a dust removal generator based on a countermeasure generation network to obtain a post-crushing image;
the multi-scale convolution module is used for enabling the generated images after the crushing operation to pass through a double-flow detection network comprising a first convolution neural network model and a second convolution neural network model to obtain a first scale feature map and a second scale feature map, wherein the first convolution neural network model uses a first hole convolution kernel with a first hole rate, and the second convolution neural network model uses a second hole convolution kernel with a second hole rate;
a difference module, configured to calculate a difference feature map between the first scale feature map and the second scale feature map;
the feature vector generation module is used for expanding each feature matrix of the differential feature map along the channel dimension into feature vectors to obtain a plurality of local differential feature vectors;
a context coding module, configured to pass the plurality of local differential feature vectors through a converter-based context coder to obtain a global differential feature vector; and
and the classification result generation module is used for enabling the global differential feature vector to pass through a classifier to obtain a classification result, and the classification result is used for indicating whether the raw coal crushing effect meets a preset standard or not.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211470253.6A CN115761642A (en) | 2022-11-23 | 2022-11-23 | Image processing-based crushing operation monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211470253.6A CN115761642A (en) | 2022-11-23 | 2022-11-23 | Image processing-based crushing operation monitoring method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115761642A true CN115761642A (en) | 2023-03-07 |
Family
ID=85335447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211470253.6A Pending CN115761642A (en) | 2022-11-23 | 2022-11-23 | Image processing-based crushing operation monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115761642A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048031A (en) * | 2023-03-31 | 2023-05-02 | 克拉玛依市蓝润环保科技有限责任公司 | Control system and method for petroleum auxiliary production |
| CN116468699A (en) * | 2023-04-23 | 2023-07-21 | 浙江酷趣网络科技有限公司杭州分公司 | Intelligent production system and method for fabric changing color along with light intensity |
| CN116561814A (en) * | 2023-05-17 | 2023-08-08 | 杭州君方科技有限公司 | Textile chemical fiber supply chain information tamper-proof method and system thereof |
| CN116758029A (en) * | 2023-06-15 | 2023-09-15 | 东莞市商斯迈智能科技有限公司 | Window cleaning machine movement control method and system based on machine vision |
| CN116777892A (en) * | 2023-07-03 | 2023-09-19 | 东莞市震坤行胶粘剂有限公司 | Method and system for detecting dispensing quality based on visual detection |
| CN116980976A (en) * | 2023-09-21 | 2023-10-31 | 深圳市聚亚科技有限公司 | Data transparent transmission method based on 4G communication module |
| CN117034123A (en) * | 2023-08-28 | 2023-11-10 | 定州市云领域体育用品有限公司 | Fault monitoring system and method for fitness equipment |
| CN117021417A (en) * | 2023-08-22 | 2023-11-10 | 河北泽源橡胶制品有限公司 | Rubber crushing and decomposing control method and system thereof |
-
2022
- 2022-11-23 CN CN202211470253.6A patent/CN115761642A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048031A (en) * | 2023-03-31 | 2023-05-02 | 克拉玛依市蓝润环保科技有限责任公司 | Control system and method for petroleum auxiliary production |
| CN116048031B (en) * | 2023-03-31 | 2023-08-04 | 克拉玛依市蓝润环保科技有限责任公司 | Control system and method for petroleum auxiliary production |
| CN116468699A (en) * | 2023-04-23 | 2023-07-21 | 浙江酷趣网络科技有限公司杭州分公司 | Intelligent production system and method for fabric changing color along with light intensity |
| CN116561814B (en) * | 2023-05-17 | 2023-11-24 | 杭州君方科技有限公司 | Textile chemical fiber supply chain information tamper-proof method and system thereof |
| CN116561814A (en) * | 2023-05-17 | 2023-08-08 | 杭州君方科技有限公司 | Textile chemical fiber supply chain information tamper-proof method and system thereof |
| CN116758029A (en) * | 2023-06-15 | 2023-09-15 | 东莞市商斯迈智能科技有限公司 | Window cleaning machine movement control method and system based on machine vision |
| CN116777892A (en) * | 2023-07-03 | 2023-09-19 | 东莞市震坤行胶粘剂有限公司 | Method and system for detecting dispensing quality based on visual detection |
| CN116777892B (en) * | 2023-07-03 | 2024-01-26 | 东莞市震坤行胶粘剂有限公司 | Method and system for detecting dispensing quality based on visual detection |
| CN117021417A (en) * | 2023-08-22 | 2023-11-10 | 河北泽源橡胶制品有限公司 | Rubber crushing and decomposing control method and system thereof |
| CN117021417B (en) * | 2023-08-22 | 2024-03-29 | 河北泽源橡胶制品有限公司 | Rubber crushing and decomposing control method and system thereof |
| CN117034123A (en) * | 2023-08-28 | 2023-11-10 | 定州市云领域体育用品有限公司 | Fault monitoring system and method for fitness equipment |
| CN117034123B (en) * | 2023-08-28 | 2024-05-07 | 定州市云领域体育用品有限公司 | Fault monitoring system and method for fitness equipment |
| CN116980976A (en) * | 2023-09-21 | 2023-10-31 | 深圳市聚亚科技有限公司 | Data transparent transmission method based on 4G communication module |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115761642A (en) | Image processing-based crushing operation monitoring method and system | |
| Wei et al. | Heuristic black-box adversarial attacks on video recognition models | |
| CN115783923B (en) | Elevator fault mode identification system based on big data | |
| Ou et al. | A CNN framework with slow-fast band selection and feature fusion grouping for hyperspectral image change detection | |
| CN116247824B (en) | Control method and system for power equipment | |
| CN115456789B (en) | Abnormal transaction detection method and system based on transaction pattern recognition | |
| Chen et al. | An efficient network behavior anomaly detection using a hybrid DBN-LSTM network | |
| CN115759658B (en) | Enterprise energy consumption data management system suitable for smart city | |
| CN114782882B (en) | Video target behavior anomaly detection method and system based on multi-modal feature fusion | |
| CN111079374B (en) | Font generation method, apparatus and storage medium | |
| CN115909260A (en) | Method and system for early warning of workplace intrusion based on machine vision | |
| CN116309580B (en) | Oil and gas pipeline corrosion detection method based on magnetic stress | |
| CN112182585B (en) | Source code vulnerability detection method, system and storage medium | |
| CN117058622A (en) | Intelligent monitoring system and method for sewage treatment equipment | |
| Chen et al. | Learning capsules for vehicle logo recognition | |
| CN116257406A (en) | Gateway data management method and system for smart city | |
| CN115600140A (en) | Fan variable pitch system fault identification method and system based on multi-source data fusion | |
| CN115620303A (en) | Personnel file intelligent management system | |
| CN115392320A (en) | Transformer with anti-theft function and method thereof | |
| CN115324843A (en) | Wind generating set fault diagnosis system and method based on monitoring data | |
| CN116168243A (en) | Intelligent production system and method for shaver | |
| Lang et al. | A Survey on Out-of-Distribution Detection in NLP | |
| CN113449679B (en) | Method and device for identifying abnormal behaviors of human body | |
| CN113778256A (en) | Electronic equipment with touch screen and touch unlocking method thereof | |
| CN113255752A (en) | Solid material consistency sorting method based on feature clustering |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |