WO2024066061A1 - 基于图像识别的斗轮损伤判定方法及其系统 - Google Patents

基于图像识别的斗轮损伤判定方法及其系统 Download PDF

Info

Publication number
WO2024066061A1
WO2024066061A1 PCT/CN2022/138434 CN2022138434W WO2024066061A1 WO 2024066061 A1 WO2024066061 A1 WO 2024066061A1 CN 2022138434 W CN2022138434 W CN 2022138434W WO 2024066061 A1 WO2024066061 A1 WO 2024066061A1
Authority
WO
WIPO (PCT)
Prior art keywords
bucket tooth
similarity
feature
matrix
bucket
Prior art date
Application number
PCT/CN2022/138434
Other languages
English (en)
French (fr)
Inventor
郑安
张波
刘鹏飞
马广玉
咸金龙
刘强
刘跃
郑树坤
冯川
刘立丰
Original Assignee
华能伊敏煤电有限责任公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华能伊敏煤电有限责任公司 filed Critical 华能伊敏煤电有限责任公司
Publication of WO2024066061A1 publication Critical patent/WO2024066061A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present application relates to the field of intelligent detection technology, and more specifically, to a bucket wheel damage determination method and system based on image recognition.
  • excavators are indispensable in urban construction.
  • the excavator mainly uses the bucket to dig materials above or below the bearing surface, and loads them into transport vehicles or unloads them to the stockpile yard.
  • the development of excavators is relatively fast.
  • Excavators have become one of the most important engineering machinery in engineering construction.
  • Excavator bucket teeth are key vulnerable parts of excavators. Because they are in direct contact with ore, sand, rock, etc. during use, the working conditions are very harsh, the service life is short, the replacement is frequent, and the consumption is huge.
  • the embodiment of the present application provides a bucket wheel damage determination method and system based on image recognition, which extracts local implicit features of bucket tooth monitoring images at multiple predetermined time points within a predetermined time period in a high-dimensional feature space through a first convolutional neural network model as a filter; then, the implicit feature differential similarity comparison of each two bucket tooth monitoring images is used to determine whether there is damage on the bucket tooth surface based on the global implicit feature information of the bucket tooth monitoring images. In this way, it is possible to accurately determine whether there is damage on the bucket tooth surface, and then timely and effectively repair and replace bucket teeth with damaged surfaces to ensure the normal and safe operation of the excavator.
  • a bucket wheel damage determination method based on image recognition which includes:
  • the bucket tooth monitoring images at each predetermined time point are respectively passed through a first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors;
  • the similarity matrix is passed through a second convolutional neural network model as a feature extractor to obtain a similarity feature matrix; the multiple bucket tooth local feature vectors are arranged in two dimensions to obtain a global bucket tooth feature matrix;
  • the global bucket tooth feature matrix and the similarity feature matrix are passed through a graph neural network to obtain a similarity topology global bucket tooth feature matrix;
  • the optimized similarity topology global bucket tooth feature matrix is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether there is damage on the bucket tooth surface.
  • the bucket tooth monitoring images at each predetermined time point are respectively passed through the first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors, including: using each layer of the first convolutional neural network model as a filter to perform the following on the input data in the forward pass of the layer: performing convolution processing on the input data to obtain a convolution feature map; performing mean pooling processing based on a feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network model as a filter is the plurality of bucket tooth local feature vectors, and the input of the first layer of the first convolutional neural network model as a filter is the bucket tooth monitoring images at each predetermined time point.
  • the calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors to obtain a similarity matrix includes: calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors using the following formula to obtain multiple similarities; wherein the formula is:
  • Vi and Vj represent every two bucket tooth local feature vectors in the plurality of bucket tooth local feature vectors, respectively. and Respectively representing the eigenvalues of each position of every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors, d(V i , V j ) represents the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors; and, arranging the multiple similarities in two dimensions to obtain the similarity matrix.
  • the similarity matrix is passed through the second convolutional neural network model as a feature extractor to obtain a similarity feature matrix, including: using each layer of the second convolutional neural network model as a feature extractor to perform the following on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolution feature map; pooling processing on the convolution feature map along the channel dimension to obtain a pooling feature map; and nonlinear activation on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the second convolutional neural network model as a feature extractor is the similarity feature matrix, and the input of the first layer of the second convolutional neural network model as a feature extractor is the similarity matrix.
  • the similarity topology global bucket tooth feature matrix is subjected to feature compensation based on pre-classification to obtain the optimized similarity topology global bucket tooth feature matrix, including: the similarity topology global bucket tooth feature matrix is subjected to feature compensation based on pre-classification to obtain the optimized similarity topology global bucket tooth feature matrix according to the following formula; wherein the formula is:
  • M represents the similarity topology global bucket tooth feature matrix
  • M′ represents the optimized similarity topology global bucket tooth feature matrix
  • p represents the probability value of the similarity topology global bucket tooth feature matrix obtained by the classifier
  • ⁇ represents the point multiplication by position.
  • the step of passing the optimized similarity topology global bucket tooth feature matrix through a classifier to obtain a classification result includes: using the classifier to process the optimized similarity topology global bucket tooth feature matrix using the following formula to generate a classification result, wherein the formula is: softmax ⁇ ( Wn , Bn ):...:( W1 , B1 )
  • a bucket wheel damage determination system based on image recognition which includes:
  • An image acquisition module is used to acquire bucket tooth monitoring images at multiple predetermined time points within a predetermined time period acquired by a camera;
  • a bucket tooth local feature extraction module is used to pass the bucket tooth monitoring images at each predetermined time point through a first convolutional neural network model as a filter to obtain multiple bucket tooth local feature vectors;
  • a similarity calculation module used for calculating the similarity between every two bucket tooth local feature vectors in the plurality of bucket tooth local feature vectors to obtain a similarity matrix
  • a similarity feature extraction module used for passing the similarity matrix through a second convolutional neural network model as a feature extractor to obtain a similarity feature matrix
  • a two-dimensional arrangement module used for two-dimensionally arranging the plurality of bucket tooth local feature vectors to obtain a global bucket tooth feature matrix
  • a graph structure data encoding module used for obtaining a similarity topology global bucket tooth feature matrix by passing the global bucket tooth feature matrix and the similarity feature matrix through a graph neural network
  • a feature compensation module used for performing feature compensation based on pre-classification on the similarity topology global bucket tooth feature matrix to obtain an optimized similarity topology global bucket tooth feature matrix
  • the damage judgment result generation module is used to pass the optimized similarity topology global bucket tooth feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is damage on the bucket tooth surface.
  • the bucket wheel damage determination method and system based on image recognition extracts the local implicit features of the bucket tooth monitoring images at multiple predetermined time points within a predetermined time period in the high-dimensional feature space through the first convolutional neural network model as a filter; then, the implicit feature differential similarity of each two bucket tooth monitoring images is used to compare the global implicit feature information of the bucket tooth monitoring images to determine whether there is damage on the bucket tooth surface. In this way, it is possible to accurately determine whether there is damage on the bucket tooth surface, and then timely and effectively repair and replace the bucket teeth with surface damage to ensure the normal and safe operation of the excavator.
  • FIG1 illustrates an application scenario diagram of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • FIG2 illustrates a flow chart of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • FIG3 illustrates a schematic diagram of the architecture of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • FIG. 4 illustrates a flow chart of a method for determining bucket wheel damage based on image recognition according to an embodiment of the present application, in which the bucket tooth monitoring images at each predetermined time point are respectively passed through a first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors.
  • Figure 5 illustrates a flow chart of obtaining a similarity feature matrix by passing the similarity matrix through a second convolutional neural network model as a feature extractor in a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • FIG6 illustrates a block diagram of a bucket wheel damage determination system based on image recognition according to an embodiment of the present application.
  • deep learning and neural networks have been widely used in computer vision, natural language processing, speech signal processing and other fields.
  • deep learning and neural networks have also shown a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields.
  • an artificial intelligence judgment method based on deep learning is adopted to utilize image recognition technology to accurately judge whether the bucket wheel is damaged.
  • the deep implicit features of the bucket teeth at that time point are obtained by extracting the implicit feature distribution information in multiple bucket tooth monitoring images in the time series dimension, thereby filtering out the recognition effects caused by interfering objects such as soil.
  • the damage judgment of the bucket tooth surface is improved based on the global implicit feature information of the bucket teeth by further comparing the implicit feature differential similarity of each two bucket tooth monitoring images. In this way, it is possible to accurately judge whether the bucket tooth surface is damaged, and then the bucket teeth can be repaired and replaced in a timely and effective manner to ensure the normal and safe operation of the excavator.
  • bucket tooth monitoring images at multiple predetermined time points within a predetermined time period are collected by a camera.
  • the bucket tooth monitoring images at each predetermined time point are further subjected to feature mining in a first convolutional neural network model as a filter to extract local implicit features of the bucket tooth monitoring images in a high-dimensional feature space, thereby obtaining multiple bucket tooth local feature vectors.
  • the camera can only capture images of local areas of the bucket teeth at each predetermined time point.
  • the damage on the bucket tooth surface has local characteristics, but global detection is required based on the entire bucket tooth surface. Therefore, in the technical solution of the present application, the judgment of the damage on the bucket tooth surface is further optimized by comparing the implicit feature difference similarity of every two bucket tooth monitoring images.
  • the similarity between each two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors is further calculated, such as the cosine distance, to obtain a similarity matrix.
  • the similarity matrix is subjected to feature extraction in a second convolutional neural network model as a feature extractor to extract hidden correlation features of the similarity of implicit features of each two bucket tooth monitoring images in the similarity matrix, thereby obtaining a similarity feature matrix.
  • the local feature vectors of the bucket teeth at each predetermined time point are used as feature representations of the nodes, and the similarity feature matrix is used as feature representations of the edges between nodes.
  • the global bucket tooth feature matrix and the similarity feature matrix obtained by two-dimensionally arranging the multiple local feature vectors of the bucket teeth are passed through a graph neural network to obtain a similarity topology global bucket tooth feature matrix.
  • the graph neural network performs graph structure data encoding on the global bucket tooth feature matrix and the similarity feature matrix through learnable neural network parameters to obtain the similarity topology global bucket tooth feature matrix containing similarity association features and implicit feature information of the bucket teeth at each predetermined time point.
  • the optimized similarity topology global bucket tooth feature matrix is passed through a classifier to obtain a classification result for indicating whether there is damage on the bucket tooth surface.
  • each row vector of the similarity topology global bucket tooth feature matrix has a similarity topology association relationship. Therefore, when the similarity topology global bucket tooth feature matrix is classified by a classifier, quasi-coherent interference is likely to occur.
  • the similarity topology global tooth feature matrix for example, denoted as M
  • M is corrected by a class probability coherence compensation mechanism based on pre-classification, which is expressed as:
  • p is the probability value obtained by the similarity topology global bucket tooth feature matrix M through the classifier.
  • the weight matrix of the classifier itself will have class coherence for each row vector during the classification process, thereby causing class coherence interference to the similarity topology global bucket tooth feature matrix M.
  • the class probability value of the classifier obtained by pre-classification is used as the multiplicative interference noise term of the similarity topology global bucket tooth feature matrix M to perform class probability coherence compensation on the similarity topology global bucket tooth feature matrix M, so that the equivalent probability intensity representation of the similarity topology global bucket tooth feature matrix M in the absence of interference can be restored, that is, the optimized similarity topology global bucket tooth feature matrix M′, thereby realizing the correction of the similarity topology global bucket tooth feature matrix M and improving the accuracy of the classification result. In this way, it is possible to accurately judge whether the bucket tooth surface is damaged, and then timely and effectively repair and replace the bucket teeth with damaged surfaces to ensure the normal and safe operation of the excavator.
  • the present application proposes a bucket wheel damage judgment method based on image recognition, which includes: obtaining bucket tooth monitoring images at multiple predetermined time points within a predetermined time period collected by a camera; passing the bucket tooth monitoring images at each predetermined time point through a first convolutional neural network model as a filter to obtain multiple bucket tooth local feature vectors; calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors to obtain a similarity matrix; passing the similarity matrix through a second convolutional neural network model as a feature extractor to obtain a similarity feature matrix; arranging the multiple bucket tooth local feature vectors in two dimensions to obtain a global bucket tooth feature matrix; passing the global bucket tooth feature matrix and the similarity feature matrix through a graph neural network to obtain a similarity topology global bucket tooth feature matrix; performing feature compensation based on pre-classification on the similarity topology global bucket tooth feature matrix to obtain an optimized similarity topology global bucket tooth feature matrix; and passing the optimized similarity topology global bucket tooth feature matrix through a class
  • Fig. 1 illustrates an application scenario of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • bucket tooth monitoring images e.g., M as shown in Fig. 1
  • an excavator bucket tooth e.g., F as shown in Fig. 1
  • the collected bucket tooth monitoring images are input into a server (e.g., S as shown in Fig. 1) in which a bucket wheel damage determination algorithm based on image recognition is deployed, wherein the server processes the bucket tooth monitoring images with a bucket wheel damage determination algorithm based on image recognition to output a classification result for indicating whether there is damage on the bucket tooth surface.
  • FIG2 illustrates a flow chart of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • the bucket wheel damage determination method based on image recognition according to the embodiment of the present application includes: S110, acquiring bucket tooth monitoring images at multiple predetermined time points within a predetermined time period collected by a camera; S120, respectively passing the bucket tooth monitoring images at each predetermined time point through a first convolutional neural network model as a filter to obtain multiple bucket tooth local feature vectors; S130, calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors to obtain a similarity matrix; S140, passing the similarity matrix through a second convolutional neural network model as a feature extractor to obtain a similarity feature matrix; S150, arranging the multiple bucket tooth local feature vectors in two dimensions to obtain a global bucket tooth feature matrix; S160, passing the global bucket tooth feature matrix and the similarity feature matrix through a graph neural network to obtain a similarity topology global bucket tooth feature matrix;
  • FIG3 illustrates a schematic diagram of the architecture of a bucket wheel damage determination method based on image recognition according to an embodiment of the present application.
  • the network architecture of the bucket wheel damage determination method based on image recognition first, bucket tooth monitoring images at multiple predetermined time points within a predetermined time period collected by a camera are obtained; then, the bucket tooth monitoring images at each predetermined time point are respectively passed through a first convolutional neural network model as a filter to obtain multiple bucket tooth local feature vectors; then, the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors is calculated to obtain a similarity matrix; then, the similarity matrix is passed through a second convolutional neural network model as a feature extractor to obtain a similarity feature matrix; then, the multiple bucket tooth local feature vectors are two-dimensionally arranged to obtain a global bucket tooth feature matrix; then, the global bucket tooth feature matrix and the similarity feature matrix are passed through a graph neural network to obtain a similarity topology global bucket tooth feature matrix; then,
  • step S110 bucket tooth monitoring images at multiple predetermined time points within a predetermined time period are acquired by a camera.
  • existing excavators collect images by installing a camera for processing to determine whether the bucket teeth are damaged.
  • the bucket teeth are rotating during operation, and the camera can only collect images of a local area of the bucket teeth at each predetermined time point, which will result in inaccurate judgment of bucket tooth damage.
  • interference objects such as soil may adhere to the surface of the bucket teeth, which may interfere with the judgment of bucket tooth damage. Therefore, an optimized bucket wheel damage judgment scheme based on image recognition is desired.
  • deep learning and neural networks have been widely used in computer vision, natural language processing, speech signal processing and other fields.
  • deep learning and neural networks have also shown a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields.
  • an artificial intelligence judgment method based on deep learning is adopted to utilize image recognition technology to accurately judge whether the bucket wheel is damaged.
  • the deep implicit features of the bucket teeth at that time point are obtained by extracting the implicit feature distribution information in multiple bucket tooth monitoring images in the time series dimension, thereby filtering out the recognition effects caused by interfering objects such as soil.
  • the damage judgment of the bucket tooth surface is improved based on the global implicit feature information of the bucket teeth by further comparing the implicit feature differential similarity of each two bucket tooth monitoring images. In this way, it is possible to accurately judge whether the bucket tooth surface is damaged, and then the bucket teeth can be repaired and replaced in a timely and effective manner to ensure the normal and safe operation of the excavator.
  • bucket tooth monitoring images at multiple predetermined time points within a predetermined time period are collected by a camera.
  • step S120 the bucket tooth monitoring images at the predetermined time points are respectively passed through the first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors.
  • the bucket tooth monitoring images at the predetermined time points are further respectively passed through the first convolutional neural network model as a filter for feature mining to extract the local implicit features of the bucket tooth monitoring images in the high-dimensional feature space, thereby obtaining a plurality of bucket tooth local feature vectors.
  • FIG4 illustrates a flowchart of a method for determining bucket wheel damage based on image recognition according to an embodiment of the present application, in which the bucket tooth monitoring images at each predetermined time point are respectively passed through the first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors.
  • the bucket tooth monitoring images at each predetermined time point are respectively passed through the first convolutional neural network model as a filter to obtain a plurality of bucket tooth local feature vectors, including: using each layer of the first convolutional neural network model as a filter to perform the following on the input data in the forward pass of the layer: S210, performing convolution processing on the input data to obtain a convolutional feature map; S220, performing mean pooling processing based on a feature matrix on the convolutional feature map to obtain a pooled feature map; and, S230, performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the first convolutional neural network model as a filter is the plurality of bucket tooth local feature vectors, and the input of the first layer of the first convolutional neural network model as a filter is the bucket tooth monitoring images at each predetermined time point.
  • step S130 the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors is calculated to obtain a similarity matrix. Since the bucket tooth keeps rotating during operation, the camera can only capture the image of the local area of the bucket tooth at each predetermined time point. In addition, considering that in the process of judging the damage on the bucket tooth surface, since the damage on the bucket tooth surface has local characteristics, it is necessary to perform global detection based on the entire bucket tooth surface.
  • the determination of the surface damage of the bucket teeth is further optimized by comparing the implicit feature difference similarity of each two bucket tooth monitoring images. That is, specifically, the similarity between each two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors is further calculated, such as the cosine distance, to obtain a similarity matrix.
  • the calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors to obtain a similarity matrix includes: calculating the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors using the following formula to obtain multiple similarities; wherein the formula is:
  • Vi and Vj represent every two bucket tooth local feature vectors in the plurality of bucket tooth local feature vectors, respectively. and Respectively representing the eigenvalues of each position of every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors, d(V i , V j ) represents the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors; and, arranging the multiple similarities in two dimensions to obtain the similarity matrix.
  • step S140 the similarity matrix is passed through the second convolutional neural network model as a feature extractor to obtain a similarity feature matrix. That is, the similarity matrix is passed through the second convolutional neural network model as a feature extractor to perform feature extraction, so as to extract the hidden correlation features of the similarity of the implicit features of every two bucket tooth monitoring images in the similarity matrix, thereby obtaining a similarity feature matrix.
  • Figure 5 illustrates a flowchart of passing the similarity matrix through the second convolutional neural network model as a feature extractor to obtain a similarity feature matrix in the bucket wheel damage judgment method based on image recognition according to an embodiment of the present application.
  • the passing of the similarity matrix through the second convolutional neural network model as a feature extractor to obtain a similarity feature matrix includes: using each layer of the second convolutional neural network model as a feature extractor to perform the following on the input data in the forward pass of the layer: S310, performing convolution processing on the input data to obtain a convolution feature map; S320, performing pooling processing on the convolution feature map along the channel dimension to obtain a pooling feature map; and, S330, performing nonlinear activation on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the second convolutional neural network model as a feature extractor is the similarity feature matrix, and the input of the first layer of the second convolutional neural network model as a feature extractor is the similarity matrix.
  • step S150 and step S160 the multiple bucket tooth local feature vectors are arranged in two dimensions to obtain a global bucket tooth feature matrix, and then the global bucket tooth feature matrix and the similarity feature matrix are passed through a graph neural network to obtain a similarity topology global bucket tooth feature matrix. Further, the bucket tooth local feature vectors at each predetermined time point are used as feature representations of nodes, and the similarity feature matrix is used as feature representations of edges between nodes, and the global bucket tooth feature matrix and the similarity feature matrix obtained by two-dimensional arrangement of the multiple bucket tooth local feature vectors are passed through a graph neural network to obtain a similarity topology global bucket tooth feature matrix.
  • the graph neural network encodes the global bucket tooth feature matrix and the similarity feature matrix through learnable neural network parameters to obtain the similarity topology global bucket tooth feature matrix containing similarity association features and bucket tooth implicit feature information at each predetermined time point. Then, the optimized similarity topology global bucket tooth feature matrix is passed through a classifier to obtain a classification result for indicating whether there is damage on the bucket tooth surface.
  • step S170 the similarity topology global bucket tooth feature matrix is subjected to feature compensation based on pre-classification to obtain an optimized similarity topology global bucket tooth feature matrix.
  • the similarity topology global bucket tooth feature matrix is obtained by combining the global bucket tooth feature matrix and the similarity feature matrix through a graph neural network, each row vector of the similarity topology global bucket tooth feature matrix has a similarity topological association relationship. Therefore, when the similarity topology global bucket tooth feature matrix is classified through a classifier, quasi-coherent interference is likely to occur.
  • the similarity topology global bucket tooth feature matrix for example, denoted as M
  • M is corrected by a class probability coherence compensation mechanism based on pre-classification, that is, the similarity topology global bucket tooth feature matrix is subjected to feature compensation based on pre-classification using the following formula to obtain the optimized similarity topology global bucket tooth feature matrix; wherein the formula is:
  • M represents the similarity topology global bucket tooth feature matrix
  • M′ represents the optimized similarity topology global bucket tooth feature matrix
  • p represents the probability value of the similarity topology global bucket tooth feature matrix obtained by the classifier
  • ⁇ represents the point multiplication by position.
  • the weight matrix of the classifier itself will have class coherence for each row vector during the classification process, thereby causing class coherence interference to the similarity topology global bucket tooth feature matrix M.
  • the class probability value of the classifier obtained by pre-classification is used as the multiplicative interference noise term of the similarity topology global bucket tooth feature matrix M to perform class probability coherence compensation on the similarity topology global bucket tooth feature matrix M, so that the equivalent probability intensity representation of the similarity topology global bucket tooth feature matrix M in the absence of interference can be restored, that is, the optimized similarity topology global bucket tooth feature matrix M′, thereby realizing the correction of the similarity topology global bucket tooth feature matrix M and improving the accuracy of the classification result. In this way, it is possible to accurately judge whether the bucket tooth surface is damaged, and then timely and effectively repair and replace the bucket teeth with damaged surfaces to ensure the normal and safe operation of the excavator.
  • step S180 the optimized similarity topology global bucket tooth feature matrix is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether there is damage on the bucket tooth surface.
  • the classifier is used to process the optimized similarity topology global bucket tooth feature matrix using the following formula to generate a classification result, wherein the formula is: softmax ⁇ ( Wn , Bn ):...:( W1 , B1 )
  • the bucket wheel damage determination method based on image recognition also includes: extracting the row vector corresponding to the bucket tooth to be evaluated from the optimized similarity topology global bucket tooth feature matrix as a classification feature vector; and passing the classification feature vector through the classifier to obtain a second classification result indicating whether the bucket tooth to be evaluated is damaged.
  • the bucket wheel damage determination method and system based on image recognition according to the embodiment of the present application are explained, which extracts the local implicit features of the bucket tooth monitoring images at multiple predetermined time points in a predetermined time period in the high-dimensional feature space through the first convolutional neural network model as a filter; then, the implicit feature differential similarity of each two bucket tooth monitoring images is used to compare the global implicit feature information of the bucket tooth monitoring images to determine whether there is damage on the bucket tooth surface. In this way, it is possible to accurately determine whether there is damage on the bucket tooth surface, and then timely and effectively repair and replace the bucket teeth with surface damage to ensure the normal and safe operation of the excavator.
  • FIG6 illustrates a block diagram of a bucket wheel damage determination system based on image recognition according to an embodiment of the present application.
  • a bucket wheel damage determination system 100 based on image recognition includes: an image acquisition module 110, which is used to acquire bucket tooth monitoring images at multiple predetermined time points within a predetermined time period acquired by a camera; a bucket tooth local feature extraction module 120, which is used to pass the bucket tooth monitoring images at each predetermined time point through a first convolutional neural network model as a filter to obtain multiple bucket tooth local feature vectors; a similarity calculation module 130, which is used to calculate the similarity between each two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors to obtain a similarity matrix; a similarity feature extraction module 140, which is used to pass the similarity matrix through a second convolutional neural network model as a feature extractor.
  • a two-dimensional arrangement module 150 used to arrange the multiple bucket tooth local feature vectors in two dimensions to obtain a global bucket tooth feature matrix
  • a graph structure data encoding module 160 used to pass the global bucket tooth feature matrix and the similarity feature matrix through a graph neural network to obtain a similarity topology global bucket tooth feature matrix
  • a feature compensation module 170 used to perform feature compensation based on pre-classification on the similarity topology global bucket tooth feature matrix to obtain an optimized similarity topology global bucket tooth feature matrix
  • a damage judgment result generation module 180 used to pass the optimized similarity topology global bucket tooth feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is damage on the bucket tooth surface.
  • the bucket tooth local feature extraction module is used to: use each layer of the first convolutional neural network model as a filter to perform the following on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolution feature map; mean pooling processing based on a feature matrix on the convolution feature map to obtain a pooled feature map; and nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the first convolutional neural network model as a filter is the multiple bucket tooth local feature vectors, and the input of the first layer of the first convolutional neural network model as a filter is the bucket tooth monitoring image at each predetermined time point.
  • an artificial intelligence judgment method based on deep learning is adopted to utilize image recognition technology to accurately judge whether the bucket wheel is damaged.
  • the deep implicit features of the bucket teeth at that time point are obtained by extracting the implicit feature distribution information in multiple bucket tooth monitoring images in the time series dimension, thereby filtering out the recognition effects caused by interfering objects such as soil.
  • the damage judgment of the bucket tooth surface is improved based on the global implicit feature information of the bucket teeth by further comparing the implicit feature differential similarity of each two bucket tooth monitoring images. In this way, it is possible to accurately judge whether the bucket tooth surface is damaged, and then the bucket teeth can be repaired and replaced in a timely and effective manner to ensure the normal and safe operation of the excavator.
  • bucket tooth monitoring images at multiple predetermined time points within a predetermined time period are collected by a camera.
  • the bucket tooth monitoring images at each predetermined time point are further subjected to feature mining in a first convolutional neural network model as a filter to extract local implicit features of the bucket tooth monitoring images in a high-dimensional feature space, thereby obtaining multiple bucket tooth local feature vectors.
  • the similarity calculation module is used to calculate the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors using the following formula to obtain multiple similarities; wherein the formula is:
  • Vi and Vj represent every two bucket tooth local feature vectors in the plurality of bucket tooth local feature vectors, respectively. and Respectively representing the eigenvalues of each position of every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors, d(V i , V j ) represents the similarity between every two bucket tooth local feature vectors in the multiple bucket tooth local feature vectors; and, arranging the multiple similarities in two dimensions to obtain the similarity matrix.
  • the camera can only capture images of local areas of the bucket teeth at each predetermined time point.
  • the damage on the bucket tooth surface has local characteristics, but global detection is required based on the entire bucket tooth surface. Therefore, in the technical solution of the present application, the judgment of the damage on the bucket tooth surface is further optimized by comparing the implicit feature difference similarity of every two bucket tooth monitoring images.
  • the similarity between every two bucket tooth local feature vectors in the plurality of bucket tooth local feature vectors is further calculated, such as the cosine distance, to obtain a similarity matrix.
  • the similarity feature extraction module is used to: use each layer of the second convolutional neural network model as a feature extractor to perform the following on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolution feature map; pooling processing on the convolution feature map along the channel dimension to obtain a pooled feature map; and, nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the second convolutional neural network model as a feature extractor is the similarity feature matrix, and the input of the first layer of the second convolutional neural network model as a feature extractor is the similarity matrix.
  • the similarity matrix is then subjected to feature extraction in a second convolutional neural network model serving as a feature extractor to extract hidden correlation features of the similarity of implicit features of every two bucket tooth monitoring images in the similarity matrix, thereby obtaining a similarity feature matrix.
  • the feature compensation module is used to perform feature compensation based on pre-classification on the similarity topology global bucket tooth feature matrix according to the following formula to obtain the optimized similarity topology global bucket tooth feature matrix; wherein the formula is:
  • M represents the similarity topology global bucket tooth feature matrix
  • M′ represents the optimized similarity topology global bucket tooth feature matrix
  • p represents the probability value of the similarity topology global bucket tooth feature matrix obtained by the classifier
  • ⁇ represents the point multiplication by position.
  • each row vector of the similarity topology global bucket tooth feature matrix has a similarity topology association relationship. Therefore, when the similarity topology global bucket tooth feature matrix is classified by a classifier, quasi-coherent interference is likely to occur.
  • a class probability coherence compensation mechanism correction based on pre-classification on the similarity topology global bucket tooth feature matrix, for example, denoted as M, that is, due to the similarity topological association relationship between the row vectors of the similarity topology global bucket tooth feature matrix M, when it is classified, the weight matrix of the classifier itself will have class coherence for each row vector during the classification process, thereby causing class coherence interference to the similarity topology global bucket tooth feature matrix M.
  • the category probability value of the classifier obtained by pre-classification is used as the multiplicative interference noise term of the similarity topology global bucket tooth feature matrix M to perform class probability coherence compensation on the similarity topology global bucket tooth feature matrix M, which can restore the equivalent probability intensity representation of the similarity topology global bucket tooth feature matrix M in the absence of interference, that is, the optimized similarity topology global bucket tooth feature matrix M′, thereby realizing the correction of the similarity topology global bucket tooth feature matrix M and improving the accuracy of the classification results.
  • the optimized similarity topology global bucket tooth feature matrix M′ realizing the correction of the similarity topology global bucket tooth feature matrix M and improving the accuracy of the classification results.
  • the damage judgment result generation module is used to: use the classifier to process the optimized similarity topology global bucket tooth feature matrix with the following formula to generate a classification result, wherein the formula is: softmax ⁇ ( Wn , Bn ):...:( W1 , B1 )
  • the above-mentioned bucket wheel damage judgment system 100 also includes: an extraction unit: used to extract the row vector corresponding to the bucket tooth to be evaluated from the optimized similarity topology global bucket tooth feature matrix as a classification feature vector; and a second classification result generation unit: used to pass the classification feature vector through the classifier to obtain a second classification result used to indicate whether the bucket tooth to be evaluated is damaged.
  • the bucket wheel damage determination system 100 based on image recognition can be implemented in various terminal devices, such as a server for bucket wheel damage determination based on image recognition.
  • the bucket wheel damage determination system 100 based on image recognition according to the embodiment of the present application can be integrated into the terminal device as a software module and/or a hardware module.
  • the bucket wheel damage determination system 100 based on image recognition can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the bucket wheel damage determination system 100 based on image recognition can also be one of the many hardware modules of the terminal device.
  • the bucket wheel damage determination system 100 based on image recognition and the terminal device may also be separate devices, and the bucket wheel damage determination system 100 based on image recognition may be connected to the terminal device via a wired and/or wireless network, and transmit interactive information in accordance with an agreed data format.
  • each component or each step can be decomposed and/or recombined.
  • Such decomposition and/or recombination should be regarded as equivalent solutions of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

公开了一种基于图像识别的斗轮损伤判定方法及其系统，其通过作为过滤器的第一卷积神经网络模型提取预定时间段内多个预定时间点的斗齿监控图像在高维特征空间中的局部隐含特征；然后，利用每两个斗齿监控图像的隐含特征差分相似性来对于斗齿表面是否存在损伤进行判定。这样，可以对于斗齿表面是否存在损伤进行准确地判定，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。

Description

基于图像识别的斗轮损伤判定方法及其系统 技术领域
本申请涉及智能化检测技术领域，且更为具体地，涉及一种基于图像识别的斗轮损伤判定方法及其系统。
背景技术
随着社会和经济的发展，城市的建设越来越快，在城市建设中当然少不了挖掘机，挖掘机挖掘时，主要是利用铲斗挖掘高于或低于承机面的物料，并装入运输车辆或卸至堆料场的土方机械，从近几年工程机械的发展来看，挖掘机的发展相对较快，挖掘机已经成为工程建设中最主要的工程机械之一。挖掘机斗齿是挖掘机的关键易损件，由于其在使用过程中直接与矿石、砂土、岩石等接触，工作条件十分恶劣，使用寿命短，更换频繁，消耗量巨大。
目前，现有的挖掘机上通过安装摄像头来采集图像进行处理，以判断斗齿是否存在损伤。但是，斗齿在工作的过程中保持着旋转状态，摄像头在每个预定时间点仅能采集到斗齿的局部区域的图像，这会导致对于斗齿损伤的判断不够精准。并且，由于在检测的过程中，斗齿表面会附着泥土等干扰对象以对于斗齿损伤的判断带来干扰。
因此，期待一种优化的基于图像识别的斗轮损伤判定方案。
发明内容
为了解决上述技术问题，提出了本申请。本申请的实施例提供了一种基于图像识别的斗轮损伤判定方法及其系统，其通过作为过滤器的第一卷积神经网络模型提取预定时间段内多个预定时间点的斗齿监控图像在高维特征空间中的局部隐含特征；然后，利用每两个斗齿监控图像的隐含特征差分相似性比较基于斗齿监控图像的全局隐含特征信息来对于斗齿表面是否存在损伤进行判定。这样，可以对于斗齿表面是否存在损伤进行准确地判定，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
根据本申请的一个方面，提供了一种基于图像识别的斗轮损伤判定方法，其包括：
获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；
将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；
计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；
将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；
将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；
对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及
将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
在上述基于图像识别的斗轮损伤判定方法中，所述将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量，包括：使用所述作为过滤器的第一卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：对所述输入数据进行卷积处理以得到卷积特征图；对所述卷积特征图进行基于特征矩阵的均值池化处理以得到池化特征图；以及，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为过滤器的第一卷积神经网络模型的最后一层的输出为所述多个斗齿局部特征 向量，所述作为过滤器的第一卷积神经网络模型的第一层的输入为所述各个预定时间点的斗齿监控图像。
在上述基于图像识别的斗轮损伤判定方法中，所述计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵，包括：以如下公式来计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到多个相似度；其中，所述公式为：
Figure PCTCN2022138434-appb-000001
其中V i和V j分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量，
Figure PCTCN2022138434-appb-000002
和
Figure PCTCN2022138434-appb-000003
分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量的各个位置的特征值，d(V i，V j)表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度；以及，将所述多个相似度进行二维排列以得到所述相似度矩阵。
在上述基于图像识别的斗轮损伤判定方法中，所述将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵，包括：使用所述作为特征提取器的第二卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：对所述输入数据进行卷积处理以得到卷积特征图；对所述卷积特征图进行沿通道维度的池化处理以得到池化特征图；以及，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为特征提取器的第二卷积神经网络模型的最后一层的输出为所述相似度特征矩阵，所述作为特征提取器的第二卷积神经网络模型的第一层的输入为所述相似度矩阵。
在上述基于图像识别的斗轮损伤判定方法中，所述对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵，包括：以如下公式对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到所述优化相似度拓扑全局斗齿特征矩阵；其中，所述公式为：
M′＝p p·M p-1⊙e -p·M
其中M表示所述相似度拓扑全局斗齿特征矩阵，M′表示所述优化相似度拓扑全局斗齿特征矩阵，p表示所述相似度拓扑全局斗齿特征矩阵通过所述分类器获得的概率值，⊙表示按位置点乘。
在上述基于图像识别的斗轮损伤判定方法中，所述将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，包括：使用所述分类器以如下公式对所述优化相似度拓扑全局斗齿特征矩阵进行处理以生成分类结果，其中，所述公式为：softmax{(W n，B n)：...：(W 1，B 1)|Project(F)}，其中Project(F)表示将所述优化相似度拓扑全局斗齿特征矩阵投影为向量，W 1至W n为各层全连接层的权重矩阵，B 1至B n表示各层全连接层的偏置矩阵。
根据本申请的另一方面，提供了一种基于图像识别的斗轮损伤判定系统，其包括：
图像采集模块，用于获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；斗齿局部特征提取模块，用于将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；
相似度计算模块，用于计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；
相似度特征提取模块，用于将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；
二维排列模块，用于将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；图结构数据编码模块，用于将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；
特征补偿模块，用于对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及
损伤判定结果生成模块，用于将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
与现有技术相比，本申请提供的基于图像识别的斗轮损伤判定方法及其系统，其通过作为过滤器的第一卷积神经网络模型提取预定时间段内多个预定时间点的斗齿监控图像在高维特征空间中的局部隐含特征；然后，利用每两个斗齿监控图像的隐含特征差分相似性比较基于斗齿监控图像的全局隐含特征信息来对于斗齿表面是否存在损伤进行判定。这样，可以对于斗齿表面是否存在损伤进行准确地判定，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
附图说明
通过结合附图对本申请实施例进行更详细的描述，本申请的上述以及其他目的、特征和优势将变得更加明显。附图用来提供对本申请实施例的进一步理解，并且构成说明书的一部分，与本申请实施例一起用于解释本申请，并不构成对本申请的限制。在附图中，相同的参考标号通常代表相同部件或步骤。
图1图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的应用场景图。
图2图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的流程图。
图3图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的架构示意图。
图4图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法中，将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量的流程图。
图5图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法中，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵的流程图。
图6图示了根据本申请实施例的基于图像识别的斗轮损伤判定系统的框图。
具体实施方式
下面，将参考附图详细地描述根据本申请的示例实施例。显然，所描述的实施例仅仅是本申请的一部分实施例，而不是本申请的全部实施例，应理解，本申请不受这里描述的示例实施例的限制。
场景概述
如上所述，现有的挖掘机上通过安装摄像头来采集图像进行处理，以判断斗齿是否存在损伤。但是，斗齿在工作的过程中保持着旋转状态，摄像头在每个预定时间点仅能采集到斗齿的局部区域的图像，这会导致对于斗齿损伤的判断不够精准。并且，由于在检测的过程中，斗齿表面会附着泥土等干扰对象以对于斗齿损伤的判断带来干扰。因此，期待一种优化的基于图像识别的斗轮损伤判定方案。
目前，深度学习以及神经网络已经广泛应用于计算机视觉、自然语言处理、语音信号处理等领域。此外，深度学习以及神经网络在图像分类、物体检测、语义分割、文本翻译等领域，也展现出了接近甚至超越人类的水平。
深度学习以及神经网络的发展为斗轮损伤的智能判定提供了新的解决思路和方案。
具体地，在本申请的技术方案中，通过采用基于深度学习的人工智能判定方法以利用图像识别技术来对于斗轮是否损伤进行精准判断。具体地，在此过程中，通过对于在时序维度上的多个斗齿监控图像中的隐含特征分布信息进行提取来得到所述斗齿在该时间点上的深层隐含特征，从而滤除泥土等干扰对象带来的识别影响。并且，进一步利用每两个所述斗齿监控图像的隐含特征差分相似性比较来基于所述斗齿的全局隐含特征信息来提高对于所述斗齿表面的损伤判定。这样，能够准确地对于所述斗齿表面是否存在损伤进行判断，进而能够及时有效地对于所述斗齿进行维修更换，以保证挖掘机的正常安全运行。
具体地，在本申请的技术方案中，首先，通过摄像头采集预定时间段内多个预定时间点的斗齿监控图像。然后，为了在斗齿损伤判断中滤除所述斗齿表面附着的泥土等干扰对象的影响，进一步将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型中进行特征挖掘，以提取出所述斗齿监控图像在高维特征空间中的局部隐含特征，从而得到多个斗齿局部特征向量。
接着，由于所述斗齿在工作过程中保持旋转状态，因此，所述摄像头在每个预定时间点仅能采集到所述斗齿的局部区域的图像。并且，考虑到在对于所述斗齿表面损伤的判断过程中，由于该所述斗齿表面的损伤是存在局部特性的，但是需要基于所述斗齿表面的整体来进行全局检测。因此，在本申请的技术方案中，进一步以每两个所述斗齿监控图像的隐含特征差分相似性比较来优化对于所述斗齿表面损伤的判定。
也就是，具体地，进一步计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度，例如余弦距离来得到相似度矩阵。然后，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型中进行特征提取，以提取出所述相似度矩阵中每两个所述斗齿监控图像的隐含特征的相似度的隐藏关联性特征，从而得到相似度特征矩阵。
进一步地，以所述各个预定时间点的斗齿局部特征向量作为节点的特征表示，而以所述相似度特征矩阵作为节点与节点之间的边的特征表示，将由所述多个斗齿局部特征向量经二维排列得到的所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵。具体地，所述图神经网络通过可学习的神经网络参数对所述全局斗齿特征矩阵和所述相似度特征矩阵进行图结构数据编码以得到包含相似度关联特征和所述各个预定时间点的斗齿隐含特征信息的所述相似度拓扑全局斗齿特征矩阵。然后，将所述优化相似度拓扑全局斗齿特征矩阵通过分类器就可以得到用于表示斗齿表面是否存在损伤的分类结果。
特别地，在本申请的技术方案中，这里，由于所述相似度拓扑全局斗齿特征矩阵是将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络得到的，因此所述相似度拓扑全局斗齿特征矩阵的各个行向量之间具有相似性拓扑关联关系。因此，当将所述相似度拓扑全局斗齿特征矩阵通过分类器进行分类时，容易发生类相干干涉。
因此，在本申请的技术方案中，优选地对所述相似度拓扑全局斗齿特征矩阵，例如记为M进行基于预分类的类概率相干补偿机制校正，表示为：
M′＝p p·M p-1⊙ e-p·M
其中p是所述相似度拓扑全局斗齿特征矩阵M通过分类器获得的概率值。
也就是，由于所述相似度拓扑全局斗齿特征矩阵M的各个行向量之间具有的相似性拓扑关联关系，在对其进行分类时，分类器本身的权重矩阵会在分类过程中对各个行向量具有类相干性，从而对所述相似度拓扑全局斗齿特征矩阵M造成类相干干涉。基于此，将通过预分类得到的分类器的类别概率值作为所述相似度拓扑全局斗齿特征矩阵M的乘性干扰噪声项，来对所述相似度拓扑全局斗齿特征矩阵M来进行类概率的相干补偿，可以恢复无干扰情况下的所述相似度拓扑全局斗齿特征矩阵M的等效概率强度表征，即优化后的所述相似度拓扑全局斗齿特征矩阵M′，从而实现所述相似度拓扑全局斗齿特征矩阵M的校正，提高了分类结果的准确性。这样，能够对于所述斗齿表面是否存在损伤进行准确地判断，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
基于此，本申请提出了一种基于图像识别的斗轮损伤判定方法，其包括：获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及，将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
图1图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的应用场景图。如图1所示，在该应用场景中，首先通过摄像头(例如，如图1中所示意的C)采集挖掘机斗齿(例如，如图1中所示意的F)的预定时间段内多个预定时间点的斗齿监控图像(例如，如图1中所示意的M)；然后，将采集的斗齿监控图像输入至部署有基于图像识别的斗轮损伤判定算法的服务器中(例如，如图1中所示意的S)，其中，所述服务器以基于图像识别的斗轮损伤判定算法对所述斗齿监控图像进行处理，以输出得到用于表示斗齿表面是否存在损伤的分类结果。
在介绍了本申请的基本原理之后，下面将参考附图来具体介绍本申请的各种非限制性实施例。
示例性方法
图2图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的流程图。如图2所示，根据本申请实施例的基于图像识别的斗轮损伤判定方法，包括：S110，获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；S120，将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；S130，计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；S140，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；S150，将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；S160，将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；S170，对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及，S180，将所述优化相似度拓扑全局斗齿特征矩阵通 过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
图3图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法的架构示意图。如图3所示，在所述基于图像识别的斗轮损伤判定方法的网络架构中，首先，获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；然后，将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；接着，计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；然后，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；接着，将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；然后，将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；接着，对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及，最后，将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
在步骤S110中，获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像。如前所述，现有的挖掘机上通过安装摄像头来采集图像进行处理，以判断斗齿是否存在损伤。但是，斗齿在工作的过程中保持着旋转状态，摄像头在每个预定时间点仅能采集到斗齿的局部区域的图像，这会导致对于斗齿损伤的判断不够精准。并且，由于在检测的过程中，斗齿表面会附着泥土等干扰对象以对于斗齿损伤的判断带来干扰。因此，期待一种优化的基于图像识别的斗轮损伤判定方案。
目前，深度学习以及神经网络已经广泛应用于计算机视觉、自然语言处理、语音信号处理等领域。此外，深度学习以及神经网络在图像分类、物体检测、语义分割、文本翻译等领域，也展现出了接近甚至超越人类的水平。
深度学习以及神经网络的发展为斗轮损伤的智能判定提供了新的解决思路和方案。
具体地，在本申请的技术方案中，通过采用基于深度学习的人工智能判定方法以利用图像识别技术来对于斗轮是否损伤进行精准判断。具体地，在此过程中，通过对于在时序维度上的多个斗齿监控图像中的隐含特征分布信息进行提取来得到所述斗齿在该时间点上的深层隐含特征，从而滤除泥土等干扰对象带来的识别影响。并且，进一步利用每两个所述斗齿监控图像的隐含特征差分相似性比较来基于所述斗齿的全局隐含特征信息来提高对于所述斗齿表面的损伤判定。这样，能够准确地对于所述斗齿表面是否存在损伤进行判断，进而能够及时有效地对于所述斗齿进行维修更换，以保证挖掘机的正常安全运行。
更具体地，在本申请的技术方案中，首先，通过摄像头采集预定时间段内多个预定时间点的斗齿监控图像。
在步骤S120中，将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量。应可以理解，在得到所述斗齿监控图像后，为了在斗齿损伤判断中滤除所述斗齿表面附着的泥土等干扰对象的影响，进一步将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型中进行特征挖掘，以提取出所述斗齿监控图像在高维特征空间中的局部隐含特征，从而得到多个斗齿局部特征向量。具体地，在本申请实施例中，图4图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法中，将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量的流程图，如图4所示，所述将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量，包括：使用所述作为过滤器的第一卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：S210，对所述输入数据进行卷积处理以得到卷积特征图；S220，对所述卷积特征图进行基于特征矩阵的均值池化处理以得到池化特征图；以及，S230，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为过滤器的第一卷积神经网络模型的最后一层的输 出为所述多个斗齿局部特征向量，所述作为过滤器的第一卷积神经网络模型的第一层的输入为所述各个预定时间点的斗齿监控图像。
在步骤S130中，计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵。由于所述斗齿在工作过程中保持旋转状态，因此，所述摄像头在每个预定时间点仅能采集到所述斗齿的局部区域的图像。并且，考虑到在对于所述斗齿表面损伤的判断过程中，由于该所述斗齿表面的损伤是存在局部特性的，但是需要基于所述斗齿表面的整体来进行全局检测。
因此，在本申请的技术方案中，进一步以每两个所述斗齿监控图像的隐含特征差分相似性比较来优化对于所述斗齿表面损伤的判定。也就是，具体地，进一步计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度，例如余弦距离来得到相似度矩阵。
进一步地，所述计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵，包括：以如下公式来计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到多个相似度；其中，所述公式为：
Figure PCTCN2022138434-appb-000004
其中V i和V j分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量，
Figure PCTCN2022138434-appb-000005
和
Figure PCTCN2022138434-appb-000006
分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量的各个位置的特征值，d(V i，V j)表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度；以及，将所述多个相似度进行二维排列以得到所述相似度矩阵。
在步骤S140中，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵。也就是，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型中进行特征提取，以提取出所述相似度矩阵中每两个所述斗齿监控图像的隐含特征的相似度的隐藏关联性特征，从而得到相似度特征矩阵。
具体地，在本申请实施例中，图5图示了根据本申请实施例的基于图像识别的斗轮损伤判定方法中，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵的流程图，如图5所示，所述将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵，包括：使用所述作为特征提取器的第二卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：S310，对所述输入数据进行卷积处理以得到卷积特征图；S320，对所述卷积特征图进行沿通道维度的池化处理以得到池化特征图；以及，S330，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为特征提取器的第二卷积神经网络模型的最后一层的输出为所述相似度特征矩阵，所述作为特征提取器的第二卷积神经网络模型的第一层的输入为所述相似度矩阵。
在步骤S150和步骤S160中，将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵，然后，将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵。进一步地，以所述各个预定时间点的斗齿局部特征向量作为节点的特征表示，而以所述相似度特征矩阵作为节点与节点之间的边的特征表示，将由所述多个斗齿局部特征向量经二维排列得到的所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵。
具体地，所述图神经网络通过可学习的神经网络参数对所述全局斗齿特征矩阵和所述 相似度特征矩阵进行图结构数据编码以得到包含相似度关联特征和所述各个预定时间点的斗齿隐含特征信息的所述相似度拓扑全局斗齿特征矩阵。然后，将所述优化相似度拓扑全局斗齿特征矩阵通过分类器就可以得到用于表示斗齿表面是否存在损伤的分类结果。
在步骤S170中，对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵。特别地，在本申请的技术方案中，这里，由于所述相似度拓扑全局斗齿特征矩阵是将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络得到的，因此所述相似度拓扑全局斗齿特征矩阵的各个行向量之间具有相似性拓扑关联关系。因此，当将所述相似度拓扑全局斗齿特征矩阵通过分类器进行分类时，容易发生类相干干涉。
因此，在本申请的技术方案中，优选地对所述相似度拓扑全局斗齿特征矩阵，例如记为M进行基于预分类的类概率相干补偿机制校正，也就是，以如下公式对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到所述优化相似度拓扑全局斗齿特征矩阵；其中，所述公式为：
M′＝p p·M p-1⊙ e-p·M
其中M表示所述相似度拓扑全局斗齿特征矩阵，M′表示所述优化相似度拓扑全局斗齿特征矩阵，p表示所述相似度拓扑全局斗齿特征矩阵通过所述分类器获得的概率值，⊙表示按位置点乘。
也就是，由于所述相似度拓扑全局斗齿特征矩阵M的各个行向量之间具有的相似性拓扑关联关系，在对其进行分类时，分类器本身的权重矩阵会在分类过程中对各个行向量具有类相干性，从而对所述相似度拓扑全局斗齿特征矩阵M造成类相干干涉。基于此，将通过预分类得到的分类器的类别概率值作为所述相似度拓扑全局斗齿特征矩阵M的乘性干扰噪声项，来对所述相似度拓扑全局斗齿特征矩阵M来进行类概率的相干补偿，可以恢复无干扰情况下的所述相似度拓扑全局斗齿特征矩阵M的等效概率强度表征，即优化后的所述相似度拓扑全局斗齿特征矩阵M′，从而实现所述相似度拓扑全局斗齿特征矩阵M的校正，提高了分类结果的准确性。这样，能够对于所述斗齿表面是否存在损伤进行准确地判断，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
在步骤S180中，将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
具体地，在本申请实施例中，使用所述分类器以如下公式对所述优化相似度拓扑全局斗齿特征矩阵进行处理以生成分类结果，其中，所述公式为：softmax{(W n，B n)：...：(W 1，B 1)|Project(F)}，其中Project(F)表示将所述优化相似度拓扑全局斗齿特征矩阵投影为向量，W 1至W n为各层全连接层的权重矩阵，B 1至B n表示各层全连接层的偏置矩阵。
进一步地，所述基于图像识别的斗轮损伤判定方法，还包括：从所述优化相似度拓扑全局斗齿特征矩阵提取待评估斗齿对应的行向量作为分类特征向量；以及，将所述分类特征向量通过所述分类器以得到用于表示待评估斗齿是否发生损坏的第二分类结果。
综上，基于本申请实施例的基于图像识别的斗轮损伤判定方法及其系统被阐明，其通过作为过滤器的第一卷积神经网络模型提取预定时间段内多个预定时间点的斗齿监控图像在高维特征空间中的局部隐含特征；然后，利用每两个斗齿监控图像的隐含特征差分相似性比较基于斗齿监控图像的全局隐含特征信息来对于斗齿表面是否存在损伤进行判定。这样，可以对于斗齿表面是否存在损伤进行准确地判定，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
示例性系统
图6图示了根据本申请实施例的基于图像识别的斗轮损伤判定系统的框图。如图6所示，根据本申请实施例的基于图像识别的斗轮损伤判定系统100，包括：图像采集模块110，用于获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；斗齿局部特征提取模块120，用于将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；相似度计算模块130，用于计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；相似度特征提取模块140，用于将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；二维排列模块150，用于将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；图结构数据编码模块160，用于将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；特征补偿模块170，用于对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及，损伤判定结果生成模块180，用于将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，所述斗齿局部特征提取模块，用于：使用所述作为过滤器的第一卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：对所述输入数据进行卷积处理以得到卷积特征图；对所述卷积特征图进行基于特征矩阵的均值池化处理以得到池化特征图；以及，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为过滤器的第一卷积神经网络模型的最后一层的输出为所述多个斗齿局部特征向量，所述作为过滤器的第一卷积神经网络模型的第一层的输入为所述各个预定时间点的斗齿监控图像。
如前所述，现有的挖掘机上通过安装摄像头来采集图像进行处理，以判断斗齿是否存在损伤。但是，斗齿在工作的过程中保持着旋转状态，摄像头在每个预定时间点仅能采集到斗齿的局部区域的图像，这会导致对于斗齿损伤的判断不够精准。并且，由于在检测的过程中，斗齿表面会附着泥土等干扰对象以对于斗齿损伤的判断带来干扰。因此，期待一种优化的基于图像识别的斗轮损伤判定方案。
具体地，在本申请的技术方案中，通过采用基于深度学习的人工智能判定方法以利用图像识别技术来对于斗轮是否损伤进行精准判断。具体地，在此过程中，通过对于在时序维度上的多个斗齿监控图像中的隐含特征分布信息进行提取来得到所述斗齿在该时间点上的深层隐含特征，从而滤除泥土等干扰对象带来的识别影响。并且，进一步利用每两个所述斗齿监控图像的隐含特征差分相似性比较来基于所述斗齿的全局隐含特征信息来提高对于所述斗齿表面的损伤判定。这样，能够准确地对于所述斗齿表面是否存在损伤进行判断，进而能够及时有效地对于所述斗齿进行维修更换，以保证挖掘机的正常安全运行。
具体地，在本申请的技术方案中，首先，通过摄像头采集预定时间段内多个预定时间点的斗齿监控图像。然后，为了在斗齿损伤判断中滤除所述斗齿表面附着的泥土等干扰对象的影响， 进一步将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型中进行特征挖掘，以提取出所述斗齿监控图像在高维特征空间中的局部隐含特征，从而得到多个斗齿局部特征向量。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，所述相似度计算模块，用于：以如下公式来计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到多个相似度；其中，所述公式为：
Figure PCTCN2022138434-appb-000007
其中V i和V j分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量，
Figure PCTCN2022138434-appb-000008
和
Figure PCTCN2022138434-appb-000009
分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量的各个位置的特征值，d(V i，V j)表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度；以及，将所述多个相似度进行二维排列以得到所述相似度矩阵。
接着，由于所述斗齿在工作过程中保持旋转状态，因此，所述摄像头在每个预定时间点仅能采集到所述斗齿的局部区域的图像。并且，考虑到在对于所述斗齿表面损伤的判断过程中，由于该所述斗齿表面的损伤是存在局部特性的，但是需要基于所述斗齿表面的整体来进行全局检测。因此，在本申请的技术方案中，进一步以每两个所述斗齿监控图像的隐含特征差分相似性比较来优化对于所述斗齿表面损伤的判定。
也就是，具体地，进一步计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度，例如余弦距离来得到相似度矩阵。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，所述相似度特征提取模块，用于：使用所述作为特征提取器的第二卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：对所述输入数据进行卷积处理以得到卷积特征图；对所述卷积特征图进行沿通道维度的池化处理以得到池化特征图；以及，对所述池化特征图进行非线性激活以得到激活特征图；其中，所述作为特征提取器的第二卷积神经网络模型的最后一层的输出为所述相似度特征矩阵，所述作为特征提取器的第二卷积神经网络模型的第一层的输入为所述相似度矩阵。
也就是，得到所述相似度矩阵后，接着，将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型中进行特征提取，以提取出所述相似度矩阵中每两个所述斗齿监控图像的隐含特征的相似度的隐藏关联性特征，从而得到相似度特征矩阵。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，所述特征补偿模块，用于：以如下公式对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到所述优化相似度拓扑全局斗齿特征矩阵；其中，所述公式为：
M′＝p p·M p-1⊙ e-p·M
其中M表示所述相似度拓扑全局斗齿特征矩阵，M′表示所述优化相似度拓扑全局斗齿特征矩阵，p表示所述相似度拓扑全局斗齿特征矩阵通过所述分类器获得的概率值，⊙表示按位置点乘。
特别地，在本申请的技术方案中，这里，由于所述相似度拓扑全局斗齿特征矩阵是将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络得到的，因此所述相似度拓扑全局斗齿特征矩阵的各个行向量之间具有相似性拓扑关联关系。因此，当将所述相似度拓扑全局斗齿特征矩阵通过分类器进行分类时，容易发生类相干干涉。
因此，在本申请的技术方案中，优选地对所述相似度拓扑全局斗齿特征矩阵，例如记为M进行基于预分类的类概率相干补偿机制校正，也就是，由于所述相似度拓扑全局斗齿特征矩阵M的各个行向量之间具有的相似性拓扑关联关系，在对其进行分类时，分类器本身的权重矩阵会在分类过程中对各个行向量具有类相干性，从而对所述相似度拓扑全局斗齿特征矩阵M造成类相干干涉。基于此，将通过预分类得到的分类器的类别概率值作为所述相似度拓扑全局斗齿特征矩阵M的乘性干扰噪声项，来对所述相似度拓扑全局斗齿特征矩阵M来进行类概率的相干补偿，可以恢复无干扰情况下的所述相似度拓扑全局斗齿特征矩阵M的等效概率强度表征，即优化后的所述相似度拓扑全局斗齿特征矩阵M′，从而实现所述相似度拓扑全局斗齿特征矩阵M的校正，提高了分类结果的准确性。这样，能够对于所述斗齿表面是否存在损伤进行准确地判断，进而能够及时有效地对于表面有损伤的斗齿进行维修更换，以保证挖掘机的正常安全运行。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，所述损伤判定结果生成模块，用于：使用所述分类器以如下公式对所述优化相似度拓扑全局斗齿特征矩阵进行处理以生成分类结果，其中，所述公式为：softmax{(W n，B n)：...：(W 1，B 1)|Project(F)}，其中Project(F)表示将所述优化相似度拓扑全局斗齿特征矩阵投影为向量，W 1至W n为各层全连接层的权重矩阵，B 1至B n表示各层全连接层的偏置矩阵。
在一个示例中，在上述基于图像识别的斗轮损伤判定系统100中，还包括：提取单元：用于从所述优化相似度拓扑全局斗齿特征矩阵提取待评估斗齿对应的行向量作为分类特征向量；以及，第二分类结果生成单元：用于将所述分类特征向量通过所述分类器以得到用于表示待评估斗齿是否发生损坏的第二分类结果。
这里，本领域技术人员可以理解，上述基于图像识别的斗轮损伤判定系统100中的各个单元和模块的具体功能和操作已经在上面参考图1到图5的基于图像识别的斗轮损伤判定方法描述中得到了详细介绍，并因此，将省略其重复描述。
如上所述，根据本申请实施例的基于图像识别的斗轮损伤判定系统100可以实现在各种终端设备中，例如用于基于图像识别的斗轮损伤判定的服务器等。在一个示例中，根据本申请实施例的基于图像识别的斗轮损伤判定系统100可以作为一个软件模块和/或硬件模块而集成到终端设备中。例如，该基于图像识别的斗轮损伤判定系统100可以是该终端设备的操作系统中的一个软件模块，或者可以是针对于该终端设备所开发的一个应用程序；当然，该基于图像识别的斗轮损伤判定系统100同样可以是该终端设备的众多硬件模块之一。
替换地，在另一示例中，该基于图像识别的斗轮损伤判定系统100与该终端设备也可以是分 立的设备，并且该基于图像识别的斗轮损伤判定系统100可以通过有线和/或无线网络连接到该终端设备，并且按照约定的数据格式来传输交互信息。
以上结合具体实施例描述了本申请的基本原理，但是，需要指出的是，在本申请中提及的优点、优势、效果等仅是示例而非限制，不能认为这些优点、优势、效果等是本申请的各个实施例必须具备的。另外，上述公开的具体细节仅是为了示例的作用和便于理解的作用，而非限制，上述细节并不限制本申请为必须采用上述具体的细节来实现。
本申请中涉及的器件、装置、设备、系统的方框图仅作为例示性的例子并且不意图要求或暗示必须按照方框图示出的方式进行连接、布置、配置。如本领域技术人员将认识到的，可以按任意方式连接、布置、配置这些器件、装置、设备、系统。诸如“包括”、“包含”、“具有”等等的词语是开放性词汇，指“包括但不限于”，且可与其互换使用。这里所使用的词汇“或”和“和”指词汇“和/或”，且可与其互换使用，除非上下文明确指示不是如此。这里所使用的词汇“诸如”指词组“诸如但不限于”，且可与其互换使用。
还需要指出的是，在本申请的装置、设备和方法中，各部件或各步骤是可以分解和/或重新组合的。这些分解和/或重新组合应视为本申请的等效方案。
提供所公开的方面的以上描述以使本领域的任何技术人员能够做出或者使用本申请。对这些方面的各种修改对于本领域技术人员而言是非常显而易见的，并且在此定义的一般原理可以应用于其他方面而不脱离本申请的范围。因此，本申请不意图被限制到在此示出的方面，而是按照与在此公开的原理和新颖的特征一致的最宽范围。

Claims (10)

  1. 一种基于图像识别的斗轮损伤判定方法，其特征在于，包括：
    获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；
    将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；
    计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；
    将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；
    将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；
    将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；
    对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及
    将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
  2. 根据权利要求1所述的基于图像识别的斗轮损伤判定方法，其特征在于，所述将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量，包括：使用所述作为过滤器的第一卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：
    对所述输入数据进行卷积处理以得到卷积特征图；
    对所述卷积特征图进行基于特征矩阵的均值池化处理以得到池化特征图；以及
    对所述池化特征图进行非线性激活以得到激活特征图；
    其中，所述作为过滤器的第一卷积神经网络模型的最后一层的输出为所述多个斗齿局部特征向量，所述作为过滤器的第一卷积神经网络模型的第一层的输入为所述各个预定时间点的斗齿监控图像。
  3. 根据权利要求2所述的基于图像识别的斗轮损伤判定方法，其特征在于，所述计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵，包括：以如下公式来计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到多个相似度；
    其中，所述公式为：
    Figure PCTCN2022138434-appb-100001
    其中V i和V j分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量，
    Figure PCTCN2022138434-appb-100002
    和
    Figure PCTCN2022138434-appb-100003
    分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量的各个位置的特征值，d(V i，V j)表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度；以及将所述多个相似度进行二维排列以得到所述相似度矩阵。
  4. 根据权利要求3所述的基于图像识别的斗轮损伤判定方法，其特征在于，所述将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵，包括：使用所述作为特征提取器的第二卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：
    对所述输入数据进行卷积处理以得到卷积特征图；
    对所述卷积特征图进行沿通道维度的池化处理以得到池化特征图；以及
    对所述池化特征图进行非线性激活以得到激活特征图；
    其中，所述作为特征提取器的第二卷积神经网络模型的最后一层的输出为所述相似度特征矩阵，所述作为特征提取器的第二卷积神经网络模型的第一层的输入为所述相似度矩阵。
  5. 根据权利要求4所述的基于图像识别的斗轮损伤判定方法，其特征在于，所述对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵，包括：
    以如下公式对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到所述优化相似度拓扑全局斗齿特征矩阵；
    其中，所述公式为：
    M′＝p p·M p-1⊙e -p·M
    其中M表示所述相似度拓扑全局斗齿特征矩阵，M′表示所述优化相似度拓扑全局斗齿特征矩阵，p表示所述相似度拓扑全局斗齿特征矩阵通过所述分类器获得的概率值，⊙表示按位置点乘。
  6. 根据权利要求5所述的基于图像识别的斗轮损伤判定方法，其特征在于，所述将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，包括：使用所述分类器以如下公式对所述优化相似度拓扑全局斗齿特征矩阵进行处理以生成分类结果，其中，所述公式为：softmax{(W n，B n)：...：(W 1，B 1)|Project(F)}，其中Project(F)表示将所述优化相似度拓扑全局斗齿特征矩阵投影为向量，W 1至W n为各层全连接层的权重矩阵，B 1至B n表示各层全连接层的偏置矩阵。
  7. 根据权利要求6所述基于图像识别的斗轮损伤判定方法，其特征在于，还包括：
    从所述优化相似度拓扑全局斗齿特征矩阵提取待评估斗齿对应的行向量作为分类特征向量；以及
    将所述分类特征向量通过所述分类器以得到用于表示待评估斗齿是否发生损坏的第二分类结果。
  8. 一种基于图像识别的斗轮损伤判定系统，其特征在于，包括：
    图像采集模块，用于获取由摄像头采集的预定时间段内多个预定时间点的斗齿监控图像；斗齿局部特征提取模块，用于将所述各个预定时间点的斗齿监控图像分别通过作为过滤器的第一卷积神经网络模型以得到多个斗齿局部特征向量；
    相似度计算模块，用于计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到相似度矩阵；
    相似度特征提取模块，用于将所述相似度矩阵通过作为特征提取器的第二卷积神经网络模型以得到相似度特征矩阵；
    二维排列模块，用于将所述多个斗齿局部特征向量进行二维排列以得到全局斗齿特征矩阵；图结构数据编码模块，用于将所述全局斗齿特征矩阵和所述相似度特征矩阵通过图神经网络以得到相似度拓扑全局斗齿特征矩阵；
    特征补偿模块，用于对所述相似度拓扑全局斗齿特征矩阵进行基于预分类的特征补偿以得到优化相似度拓扑全局斗齿特征矩阵；以及
    损伤判定结果生成模块，用于将所述优化相似度拓扑全局斗齿特征矩阵通过分类器以得到分类结果，所述分类结果用于表示斗齿表面是否存在损伤。
  9. 根据权利要求8所述的基于图像识别的斗轮损伤判定系统，其特征在于，所述斗齿局部特征提取模块，用于：使用所述作为过滤器的第一卷积神经网络模型的各层在层的正向传递中分别对输入数据进行：
    对所述输入数据进行卷积处理以得到卷积特征图；
    对所述卷积特征图进行基于特征矩阵的均值池化处理以得到池化特征图；以及
    对所述池化特征图进行非线性激活以得到激活特征图；
    其中，所述作为过滤器的第一卷积神经网络模型的最后一层的输出为所述多个斗齿局部特征向量，所述作为过滤器的第一卷积神经网络模型的第一层的输入为所述各个预定时间点的斗齿监控图像。
  10. 根据权利要求9所述的基于图像识别的斗轮损伤判定系统，其特征在于，所述相似度计算模块，用于：以如下公式来计算所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度以得到多个相似度；
    其中，所述公式为：
    Figure PCTCN2022138434-appb-100004
    其中V i和V j分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量，
    Figure PCTCN2022138434-appb-100005
    和
    Figure PCTCN2022138434-appb-100006
    分别表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量的各个位置的特征值，d(V i，V j)表示所述多个斗齿局部特征向量中每两个斗齿局部特征向量之间的相似度；以及将所述多个相似度进行二维排列以得到所述相似度矩阵。
PCT/CN2022/138434 2022-09-28 2022-12-12 基于图像识别的斗轮损伤判定方法及其系统 WO2024066061A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211189669.0 2022-09-28
CN202211189669.0A CN115620270A (zh) 2022-09-28 2022-09-28 基于图像识别的斗轮损伤判定方法及其系统

Publications (1)

Publication Number Publication Date
WO2024066061A1 true WO2024066061A1 (zh) 2024-04-04

Family

ID=84861364

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/138434 WO2024066061A1 (zh) 2022-09-28 2022-12-12 基于图像识别的斗轮损伤判定方法及其系统

Country Status (2)

Country Link
CN (1) CN115620270A (zh)
WO (1) WO2024066061A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100142759A1 (en) * 2006-05-12 2010-06-10 Alberta Research Council Inc. A system and a method for detecting a damaged or missing machine part
CN112378564A (zh) * 2020-11-16 2021-02-19 北京航空航天大学 矿用正铲挖掘机空间斗齿实时故障监测系统及其识别方法
CN112686206A (zh) * 2021-01-20 2021-04-20 塞尔昂(上海)工业技术有限公司 一种智能斗齿监测系统
CN114419671A (zh) * 2022-01-18 2022-04-29 北京工业大学 一种基于超图神经网络的遮挡行人重识别方法
US20220180126A1 (en) * 2020-12-03 2022-06-09 Ping An Technology (Shenzhen) Co., Ltd. Method, device, and computer program product for self-supervised learning of pixel-wise anatomical embeddings in medical images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100142759A1 (en) * 2006-05-12 2010-06-10 Alberta Research Council Inc. A system and a method for detecting a damaged or missing machine part
CN112378564A (zh) * 2020-11-16 2021-02-19 北京航空航天大学 矿用正铲挖掘机空间斗齿实时故障监测系统及其识别方法
US20220180126A1 (en) * 2020-12-03 2022-06-09 Ping An Technology (Shenzhen) Co., Ltd. Method, device, and computer program product for self-supervised learning of pixel-wise anatomical embeddings in medical images
CN112686206A (zh) * 2021-01-20 2021-04-20 塞尔昂(上海)工业技术有限公司 一种智能斗齿监测系统
CN114419671A (zh) * 2022-01-18 2022-04-29 北京工业大学 一种基于超图神经网络的遮挡行人重识别方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIU GUOSHUAI; ZHONG WEIFENG; YIN FEI;LIU CHENGLIN: "Fast classification of natural scene and born-digital images", JOURNAL OF IMAGE AND GRAPHICS, ZHONGGUO TUXIANG TUXING XUEHUI, CN, vol. 22, no. 05, 16 May 2017 (2017-05-16), CN , pages 678 - 687, XP009553467, ISSN: 1006-8961, DOI: 10.11834/jig.160597 *

Also Published As

Publication number Publication date
CN115620270A (zh) 2023-01-17

Similar Documents

Publication Publication Date Title
Wang et al. A unified convolutional neural network integrated with conditional random field for pipe defect segmentation
WO2019080367A1 (zh) 一种机械设备健康状态评估方法
Desa et al. Image subtraction for real time moving object extraction
CN109145516B (zh) 一种基于改进型极限学习机的模拟电路故障识别方法
CN102521534B (zh) 一种基于粗糙熵属性约简的入侵检测方法
JP2017068815A (ja) アテンション検出装置及びアテンション検出方法
Hong et al. The entropy and PCA based anomaly prediction in data streams
WO2018107492A1 (zh) 基于直觉模糊随机森林的目标跟踪方法及装置
CN115935147A (zh) 低秩和稀疏张量表示的交通数据恢复和异常值检测方法
CN111222583B (zh) 一种基于对抗训练与关键路径提取的图像隐写分析方法
CN113343123B (zh) 一种生成对抗多关系图网络的训练方法和检测方法
WO2024066061A1 (zh) 基于图像识别的斗轮损伤判定方法及其系统
Kumari et al. YOLOv8 Based Deep Learning Method for Potholes Detection
CN117313015A (zh) 一种基于时序和多变量的时间序列异常检测方法及系统
CN106778252B (zh) 基于粗糙集理论与waode算法的入侵检测方法
JP7078295B2 (ja) 変状検出装置、変状検出方法、及びプログラム
CN117176433A (zh) 网络数据的异常行为检测系统及方法
CN116680639A (zh) 一种基于深度学习的深海潜水器传感器数据的异常检测方法
CN116362120A (zh) 一种基于mgcn-gru模型的交通流预测方法及装置
Nguyen et al. Deep learning for construction emission monitoring with low-cost sensor network
CN114580534A (zh) 一种工业数据异常检测方法、装置、电子设备和存储介质
CN113469816A (zh) 基于多组学技术的数字货币识别方法、系统和存储介质
CN112527784A (zh) 基于复杂网络的异常模式挖掘和增量式异常检测方法
WO2020213106A1 (en) Image processing apparatus, image processing method and non-transitoty computer readable medium
Kembhavi et al. Low-cost image processing system for evaluating pavement surface distress

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22960656

Country of ref document: EP

Kind code of ref document: A1