CN116402777B

CN116402777B - Power equipment detection method and system based on machine vision

Info

Publication number: CN116402777B
Application number: CN202310327206.4A
Authority: CN
Inventors: 张宇飞; 郭长周; 沈文江; 桑林梅; 鲍宏伟; 闫娇; 李晓东; 董芳瑞; 张霄; 陆世杰; 李亚楠; 陈龙; 王远; 李强; 孙浩然; 马永星; 常立庆; 刘娟; 张林梅; 元亮
Original assignee: Linzhou Power Supply Co Of State Grid Henan Electric Power Co; Anyang Power Supply Co of State Grid Henan Electric Power Co Ltd
Current assignee: Linzhou Power Supply Co Of State Grid Henan Electric Power Co; Anyang Power Supply Co of State Grid Henan Electric Power Co Ltd
Priority date: 2023-03-30
Filing date: 2023-03-30
Publication date: 2023-10-24
Anticipated expiration: 2043-03-30
Also published as: CN116402777A

Abstract

A power equipment detection method and system based on machine vision are disclosed. Firstly, performing image blocking processing on a thermal infrared image of power equipment to be detected to obtain a sequence of thermal infrared image blocks, respectively passing each thermal infrared image block through a first convolution neural network model to obtain a plurality of thermal infrared image block feature vectors, then constructing a space topology matrix of the sequence of the thermal infrared image blocks, passing the space topology matrix through a second convolution neural network model to obtain a space topology feature matrix, arranging the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix, and then passing the thermal infrared image block global feature matrix and the space topology feature matrix through the graph neural network model to obtain a topology thermal infrared image block global feature matrix, and finally, passing the topology thermal infrared image block global feature matrix through a classifier to obtain a classification result for indicating whether the working state of the power equipment to be detected is normal. Thus, the efficiency of power equipment state detection can be improved.

Description

Power equipment detection method and system based on machine vision

Technical Field

The application relates to the field of intelligent detection, in particular to a machine vision-based power equipment detection method and system.

Background

Safe and stable operation of the power equipment is a key for ensuring reliable power supply of the power grid. By detecting the operation state of the power equipment by inspecting the equipment, the accident caused by the equipment defect or fault can be prevented. The infrared thermal imaging technology provides a non-contact detection mode to acquire the thermal state information of the power equipment, so that the state detection of the power equipment can be performed under the condition of no power failure, and the method is widely applied to the live detection of the power equipment. The infrared image of the power equipment can display the temperature distribution and range, and the temperature of different parts of the equipment is represented by different levels of hue and brightness changes. However, the acquired infrared image data of the power equipment still needs to be analyzed and diagnosed by an electric engineer with abundant experience, so that a great deal of labor and time cost are consumed, and the efficiency of detecting and evaluating the state of the power equipment is greatly reduced.

Thus, an optimized power device detection scheme is desired.

Disclosure of Invention

The present application has been made to solve the above-mentioned technical problems. The embodiment of the application provides a machine vision-based power equipment detection method and system. Firstly, performing image blocking processing on a thermal infrared image of power equipment to be detected to obtain a sequence of thermal infrared image blocks, respectively passing each thermal infrared image block through a first convolution neural network model to obtain a plurality of thermal infrared image block feature vectors, then constructing a space topology matrix of the sequence of the thermal infrared image blocks, passing the space topology matrix through a second convolution neural network model to obtain a space topology feature matrix, arranging the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix, and then passing the thermal infrared image block global feature matrix and the space topology feature matrix through the graph neural network model to obtain a topology thermal infrared image block global feature matrix, and finally, passing the topology thermal infrared image block global feature matrix through a classifier to obtain a classification result for indicating whether the working state of the power equipment to be detected is normal. Thus, the efficiency of power equipment state detection can be improved.

According to one aspect of the present application, there is provided a machine vision-based power equipment detection system, comprising:

the thermal infrared monitoring module is used for acquiring a thermal infrared image of the electric equipment to be detected;

the image segmentation module is used for carrying out image blocking processing on the thermal infrared image of the power equipment to be detected so as to obtain a sequence of thermal infrared image blocks;

the image block feature extraction module is used for respectively passing each thermal infrared image block in the sequence of the thermal infrared image blocks through a first convolution neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors;

the image block topology construction module is used for constructing a space topology matrix of the sequence of the thermal infrared image blocks, wherein the values of all positions in the space topology matrix are used for representing Euclidean distances between the two corresponding thermal infrared image blocks;

the space topology feature extraction module is used for enabling the space topology matrix to pass through a second convolution neural network model serving as a feature extractor to obtain a space topology feature matrix;

the image block feature global module is used for arranging the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix;

The image data coding module is used for enabling the thermal infrared image block global feature matrix and the space topology feature matrix to pass through an image neural network model to obtain a topology thermal infrared image block global feature matrix; and

the detection result generation module is used for enabling the topological thermal infrared image block global feature matrix to pass through a classifier to obtain a classification result, and the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not.

In the above power equipment detection system based on machine vision, the image block feature extraction module is configured to:

and respectively carrying out convolution processing, mean pooling processing based on a feature matrix and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model as a filter to output the plurality of thermal infrared image block feature vectors from the last layer of the first convolution neural network model as the filter, wherein the input of the first layer of the first convolution neural network model as the filter is each thermal infrared image block in the sequence of the thermal infrared image blocks.

In the above power equipment detection system based on machine vision, the image block topology construction module includes:

The space measurement unit is used for calculating Euclidean distances between every two thermal infrared image blocks in the sequence of the thermal infrared image blocks to obtain a plurality of Euclidean distances; and

and a matrix construction unit configured to construct the plurality of euclidean distances into the spatial topology matrix.

In the above power equipment detection system based on machine vision, the spatial topological feature extraction module is configured to:

and respectively carrying out two-dimensional convolution processing, feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a feature extractor to output the spatial topology feature matrix by the last layer of the second convolution neural network model serving as the feature extractor, wherein the input of the first layer of the second convolution neural network model serving as the feature extractor is the spatial topology matrix.

In the above power equipment detection system based on machine vision, the graph data encoding module is further configured to encode the global feature matrix of the thermal infrared image block and the spatial topological feature matrix with learnable neural network parameters using the graph neural network model to obtain the global feature matrix of the topological thermal infrared image block.

The power equipment detection system based on machine vision further comprises a training module for training the first convolutional neural network model serving as a filter, the second convolutional neural network model serving as a feature extractor, the graph neural network model and the classifier.

In the above power equipment detection system based on machine vision, the training module includes:

the training data acquisition unit is used for acquiring a training thermal infrared image of the power equipment to be detected and a real label value of the training thermal infrared image, wherein the real label value is used for indicating whether the working state of the power equipment to be detected is normal or not;

the training image segmentation unit is used for carrying out image blocking processing on the training thermal infrared image to obtain a sequence of training thermal infrared image blocks;

the training image block feature extraction unit is used for respectively enabling each training thermal infrared image block in the training thermal infrared image block sequence to pass through the first convolution neural network model serving as a filter so as to obtain a plurality of training thermal infrared image block feature vectors;

the training image block topology construction unit is used for constructing a space topology matrix of the sequence of the training thermal infrared image blocks;

The training space topology feature extraction unit is used for enabling the space topology matrix of the sequence of the training thermal infrared image blocks to pass through the second convolution neural network model serving as the feature extractor to obtain a training space topology feature matrix;

the training image block feature global unit is used for arranging the plurality of training thermal infrared image block feature vectors into a training thermal infrared image block global feature matrix;

the training image data coding unit is used for enabling the training thermal infrared image block global feature matrix and the training space topological feature matrix to pass through the image neural network model to obtain a training topological thermal infrared image block global feature matrix;

the classification loss unit is used for enabling the training topological thermal infrared image block global feature matrix to pass through the classifier to obtain a classification loss function value; and

and the training unit is used for training the first convolutional neural network model serving as a filter, the second convolutional neural network model serving as a feature extractor, the graph neural network model and the classifier by taking the classification loss function value as a loss function value.

In the above power equipment detection system based on machine vision, the classification loss unit is configured to:

Processing the training topological thermal infrared image block global feature matrix by using the classifier according to the following classification loss formula to obtain the classification loss function value;

wherein, the classification loss formula is:

softmax{(M _c ,B _c )|Project(F)}

wherein Project (F) represents projecting the training topology thermal infrared image block global feature matrix as a vector, M _c Weight matrix of full connection layer, B _c Representing the bias matrix of the fully connected layer.

In the above power equipment detection system based on machine vision, in the process of training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model and the classifier with the classification loss function value as a loss function value, performing spatial regularization constraint on the weight matrix of the classifier with the following iterative formula at each iteration of the weight matrix of the classifier;

wherein, the iterative formula is:

wherein M is the weight matrix of the classifier, M' represents the weight matrix of the classifier after iterative training, (-) ^T The transposed matrix of the matrix is represented, I.I _F Fr representing matrix obenius norm, M _b Is a bias matrix, exp (·) represents the exponential operation of the matrix, which represents the calculation of the natural exponential function value raised to a power by the eigenvalues of each position in the matrix,and->Representing addition by location and multiplication by location, respectively.

According to another aspect of the present application, there is provided a machine vision-based power equipment detection method, including:

acquiring a thermal infrared image of the power equipment to be detected;

performing image blocking processing on the thermal infrared image of the power equipment to be detected to obtain a sequence of thermal infrared image blocks;

respectively passing each thermal infrared image block in the sequence of the thermal infrared image blocks through a first convolution neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors;

constructing a space topology matrix of the sequence of thermal infrared image blocks, wherein values of all positions in the space topology matrix are used for representing Euclidean distances between two corresponding thermal infrared image blocks;

the space topology matrix is passed through a second convolution neural network model serving as a feature extractor to obtain a space topology feature matrix;

arranging the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix;

The global feature matrix of the thermal infrared image block and the spatial topological feature matrix are processed through a graph neural network model to obtain the global feature matrix of the topological thermal infrared image block; and

and the global feature matrix of the topological thermal infrared image block passes through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not.

Compared with the prior art, the machine vision-based power equipment detection method and system provided by the application have the advantages that firstly, the thermal infrared image of the power equipment to be detected is subjected to image blocking processing to obtain each thermal infrared image block in the sequence of thermal infrared image blocks, the thermal infrared image blocks respectively pass through a first convolution neural network model to obtain a plurality of thermal infrared image block feature vectors, then, a space topology matrix of the sequence of thermal infrared image blocks is constructed and passes through a second convolution neural network model to obtain a space topology feature matrix, then, the thermal infrared image block feature vectors are arranged into a thermal infrared image block global feature matrix, the thermal infrared image block global feature matrix is obtained through the graph neural network model with the space topology feature matrix, and finally, the topology thermal infrared image block global feature matrix is passed through a classifier to obtain a classification result for indicating whether the working state of the power equipment to be detected is normal. Thus, the efficiency of power equipment state detection can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art. The following drawings are not intended to be drawn to scale, emphasis instead being placed upon illustrating the principles of the application.

Fig. 1 is an application scenario diagram of a machine vision-based power device detection system according to an embodiment of the present application.

Fig. 2 is a block diagram schematic diagram of a machine vision-based power device detection system according to an embodiment of the present application.

Fig. 3 is a block diagram schematic diagram of the image block topology module in the machine vision-based power equipment detection system according to an embodiment of the present application.

Fig. 4 is a block diagram schematic diagram of the training module in the machine vision-based power equipment detection system according to an embodiment of the present application.

Fig. 5 is a flowchart of a machine vision-based power device detection method according to an embodiment of the present application.

Fig. 6 is a schematic diagram of a system architecture of a machine vision-based power device detection method according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are also within the scope of the application.

As used in the specification and in the claims, the terms "a," "an," "the," and/or "the" are not specific to a singular, but may include a plurality, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.

Although the present application makes various references to certain modules in a system according to embodiments of the present application, any number of different modules may be used and run on a user terminal and/or server. The modules are merely illustrative, and different aspects of the systems and methods may use different modules.

A flowchart is used in the present application to describe the operations performed by a system according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in order precisely. Rather, the various steps may be processed in reverse order or simultaneously, as desired. Also, other operations may be added to or removed from these processes.

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

It should be understood that in the technical solution of the present application, the infrared image of the power device can display its temperature distribution and range, and the temperatures of different parts of the device are represented by different levels of hue and brightness variation. However, analysis and diagnosis of the operating state of the power equipment based on the infrared image data of the power equipment still need to rely on experienced power engineers.

In recent years, deep learning and neural networks have been widely used in the fields of computer vision, natural language processing, text signal processing, and the like. In addition, deep learning and neural networks have also shown levels approaching and even exceeding humans in the fields of image classification, object detection, semantic segmentation, text translation, and the like.

Deep learning and neural networks provide new solutions and schemes for electronic device detection based on thermal infrared images.

Accordingly, in the technical scheme of the application, firstly, the thermal infrared camera is used for collecting the thermal infrared image of the power equipment to be detected, and it is understood that the infrared image of the power equipment can display the temperature distribution and range of the power equipment, and the temperature of different parts of the equipment is represented by different levels of color tone and brightness change. It should be understood that, in the technical solution of the present application, the temperature distribution and the range of the electronic device can represent whether the working state of the electronic device is normal, but the mapping relationship between the temperature distribution and the range characteristics of the electronic device and whether the state of the electronic device is normal is complex and nonlinear, and is difficult to be characterized by the existing statistical model or the machine learning model. At the same time, the temperature distribution and range characteristics of the electronic device are also difficult to characterize by conventional statistical models or machine learning models.

Specifically, in the technical scheme of the application, after the thermal infrared image of the power equipment to be detected is obtained, the thermal infrared image of the power equipment to be detected is subjected to image blocking processing to obtain a sequence of thermal infrared image blocks. Particularly, in the technical scheme of the application, in the process of carrying out image blocking processing, only the region with similar temperature distribution is possibly segmented into one image block, that is, in the technical scheme of the application, the image block can be carried out based on the pixel value of each pixel position of the thermal infrared image to obtain the sequence of the thermal infrared image blocks, so that each thermal infrared image block in the sequence of the thermal infrared image blocks can relatively completely reflect the temperature distribution and the range of a certain part of the power equipment to be detected.

After the sequence of the thermal infrared image blocks is obtained, each thermal infrared image block in the sequence of the thermal infrared image blocks is respectively passed through a first convolution neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors. That is, a convolutional neural network model with excellent performance in the field of image feature extraction is used as a feature filter to capture a high-dimensional implicit feature representation of the temperature distribution and range features of the individual thermal infrared image blocks.

In particular, in the technical solution of the present application, it is considered that the temperature distribution and the range characteristics of each thermal infrared image block in the sequence of thermal infrared image blocks are not completely independent, but there is a spatial correlation. That is, the temperature distribution and the range characteristics of each part of the electronic device to be detected are not completely independent, for example, the thermal influence between the parts with a smaller distance is more obvious.

Based on the above, in the technical scheme of the application, a space topology matrix of the sequence of the thermal infrared image blocks is constructed,

wherein the values of each position in the spatial topology matrix are used to represent the Euclidean distance between the corresponding two thermal infrared image blocks. Here, the calculation process of the euclidean distance between the two thermal infrared image blocks is: and extracting the center point coordinates of the first thermal infrared image block and the center point coordinates of the second thermal infrared image block, and calculating the Euclidean distance between the center point coordinates of the first thermal infrared image block and the center point coordinates of the second thermal infrared image block as the Euclidean distance between the two thermal infrared image blocks. Correspondingly, after the Euclidean distance between every two thermal infrared image blocks is obtained, the obtained Euclidean distance values are arranged into the space topology matrix.

The spatial topology matrix is then passed through a second convolutional neural network model as a feature extractor to obtain a spatial topology feature matrix. That is, in the technical scheme of the present application, a convolutional neural network model having excellent performance in the field of local feature extraction is used as a feature extractor to capture spatial topological features of each thermal infrared image block.

It should be noted that in the solution of the present application, if the thermal infrared image block feature vector of each thermal infrared image block is regarded as a node, the spatial topology feature matrix represents a high-dimensional feature representation of the edge between each node. Therefore, in the technical scheme of the application, the plurality of thermal infrared image block feature vectors and the spatial topological feature matrix form a high-dimensional embedded representation of the graph data. Furthermore, in the technical scheme of the application, the plurality of thermal infrared image block feature vectors are arranged into a thermal infrared image block global feature matrix, and then the thermal infrared image block global feature matrix and the spatial topological feature matrix are passed through a graph neural network model to obtain a topological thermal infrared image block global feature matrix. Here, the map neural network model processes the thermal infrared image block global feature matrix and the spatial topological feature matrix through the learnable neural network parameters to obtain the topological thermal infrared image block global feature matrix containing irregular spatial topological features and high-dimensional temperature distribution features, and in this way, the integrity and the topology of the thermal infrared distribution of the electronic equipment to be detected are fully utilized to improve the accuracy and the richness of the temperature distribution feature expression of the electronic equipment to be detected.

And then, the global feature matrix of the topological thermal infrared image block passes through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not. That is, the classifier is used to determine a class probability tag to which the topological thermal infrared image block global feature matrix belongs, wherein the class probability tag comprises a normal working state (first tag) of the power equipment to be detected and an abnormal working state (second tag) of the power equipment to be detected.

Particularly, in the technical scheme of the application, as for the topological thermal infrared image block global feature matrix obtained by using the thermal infrared image block global feature matrix and the spatial topological feature matrix through a graph neural network model, the convergence speed of the weight matrix of the classifier is slow in the training process, and the classifier is influenced so as to achieve the overall training speed of the model because each topological thermal infrared image block global feature vector, such as a row vector, expresses the topological association expression of the image feature semantics of a single thermal infrared image block under the spatial topology of each thermal infrared image block, and the association among the topological thermal infrared image block global feature vectors is weaker.

Therefore, in the technical solution of the present application, the applicant of the present application performs spatial regularization constraint of the weight matrix at each iteration of the weight matrix of the classifier, expressed as:

m is the weight matrix of the classifier, I.I _F Frobenius norms, M representing a matrix _b Is a bias matrix and may be initially set as an identity matrix, for example.

The spatial regularization constraint of the weight matrix is based on an endophytic correlation matrix obtained by spatial embedding the weight matrix with the transpose of the weight matrix, and L2 regularization based on endophytic correlation distribution of European space of the weight matrix of the classifier is carried out, so that the semantic dependency degree of the weight space on a specific class expression mode expressed by the feature to be classified is reflected irrespective of the numerical distribution of the feature to be weighted and the numerical value according to the position, the transmission effect of intrinsic knowledge of the feature extracted by the model is reflected by the weight space, the convergence of the weight matrix is accelerated, and the training speed of the classifier to the whole model is improved.

Fig. 1 is an application scenario diagram of a machine vision-based power device detection system according to an embodiment of the present application. As shown in fig. 1, in this application scenario, first, a thermal infrared image (e.g., D illustrated in fig. 1) of a power device to be detected (e.g., M illustrated in fig. 1) is acquired by a thermal infrared camera (e.g., C illustrated in fig. 1), and then, the thermal infrared image of the power device to be detected is input to a server (e.g., S illustrated in fig. 1) in which a machine vision-based power device detection algorithm is deployed, wherein the server is capable of processing the thermal infrared image of the power device to be detected using the machine vision-based power device detection algorithm to obtain a classification result for indicating whether an operation state of the power device to be detected is normal.

Having described the basic principles of the present application, various non-limiting embodiments of the present application will now be described in detail with reference to the accompanying drawings.

Fig. 2 is a block diagram schematic diagram of a machine vision-based power device detection system according to an embodiment of the present application. As shown in fig. 2, a machine vision-based power device detection system 100 according to an embodiment of the present application includes: the thermal infrared monitoring module 110 is configured to acquire a thermal infrared image of the electrical equipment to be detected; the image segmentation module 120 is configured to perform image blocking processing on the thermal infrared image of the to-be-detected power device to obtain a sequence of thermal infrared image blocks; the image block feature extraction module 130 is configured to pass each thermal infrared image block in the sequence of thermal infrared image blocks through a first convolutional neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors; an image block topology construction module 140, configured to construct a spatial topology matrix of the sequence of thermal infrared image blocks, where values of respective positions in the spatial topology matrix are used to represent euclidean distances between respective two thermal infrared image blocks; a spatial topology feature extraction module 150, configured to pass the spatial topology matrix through a second convolutional neural network model serving as a feature extractor to obtain a spatial topology feature matrix; an image block feature globally module 160, configured to arrange the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix; the map data encoding module 170 is configured to pass the thermal infrared image block global feature matrix and the spatial topological feature matrix through a map neural network model to obtain a topological thermal infrared image block global feature matrix; and a detection result generating module 180, configured to pass the topological thermal infrared image block global feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether the working state of the electrical equipment to be detected is normal.

More specifically, in the embodiment of the present application, the thermal infrared monitoring module 110 is configured to acquire a thermal infrared image of the electrical device to be detected. The thermal infrared image of the power equipment to be detected can be acquired through a thermal infrared camera. It should be appreciated that the infrared image of the power device can display its temperature profile and range, with different levels of hue and brightness variation representing the temperature at different locations of the device. It should be understood that in the technical solution of the present application, the temperature distribution and the range of the electronic device can represent whether the working state of the electronic device is normal.

More specifically, in the embodiment of the present application, the image segmentation module 120 is configured to perform image segmentation processing on the thermal infrared image of the to-be-detected power device to obtain a sequence of thermal infrared image blocks. In the technical scheme of the application, in the process of carrying out image blocking processing, the region with similar temperature distribution is only possibly segmented into one image block, that is, in the technical scheme of the application, the image block can be blocked based on the pixel value of each pixel position of the thermal infrared image to obtain the sequence of the thermal infrared image blocks, so that each thermal infrared image block in the sequence of the thermal infrared image blocks can more completely reflect the temperature distribution and the range of a certain part of the power equipment to be detected.

More specifically, in the embodiment of the present application, the image block feature extraction module 130 is configured to pass each thermal infrared image block in the sequence of thermal infrared image blocks through a first convolutional neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors. That is, a convolutional neural network model with excellent performance in the field of image feature extraction is used as a feature filter to capture a high-dimensional implicit feature representation of the temperature distribution and range features of the individual thermal infrared image blocks.

The convolutional neural network (Convolutional Neural Network, CNN) is an artificial neural network and has wide application in the fields of image recognition and the like. The convolutional neural network may include an input layer, a hidden layer, and an output layer, where the hidden layer may include a convolutional layer, a pooling layer, an activation layer, a full connection layer, etc., where the previous layer performs a corresponding operation according to input data, outputs an operation result to the next layer, and obtains a final result after the input initial data is subjected to a multi-layer operation. The convolutional neural network model has excellent performance in the aspect of image local feature extraction by taking a convolutional kernel as a feature filtering factor, and has stronger feature extraction generalization capability and fitting capability compared with the traditional image feature extraction algorithm based on statistics or feature engineering.

Accordingly, in one specific example, the image block feature extraction module 130 is configured to: and respectively carrying out convolution processing, mean pooling processing based on a feature matrix and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model as a filter to output the plurality of thermal infrared image block feature vectors from the last layer of the first convolution neural network model as the filter, wherein the input of the first layer of the first convolution neural network model as the filter is each thermal infrared image block in the sequence of the thermal infrared image blocks.

More specifically, in the embodiment of the present application, the image block topology construction module 140 is configured to construct a spatial topology matrix of the sequence of thermal infrared image blocks, where a value of each position in the spatial topology matrix is used to represent a euclidean distance between two corresponding thermal infrared image blocks. It is contemplated that the temperature distribution and range characteristics of the individual thermal infrared image blocks in the sequence of thermal infrared image blocks are not completely independent, but rather that there is a spatial correlation. That is, the temperature distribution and the range characteristics of each part of the electronic device to be detected are not completely independent, for example, the thermal influence between the parts with a smaller distance is more obvious. Based on the above, in the technical scheme of the application, a space topology matrix of the sequence of the thermal infrared image blocks is constructed, wherein the values of each position in the space topology matrix are used for representing the Euclidean distance between the corresponding two thermal infrared image blocks. Here, the calculation process of the euclidean distance between the two thermal infrared image blocks is: and extracting the center point coordinates of the first thermal infrared image block and the center point coordinates of the second thermal infrared image block, and calculating the Euclidean distance between the center point coordinates of the first thermal infrared image block and the center point coordinates of the second thermal infrared image block as the Euclidean distance between the two thermal infrared image blocks. Correspondingly, after the Euclidean distance between every two thermal infrared image blocks is obtained, the obtained Euclidean distance values are arranged into the space topology matrix.

Accordingly, in one specific example, as shown in fig. 3, the image block topology construction module 140 includes: a space measurement unit 141, configured to calculate euclidean distances between every two thermal infrared image blocks in the sequence of thermal infrared image blocks to obtain a plurality of euclidean distances; and a matrix constructing unit 142 configured to construct the plurality of euclidean distances into the spatial topology matrix.

More specifically, in an embodiment of the present application, the spatial topology feature extraction module 150 is configured to pass the spatial topology matrix through a second convolutional neural network model that is a feature extractor to obtain a spatial topology feature matrix. That is, in the technical scheme of the present application, a convolutional neural network model having excellent performance in the field of local feature extraction is used as a feature extractor to capture spatial topological features of each thermal infrared image block.

Accordingly, in one specific example, the spatial topology feature extraction module 150 is configured to: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a feature extractor to output the spatial topology feature matrix by the last layer of the second convolution neural network model serving as the feature extractor, wherein the input of the first layer of the second convolution neural network model serving as the feature extractor is the spatial topology matrix.

It should be noted that in the solution of the present application, if the thermal infrared image block feature vector of each thermal infrared image block is regarded as a node, the spatial topology feature matrix represents a high-dimensional feature representation of the edge between each node. Therefore, in the technical scheme of the application, the plurality of thermal infrared image block feature vectors and the spatial topological feature matrix form a high-dimensional embedded representation of the graph data. Furthermore, in the technical scheme of the application, the plurality of thermal infrared image block feature vectors are arranged into a thermal infrared image block global feature matrix, and then the thermal infrared image block global feature matrix and the spatial topological feature matrix are passed through a graph neural network model to obtain a topological thermal infrared image block global feature matrix.

More specifically, in an embodiment of the present application, the image block feature globalization module 160 is configured to arrange the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix.

More specifically, in the embodiment of the present application, the map data encoding module 170 is configured to pass the global feature matrix of the thermal infrared image block and the spatial topological feature matrix through a map neural network model to obtain the global feature matrix of the topological thermal infrared image block. Here, the map neural network model processes the thermal infrared image block global feature matrix and the spatial topological feature matrix through the learnable neural network parameters to obtain the topological thermal infrared image block global feature matrix containing irregular spatial topological features and high-dimensional temperature distribution features, and in this way, the integrity and the topology of the thermal infrared distribution of the electronic equipment to be detected are fully utilized to improve the accuracy and the richness of the temperature distribution feature expression of the electronic equipment to be detected.

Accordingly, in a specific example, the graph data encoding module is further configured to encode the thermal infrared image block global feature matrix and the spatial topological feature matrix with learnable neural network parameters using the graph neural network model to obtain the topological thermal infrared image block global feature matrix.

More specifically, in the embodiment of the present application, the detection result generating module 180 is configured to pass the topological thermal infrared image block global feature matrix through a classifier to obtain a classification result, where the classification result is used to indicate whether the working state of the electrical equipment to be detected is normal. That is, the classifier is used to determine a class probability tag to which the topological thermal infrared image block global feature matrix belongs, wherein the class probability tag comprises a normal working state (first tag) of the power equipment to be detected and an abnormal working state (second tag) of the power equipment to be detected.

It should be appreciated that the role of the classifier is to learn the classification rules and classifier using a given class, known training data, and then classify (or predict) the unknown data. Logistic regression (logistics), SVM, etc. are commonly used to solve the classification problem, and for multi-classification problems (multi-class classification), logistic regression or SVM can be used as well, but multiple bi-classifications are required to compose multiple classifications, but this is error-prone and inefficient, and the commonly used multi-classification method is the Softmax classification function.

Accordingly, in one specific example, the machine vision-based power device detection system further includes a training module for training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier.

Accordingly, in one specific example, as shown in fig. 4, the training module 200 includes: the training data obtaining unit 210 is configured to obtain a training thermal infrared image of the to-be-detected power device and a real tag value of the training thermal infrared image, where the real tag value is used to indicate whether the working state of the to-be-detected power device is normal; a training image segmentation unit 220, configured to perform image segmentation processing on the training thermal infrared image to obtain a sequence of training thermal infrared image blocks; a training image block feature extraction unit 230, configured to pass each training thermal infrared image block in the sequence of training thermal infrared image blocks through the first convolutional neural network model serving as a filter to obtain a plurality of training thermal infrared image block feature vectors; a training image block topology construction unit 240, configured to construct a spatial topology matrix of the sequence of training thermal infrared image blocks; a training spatial topological feature extraction unit 250, configured to pass the spatial topological matrix of the sequence of training thermal infrared image blocks through the second convolutional neural network model serving as a feature extractor to obtain a training spatial topological feature matrix; a training image block feature globally obtaining unit 260, configured to arrange the feature vectors of the plurality of training thermal infrared image blocks into a training thermal infrared image block global feature matrix; the training map data encoding unit 270 is configured to pass the training thermal infrared image block global feature matrix and the training spatial topological feature matrix through the map neural network model to obtain a training topological thermal infrared image block global feature matrix; a classification loss unit 280, configured to pass the training topological thermal infrared image block global feature matrix through the classifier to obtain a classification loss function value; and a training unit 290 for training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier with the classification loss function value as a loss function value.

Accordingly, in a specific example, the classification loss unit 280 is configured to: processing the training topological thermal infrared image block global feature matrix by using the classifier according to the following classification loss formula to obtain the classification loss function value; wherein, the classification loss formula is:

softmax{(M _c ,B _c )|Project(F)}

Particularly, in the technical scheme of the application, as for the topological thermal infrared image block global feature matrix obtained by using the thermal infrared image block global feature matrix and the spatial topological feature matrix through a graph neural network model, the convergence speed of the weight matrix of the classifier is slow in the training process, and the classifier is influenced so as to achieve the overall training speed of the model because each topological thermal infrared image block global feature vector, such as a row vector, expresses the topological association expression of the image feature semantics of a single thermal infrared image block under the spatial topology of each thermal infrared image block, and the association among the topological thermal infrared image block global feature vectors is weaker. Therefore, in the technical scheme of the application, the applicant of the application performs spatial regularization constraint of the weight matrix at each iteration of the weight matrix of the classifier.

Accordingly, in a specific example, in the training of the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model and the classifier with the classification loss function value as a loss function value, the spatial regularization constraint of the weight matrix of the classifier is performed with the following iterative formula at each iteration of the weight matrix of the classifier; wherein, the iterative formula is:

wherein M is the weight matrix of the classifier, M' is the weight matrix of the classifier after iterative training, and the weight matrix is #·) ^T The transposed matrix of the matrix is represented, I.I _F Frobenius norms, M representing a matrix _b Is a bias matrix, exp (·) represents the exponential operation of the matrix, which represents the calculation of the natural exponential function value raised to a power by the eigenvalues of each position in the matrix,and->Representing addition by location and multiplication by location, respectively.

In summary, the machine vision-based power equipment detection system 100 according to the embodiment of the present application is illustrated, first, each thermal infrared image block in a sequence of thermal infrared image blocks obtained by performing image blocking processing on a thermal infrared image of a power equipment to be detected is passed through a first convolutional neural network model to obtain a plurality of thermal infrared image block feature vectors, then, a spatial topology matrix of the sequence of thermal infrared image blocks is constructed and passed through a second convolutional neural network model to obtain a spatial topology feature matrix, then, the plurality of thermal infrared image block feature vectors are arranged into a thermal infrared image block global feature matrix, and then, the spatial topology feature matrix is passed through a graph neural network model to obtain a topological thermal infrared image block global feature matrix, and finally, the topological thermal infrared image block global feature matrix is passed through a classifier to obtain a classification result for indicating whether the working state of the power equipment to be detected is normal. Thus, the efficiency of power equipment state detection can be improved.

As described above, the machine vision-based power device detection system 100 according to the embodiment of the present application may be implemented in various terminal devices, for example, a server or the like having the machine vision-based power device detection algorithm according to the embodiment of the present application. In one example, the machine vision-based power device detection system 100, in accordance with embodiments of the present application, may be integrated into a terminal device as a software module and/or hardware module. For example, the machine vision-based power device detection system 100 according to the embodiment of the present application may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, the machine vision-based power device detection system 100 according to an embodiment of the present application may also be one of numerous hardware modules of the terminal device.

Alternatively, in another example, the machine vision-based power device detection system 100 and the terminal device according to the embodiment of the present application may be separate devices, and the machine vision-based power device detection system 100 according to the embodiment of the present application may be connected to the terminal device through a wired and/or wireless network and transmit the interaction information according to an agreed data format.

Fig. 5 is a flowchart of a machine vision-based power device detection method according to an embodiment of the present application. As shown in fig. 5, a machine vision-based power equipment detection method according to an embodiment of the present application includes: s110, acquiring a thermal infrared image of the power equipment to be detected; s120, performing image blocking processing on the thermal infrared image of the power equipment to be detected to obtain a sequence of thermal infrared image blocks; s130, respectively passing each thermal infrared image block in the sequence of the thermal infrared image blocks through a first convolution neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors;

s140, constructing a space topology matrix of the sequence of the thermal infrared image blocks, wherein values of all positions in the space topology matrix are used for representing Euclidean distances between the two corresponding thermal infrared image blocks; s150, the spatial topological matrix is passed through a second convolution neural network model serving as a feature extractor to obtain a spatial topological feature matrix; s160, arranging the plurality of thermal infrared image block feature vectors into a thermal infrared image block global feature matrix; s170, passing the thermal infrared image block global feature matrix and the space topology feature matrix through a graph neural network model to obtain a topology thermal infrared image block global feature matrix; and S180, enabling the topological thermal infrared image block global feature matrix to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not.

Fig. 6 is a schematic diagram of a system architecture of a machine vision-based power device detection method according to an embodiment of the present application. As shown in fig. 6, in the system architecture of the machine vision-based power equipment detection method, first, a thermal infrared image of a power equipment to be detected is acquired; then, performing image blocking processing on the thermal infrared image of the power equipment to be detected to obtain a sequence of thermal infrared image blocks; then, each thermal infrared image block in the sequence of the thermal infrared image blocks is respectively passed through a first convolution neural network model serving as a filter to obtain a plurality of thermal infrared image block feature vectors; then, constructing a space topology matrix of the sequence of the thermal infrared image blocks, wherein values of all positions in the space topology matrix are used for representing Euclidean distances between the two corresponding thermal infrared image blocks; then, the space topology matrix passes through a second convolution neural network model serving as a feature extractor to obtain a space topology feature matrix; then, the plurality of thermal infrared image block feature vectors are arranged into a thermal infrared image block global feature matrix; then, the global feature matrix of the thermal infrared image block and the spatial topological feature matrix are processed through a graph neural network model to obtain the global feature matrix of the topological thermal infrared image block; and finally, the global feature matrix of the topological thermal infrared image block passes through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not.

In a specific example, in the above machine vision based power equipment detection method, passing each thermal infrared image block in the sequence of thermal infrared image blocks through a first convolutional neural network model as a filter to obtain a plurality of thermal infrared image block feature vectors, includes: and respectively carrying out convolution processing, mean pooling processing based on a feature matrix and nonlinear activation processing on input data in forward transfer of layers by using each layer of the first convolution neural network model as a filter to output the plurality of thermal infrared image block feature vectors from the last layer of the first convolution neural network model as the filter, wherein the input of the first layer of the first convolution neural network model as the filter is each thermal infrared image block in the sequence of the thermal infrared image blocks.

In a specific example, in the above machine vision-based power equipment detection method, a spatial topology matrix of the sequence of thermal infrared image blocks is constructed, where values of respective positions in the spatial topology matrix are used to represent euclidean distances between respective two thermal infrared image blocks, and the method includes: calculating Euclidean distances between every two thermal infrared image blocks in the sequence of the thermal infrared image blocks to obtain a plurality of Euclidean distances; and constructing the plurality of euclidean distances into the spatial topology matrix.

In a specific example, in the above machine vision-based power equipment detection method, passing the spatial topology matrix through a second convolutional neural network model as a feature extractor to obtain a spatial topology feature matrix includes: and respectively carrying out two-dimensional convolution processing, feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a feature extractor to output the spatial topology feature matrix by the last layer of the second convolution neural network model serving as the feature extractor, wherein the input of the first layer of the second convolution neural network model serving as the feature extractor is the spatial topology matrix.

In a specific example, in the above machine vision-based power equipment detection method, the method further includes using a graph neural network model to encode the thermal infrared image block global feature matrix and the spatial topology feature matrix with learnable neural network parameters to obtain the topology thermal infrared image block global feature matrix.

In a specific example, in the above machine vision-based power equipment detection method, the method further includes a training step for training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier.

In a specific example, in the above machine vision-based power equipment detection method, the training step includes: acquiring a training thermal infrared image of the power equipment to be detected and a real tag value of the training thermal infrared image, wherein the real tag value is used for indicating whether the working state of the power equipment to be detected is normal or not; performing image blocking processing on the training thermal infrared image to obtain a sequence of training thermal infrared image blocks; respectively passing each training thermal infrared image block in the sequence of training thermal infrared image blocks through the first convolution neural network model serving as a filter to obtain a plurality of training thermal infrared image block feature vectors; constructing a space topology matrix of the sequence of training thermal infrared image blocks; passing the spatial topology matrix of the sequence of training thermal infrared image blocks through the second convolutional neural network model serving as a feature extractor to obtain a training spatial topology feature matrix; arranging the feature vectors of the plurality of training thermal infrared image blocks into a training thermal infrared image block global feature matrix; the training thermal infrared image block global feature matrix and the training space topological feature matrix pass through the graph neural network model to obtain a training topological thermal infrared image block global feature matrix; the global feature matrix of the training topological thermal infrared image block passes through the classifier to obtain a classification loss function value; and training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier with the classification loss function value as a loss function value.

In a specific example, in the above machine vision-based power equipment detection method, passing the training topological thermal infrared image block global feature matrix through the classifier to obtain a classification loss function value includes: processing the training topological thermal infrared image block global feature matrix by using the classifier according to the following classification loss formula to obtain the classification loss function value; wherein, the classification loss formula is:

softmax{(M _c ,B _c )|Project(F)}

In a specific example, in the above machine vision-based power equipment detection method, in training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier with the classification loss function value as a loss function value, the weight matrix of the classifier is subjected to spatial regularization constraint of the weight matrix with the following iterative formula at each iteration of the weight matrix of the classifier; wherein, the iterative formula is:

Wherein M is the weight matrix of the classifier, M' represents the weight matrix of the classifier after iterative training, (-) ^T The transposed matrix of the matrix is represented, I.I _F Frobenius norms, M representing a matrix _b Is a bias matrix, exp (·) represents the momentAn exponential operation of the matrix, the exponential operation of the matrix representing a calculation of a natural exponential function value raised to a power by eigenvalues at various positions in the matrix,and->Representing addition by location and multiplication by location, respectively.

Here, it will be understood by those skilled in the art that the specific operations of the respective steps in the above-described machine vision-based power equipment detection method have been described in detail in the above description of the machine vision-based power equipment detection system with reference to fig. 1 to 4, and thus, repetitive descriptions thereof will be omitted.

According to another aspect of the present application there is also provided a non-volatile computer readable storage medium having stored thereon computer readable instructions which when executed by a computer can perform a method as described above.

Program portions of the technology may be considered to be "products" or "articles of manufacture" in the form of executable code and/or associated data, embodied or carried out by a computer readable medium. A tangible, persistent storage medium may include any memory or storage used by a computer, processor, or similar device or related module. Such as various semiconductor memories, tape drives, disk drives, or the like, capable of providing storage functionality for software.

All or a portion of the software may sometimes communicate over a network, such as the internet or other communication network. Such communication may load software from one computer device or processor to another. For example: a hardware platform loaded from a server or host computer of the video object detection device to a computer environment, or other computer environment implementing the system, or similar functioning system related to providing information needed for object detection. Thus, another medium capable of carrying software elements may also be used as a physical connection between local devices, such as optical, electrical, electromagnetic, etc., propagating through cable, optical cable, air, etc. Physical media used for carrier waves, such as electrical, wireless, or optical, may also be considered to be software-bearing media. Unless limited to a tangible "storage" medium, other terms used herein to refer to a computer or machine "readable medium" mean any medium that participates in the execution of any instructions by a processor.

The application uses specific words to describe embodiments of the application. Reference to "a first/second embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the application. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various positions in this specification are not necessarily referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the application may be combined as suitable.

Furthermore, those skilled in the art will appreciate that the various aspects of the application are illustrated and described in the context of a number of patentable categories or circumstances, including any novel and useful procedures, machines, products, or materials, or any novel and useful modifications thereof. Accordingly, aspects of the application may be performed entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.) or by a combination of hardware and software. The above hardware or software may be referred to as a "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the application may take the form of a computer product, comprising computer-readable program code, embodied in one or more computer-readable media.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The foregoing is illustrative of the present invention and is not to be construed as limiting thereof. Although a few exemplary embodiments of this invention have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention as defined in the following claims. It is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the claims and their equivalents.

Claims

1. A machine vision-based power equipment detection system, comprising:

the detection result generation module is used for enabling the topological thermal infrared image block global feature matrix to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not;

the power equipment detection system further comprises a training module for training the first convolutional neural network model serving as a filter, the second convolutional neural network model serving as a feature extractor, the graph neural network model and the classifier;

The training module comprises:

a training unit for training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model, and the classifier with the classification loss function value as a loss function value;

in the process of training the first convolutional neural network model serving as a filter, the second convolutional neural network model serving as a feature extractor, the graph neural network model and the classifier by taking the classification loss function value as the loss function value, performing spatial regularization constraint of a weight matrix of the classifier according to the following iterative formula when the weight matrix of the classifier is iterated each time;

wherein, the iterative formula is:

Wherein M is the weight matrix of the classifier, M' represents the weight matrix of the classifier after iterative training, (-) ^T Represents the transposed matrix of the matrix, i.i.i.f represents the Frobenius norm of the matrix, M _b Is a bias matrix, exp (·) represents the exponential operation of the matrix, which represents the calculation of the natural exponential function value raised to a power by the eigenvalues of each position in the matrix,and->Representing addition by location and multiplication by location, respectively.

2. The machine vision-based power device detection system of claim 1, wherein the image block feature extraction module is configured to:

3. The machine vision-based power device detection system of claim 2, wherein the tile topology module comprises:

4. The machine vision-based power device detection system of claim 3, wherein the spatial topology feature extraction module is configured to:

5. The machine vision-based power device detection system of claim 4, wherein the graph data encoding module is further configured to encode the thermal infrared image block global feature matrix and the spatial topological feature matrix with learnable neural network parameters using the graph neural network model to obtain the topological thermal infrared image block global feature matrix.

6. The machine vision-based power equipment detection system of claim 5, wherein the classification loss unit is configured to:

wherein, the classification loss formula is:

softmax{(M _c ,B _c )|Project(F)}

7. A machine vision-based power equipment detection method, comprising:

acquiring a thermal infrared image of the power equipment to be detected;

the global feature matrix of the topological thermal infrared image block is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the working state of the power equipment to be detected is normal or not;

the power equipment detection method further comprises the following steps: training the first convolutional neural network model serving as a filter, the second convolutional neural network model serving as a feature extractor, the graph neural network model and the classifier;

the training step comprises the following steps:

acquiring a training thermal infrared image of the power equipment to be detected and a real tag value of the training thermal infrared image, wherein the real tag value is used for indicating whether the working state of the power equipment to be detected is normal or not;

Performing image blocking processing on the training thermal infrared image to obtain a sequence of training thermal infrared image blocks;

respectively passing each training thermal infrared image block in the sequence of training thermal infrared image blocks through the first convolution neural network model serving as a filter to obtain a plurality of training thermal infrared image block feature vectors;

constructing a space topology matrix of the sequence of training thermal infrared image blocks;

passing the spatial topology matrix of the sequence of training thermal infrared image blocks through the second convolutional neural network model serving as a feature extractor to obtain a training spatial topology feature matrix;

arranging the feature vectors of the plurality of training thermal infrared image blocks into a training thermal infrared image block global feature matrix;

the training thermal infrared image block global feature matrix and the training space topological feature matrix pass through the graph neural network model to obtain a training topological thermal infrared image block global feature matrix;

the global feature matrix of the training topological thermal infrared image block passes through the classifier to obtain a classification loss function value; and

training the first convolutional neural network model as a filter, the second convolutional neural network model as a feature extractor, the graph neural network model and the classifier with the classification loss function value as a loss function value;

wherein, the iterative formula is:

wherein M is the weight matrix of the classifier, M' represents the weight matrix of the classifier after iterative training, (-) ^T The transposed matrix of the matrix is represented, I.I _F Frobenius norms, M representing a matrix _b Is a bias matrix, exp (·) represents the exponential operation of the matrix, which represents the calculation of the natural exponential function value raised to a power by the eigenvalues of each position in the matrix,and->Representing addition by location and multiplication by location, respectively.