CN111523589B - Bolt defect classification method based on bolt pair knowledge graph - Google Patents

Bolt defect classification method based on bolt pair knowledge graph Download PDF

Info

Publication number
CN111523589B
CN111523589B CN202010316196.0A CN202010316196A CN111523589B CN 111523589 B CN111523589 B CN 111523589B CN 202010316196 A CN202010316196 A CN 202010316196A CN 111523589 B CN111523589 B CN 111523589B
Authority
CN
China
Prior art keywords
node
defect
bolt
pair
semantic object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010316196.0A
Other languages
Chinese (zh)
Other versions
CN111523589A (en
Inventor
孔英会
段记坤
赵振兵
翟永杰
赵文清
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
North China Electric Power University
Original Assignee
North China Electric Power University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by North China Electric Power University filed Critical North China Electric Power University
Priority to CN202010316196.0A priority Critical patent/CN111523589B/en
Publication of CN111523589A publication Critical patent/CN111523589A/en
Application granted granted Critical
Publication of CN111523589B publication Critical patent/CN111523589B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Abstract

The invention discloses a bolt defect classification method based on a bolt-parent pair knowledge graph, which comprises the following steps: constructing a thrombus parent pair defect data set; extracting the characteristics of the combined region of the bolt pair and the mother pair; extracting regional characteristics of the semantic objects; constructing a bolt pair knowledge graph based on a GGNN model; calculating node-level features by adopting an output network realized by a complete connection layer, calculating the features of defective nodes and semantic object nodes, and connecting the features of the self nodes and the features of the related nodes as final feature vectors of the defect types; and feeding the final feature vector to a complete connection layer, calculating score vectors of various defects, and taking the defect type with the largest score as the defect type of the bolt defect. According to the bolt defect classification method based on the bolt pair knowledge spectrum, the relation between bolts and nuts is fully considered, the bolt pair characteristics are extracted by using a convolutional neural network, the bolt pair knowledge spectrum is built by combining prior knowledge, and the bolt defect classification is efficiently completed.

Description

Bolt defect classification method based on bolt pair knowledge graph
Technical Field
The invention relates to the technical field of bolt defect analysis, in particular to a bolt defect classification method based on a bolt-parent pair knowledge graph.
Background
Along with the continuous growth of the power grid scale, helicopters and unmanned aerial vehicles are widely applied to line inspection, the number of generated aerial images is greatly increased, bolts are not only numerous and small in size, but also widely stored in various parts of a power transmission line, and because accident frequency caused by the defects of the bolts of the power transmission line occurs, the analysis of the defects of the bolts is very important. The existing research only can extract the surface features of the bolts and neglect the association among targets, so that the limitation of classifying the defects of the bolts of the transmission line by directly applying the convolutional neural network is more and more obvious, and the problems are mainly as follows:
(1) The number of bolts on the transmission line is large, and the size is small, so that the target characteristics are not easy to extract.
(2) The distribution of the transmission lines is wide, and the influence of complex environments around the bolt images is particularly serious in interference of classification results.
At present, most bolt defect classification researches are limited to the improvement of the existing algorithm, such as convolutional kernel adjustment, network depth increase and the like, and the effect is quite unsatisfactory. In addition, the existing power transmission line bolt defect classification method has the problems that the existing power transmission line bolt defect classification method is limited to surface feature extraction, and the relation among targets is ignored and is greatly influenced by complex environments.
Disclosure of Invention
The invention aims to provide a bolt defect classification method based on a bolt pair knowledge graph, which fully considers the relation between bolts and nuts, and proposes to use a convolutional neural network to extract bolt pair characteristics and combine priori knowledge to construct the bolt pair knowledge graph, so that the bolt defect classification is efficiently completed.
In order to achieve the above object, the present invention provides the following solutions:
a bolt defect classification method based on a bolt pair knowledge graph comprises the following steps:
s1, constructing a thrombus parent pair defect data set;
s2, extracting characteristics of a bolt pair binding region;
s3, extracting the regional characteristics of the semantic objects;
s4, constructing a bolt pair knowledge graph based on a GGNN model, which specifically comprises the following steps: initializing the combined region features and the semantic object region features based on the thrombus parent pairs to obtain a defect node and a semantic object node of the thrombus parent pair knowledge graph, and obtaining an edge of the thrombus parent pair knowledge graph based on an adjacency matrix, wherein the adjacency matrix represents the association between the defect node and the semantic object node;
s5, calculating node level characteristics by adopting an output network realized by a complete connection layer, calculating characteristics of defective nodes and semantic object nodes, and connecting the characteristics of the self nodes and the characteristics of the related nodes as final characteristic vectors f for defect types i ；
S6, final feature vector f i Fed to the fully connected layer, a score vector s= { s for each type of defect is calculated according to the following formula 1 ,s 2 ,...,s M }，
s i ＝Wf i +b
The defect type with the largest score is the defect type of the bolt defect.
Optionally, in the step S1, constructing a thrombus parent pair defect data set specifically includes: constructing a coarse-level defect data set, wherein the coarse-level defect data set comprises a missing gasket, a missing pin, a gasket and complete four types of data; or a fine defect data set is constructed, wherein the fine defect data set comprises a missing gasket, a visually invisible missing pin, a missing gasket, a visually invisible missing pin and complete data.
Optionally, in the step S2, the extracting the features of the bonding region of the pin-nut pair specifically includes: clipping the bolt pair defect dataset into three regions, wherein one region is a bolt pair region, the other two regions respectively comprise bolt and nut independent regions, and extracting features from the three regions by using a Faster R-CNN detector; these features are then connected to position information encoding the geometric features of the bolts and nutsAnd fed to the full join layer to produce the final feature d-dimensional feature vector f h ＝R d This feature is used as an input feature for the defective node to initialize the defective feature.
Optionally, in the step S3, extracting the features of the semantic object area specifically includes: and (3) taking the detected bolt pair region as a semantic object detection region, comparing all semantic object coordinates in the picture obtained after pre-training with the bolt pair joint coordinates, if the semantic object coordinates are in the bolt pair joint coordinates, meeting the semantic object detection region, using the semantic object o features, otherwise discarding, continuing to circulate the steps until the comparison is completed, and extracting the features of the semantic object region by using a FasterR-CNN detector.
Optionally, in the step S4, the defect node and the semantic object node of the thrombus pair knowledge map are obtained based on the thrombus pair combination region feature and the semantic object region feature initialization, which specifically includes:
using a single hot vector to explicitly distinguish between two node types, namely defective nodes and semantic object nodes, use [1,0 ]]And [0,1 ]]Respectively representing a defect node and a semantic object node, and initializing a hidden state h under other conditions v 0 By [ [0,1 ]],0 d ]Indicating that the hidden state is initialized at time step t=0, indicated as
Figure BDA0002459529540000031
Wherein the d-dimensional feature vector f h ＝R d The feature is used as an input feature of a defect node to initialize the defect feature, and a d-dimensional feature vector f o ＝R d The feature is used as an input feature for the semantic object node to initialize the defect feature.
Optionally, in the step S5, the node-level feature is calculated by using the output network implemented by the fully connected layer, and the feature of the defective node and the semantic object node is calculated, and for the defect class, the feature of the connected node and the feature of the related node are used as the final feature vector f i Specifically comprises：
S501, at each time step, each node first aggregates messages from its neighboring nodes, expressed as:
Figure BDA0002459529540000032
wherein A is v Is a sub-matrix of the adjacency matrix a, representing the connection of node v with its neighboring nodes, and then, the model updates the hidden state of each node by gating mechanism of the gating loop unit in combination with information from other nodes and previous time step, through T loops of formula (3),
Figure BDA0002459529540000033
where σ and tanh are logical sigmoid and hyperbolic tangent functions, respectively, z and r are update and reset gates,
Figure BDA0002459529540000034
control forgetting information, < >>
Figure BDA0002459529540000035
Control the newly generated information, W represents the weight of the input, U represents the weight of the sample input at the moment,/I->
Figure BDA0002459529540000036
Conceal state for other nodes,/->
Figure BDA0002459529540000037
Conceal state for node of previous time step, +.>
Figure BDA0002459529540000038
Hiding the state for the updated node;
s502, calculating node level characteristics by adopting an output network realized by a complete connection layer, wherein the node level characteristics are calculated by
Figure BDA0002459529540000041
Representing the output characteristics of the defective node and the semantic object node as { o } r1 ,o r2 ,...,o rM Sum { o } o1 ,o o2 ,..,o oN }；
For defect class r i The characteristics of the node connecting itself and the characteristics of the node concerned are taken as the final characteristics thereof, i.e
f i ＝[o ri ,α i1 o o1 ,α i2 o o2 ,...,α iN o oN ] (5)
Wherein N is i Is the neighbor node set of node i in the graph if the semantic object node j does not belong to N i Then alpha is ij Designated as 0, if the semantic object node j belongs to N i Then alpha is ij Designated as 1.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects: according to the bolt defect classification method based on the bolt pair knowledge graph, bolts are not directly researched, but bolt pairs formed by bolts and nuts which are connected together are researched, firstly, the whole bolt pin gasket of the whole nut is researched as a whole, and the single defect on the bolt is not simply researched; secondly, the defect nodes and the semantic object nodes of the knowledge graph are easy to construct by combining the bolt pairs, and the bolt and nut areas can be distinguished independently when the bolt pairs are combined, so that the defect nodes are convenient to initialize; thirdly, the detection range is increased, the detection omission is reduced fundamentally, and if a single research bolt or nut is used, the gasket or pin cannot be detected, so that the detection omission is caused fundamentally. In addition, since there may be multiple bolt pairs in a picture, it may cause that detecting a certain bolt pair may erroneously detect a gasket or pin of other bolt pairs, which may fundamentally cause erroneous detection. The invention utilizes the prior knowledge of the graphic structure representation thrombus master pair to construct the thrombus master pair knowledge graph, and can most effectively and intuitively express the association between the thrombus master pair defects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart diagram of a bolt defect classification method based on a bolt-to-knowledge graph according to an embodiment of the invention;
FIG. 2 is a schematic diagram of accuracy and recall of coarse-level defect image classification experiments according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of precision and recall of fine defect image classification experiments according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a bolt defect classification method based on a bolt pair knowledge graph, which fully considers the relation between bolts and nuts, and proposes to use a convolutional neural network to extract bolt pair characteristics and combine priori knowledge to construct the bolt pair knowledge graph, so that the bolt defect classification is efficiently completed.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
As shown in FIG. 1, the bolt defect classification method based on the bolt pair knowledge graph provided by the invention comprises the following steps:
s1, constructing a thrombus parent pair defect data set;
s2, extracting characteristics of a bolt pair binding region;
s3, extracting the regional characteristics of the semantic objects;
s4, constructing a bolt pair knowledge graph based on a GGNN model, which specifically comprises the following steps: initializing the combined region features and the semantic object region features based on the thrombus parent pairs to obtain a defect node and a semantic object node of the thrombus parent pair knowledge graph, and obtaining an edge of the thrombus parent pair knowledge graph based on an adjacency matrix, wherein the adjacency matrix represents the association between the defect node and the semantic object node;
s5, calculating node level characteristics by adopting an output network realized by a complete connection layer, calculating characteristics of defective nodes and semantic object nodes, and connecting the characteristics of the self nodes and the characteristics of the related nodes as final characteristic vectors f for defect types i ；
S6, final feature vector f i Fed to the fully connected layer, a score vector s= { s for each type of defect is calculated according to the following formula 1 ,s 2 ,...,s M }，
s i ＝Wf i +b
The defect type with the largest score is the defect type of the bolt defect.
In the step S1, a thrombus parent pair defect data set is constructed, which specifically includes: constructing a coarse-level defect data set, wherein the coarse-level defect data set comprises a missing gasket, a missing pin, a gasket and complete four types of data; or a fine defect data set is constructed, wherein the fine defect data set comprises a missing gasket, a visually invisible missing pin, a missing gasket, a visually invisible missing pin and complete data. The results of classifying the knowledge-graph-guided coarse-level defect image and the fine-level defect image based on the bolt mother of GGNN are respectively reflected in fig. 2 and 3.
For coarse-level defect datasets, the thrombus pair dataset is divided into a training set of images and relational instances, a test set of images and instances. For fine defect datasets, the parent-to-parent data sets of coarse defect datasets are re-manually annotated into fine defect categories. The fine defect dataset is divided into a training set of multiple images and relationship instances, a test set of multiple images and relationship instances. For fine defect datasets, the parent-to-parent data sets of coarse defect datasets are re-manually annotated into fine defect categories.
In the step S2, the extracting the characteristics of the coupling region of the pin-nut pair specifically includes: clipping the bolt pair defect dataset into three regions, wherein one region is a bolt pair region, the other two regions respectively comprise bolt and nut independent regions, and extracting features from the three regions by using a Faster R-CNN detector; these features are then connected with position information of the geometric features of the encoding bolts and nuts and fed to the fully connected layer to produce the final feature d-dimensional feature vector f h ＝R d This feature is used as an input feature for the defective node to initialize the defective feature.
Wherein all components of the ResNet-101 model and VGG-16 model employed by the Faster R-CNN detector are trained using random gradient descent, except GGNN is trained using adaptive moment estimation. Features of the region of the thrombus pair and the semantic object region are extracted by using a widely used ResNet-101 model and a VGG-16 model respectively. For the GGNN propagation model, the dimension of the hidden state is set to 4098, the dimension of the output feature is set to 512, and the iteration time T is set to 5.
In the step S3, extracting the semantic object region features specifically includes: and (3) taking the detected bolt pair region as a semantic object detection region, comparing all semantic object coordinates in the picture obtained after pre-training with the bolt pair joint coordinates, if the semantic object coordinates are in the bolt pair joint coordinates, meeting the semantic object detection region, using the semantic object o features, otherwise discarding, continuing to circulate the steps until the comparison is completed, and extracting the features of the semantic object region by using a Faster R-CNN detector.
The input features of the nodes of the semantic object o detected by the semantic object detection area are initialized by the features extracted from the corresponding semantic object area, otherwise, the nodes are initialized by d-dimensional zero vectors.
In the step S4, a defect node and a semantic object node of a thrombus pair knowledge graph are obtained based on the thrombus pair combination region feature and the semantic object region feature initialization, which specifically comprises:
using a single hot vector to explicitly distinguish between two node types, namely defective nodes and semantic object nodes, use [1,0 ]]And [0,1 ]]Respectively representing a defect node and a semantic object node, and initializing a hidden state h under other conditions v 0 is [ [0,1 ]],0 d ]Indicating that the hidden state is initialized at time step t=0, indicated as
Figure BDA0002459529540000071
Wherein the d-dimensional feature vector f h ＝R d The feature is used as an input feature of a defect node to initialize the defect feature, and a d-dimensional feature vector f o ＝R d The feature is used as an input feature for the semantic object node to initialize the defect feature.
In said step S5, node-level features are calculated using the output network implemented by the fully connected layer, the features of the defective node and the semantic object node are calculated, and for the defective class, the features of its own node and the features of the involved node are connected as their final feature vectors f i The method specifically comprises the following steps:
s501, at each time step, each node first aggregates messages from its neighboring nodes, expressed as:
Figure BDA0002459529540000072
wherein A is v Is a sub-matrix of the adjacency matrix a, representing the connection of node v with its neighboring nodes, and then, the model updates the hidden state of each node by gating mechanism of the gating loop unit in combination with information from other nodes and previous time step, through T loops of formula (3),
Figure BDA0002459529540000073
wherein sigma andtanh is a logical sigmoid and hyperbolic tangent function, respectively; and represents an element multiplication operation. z and r are the update and reset gates,
Figure BDA0002459529540000081
control forgetting information, < >>
Figure BDA0002459529540000082
Control the newly generated information, W represents the weight of the input, U represents the weight of the sample input at the moment,/I->
Figure BDA0002459529540000083
Conceal state for other nodes,/->
Figure BDA0002459529540000084
Conceal state for node of previous time step, +.>
Figure BDA0002459529540000085
Hiding the state for the updated node; the model then updates the hidden state of each node through a gating mechanism similar to the gating loop unit, combining information from other nodes and the previous time step, through T loops of equation (3). In this way, each node can aggregate information from its neighbor nodes while transmitting its own messages to other neighbor nodes, thereby enabling interactions between all nodes. After the time T interaction, the node message has propagated through the graph and the final hidden state of each node, i.e., { h }, can be obtained T 1 ,h T 2 ,....,H T |V| }。
S502, calculating node level characteristics by adopting an output network realized by a complete connection layer, wherein the node level characteristics are calculated by
Figure BDA0002459529540000086
Representing the output characteristics of the defective node and the semantic object node as { o } r1 ,o r2 ,...,o rM Sum { o } o1 ,o o2 ,..,o oN }；
For defect class r i The characteristics of the node connecting itself and the characteristics of the node concerned are taken as the final characteristics thereof, i.e
f i ＝[o ri ,α i1 o o1 ,α i2 o o2 ,...,α iN o oN ] (5)
Wherein N is i Is the neighbor node set of node i in the graph if the semantic object node j does not belong to N i Then alpha is ij Designated as 0, if the semantic object node j belongs to N i Then alpha is ij Designated as 1.
The method adopts a FasterR-CNN detector to extract characteristics and uses a gate-controlled neural network (GGNN) model to carry out message transmission among nodes, then fuses the final hidden state of the defective node and the final hidden state of the semantic object node to obtain final characteristics, calculates node-level characteristics through a complete connection layer, and finally calculates a classification prediction result through the complete connection layer. The invention provides a method for constructing a bolt pair knowledge map by combining the expression of related priori knowledge of bolts and nuts on a transmission line, which respectively adopts a ResNet-101 model and a VGG-16 model as a feature extractor of a FasterR-CNN deep learning framework and uses a gate control neural network (GGNN) model to carry out message transmission between nodes. The research on the power transmission line is limited to the change of the existing algorithm, such as convolution kernel adjustment, network depth increase, application parameter adjustment or algorithm adjustment data set adjustment. According to the invention, the ResNet-101 model and the VGG-16 model are respectively used as the feature extractor of the FasterR-CNN deep learning framework, so that the object features can be extracted, the defect nodes and the semantic object nodes of the GGNN model can be initialized, and the graphic nodes of the thrombus pairs can be effectively represented. The invention utilizes the prior knowledge of the graphic structure representation thrombus master pair to construct the thrombus master pair knowledge graph, and can most effectively and intuitively express the association between the thrombus master pair defects.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims (4)

1. A bolt defect classification method based on a bolt-parent pair knowledge graph is characterized by comprising the following steps:
s1, constructing a thrombus parent pair defect data set; the method specifically comprises the following steps: constructing a coarse-level defect data set, wherein the coarse-level defect data set comprises a missing gasket, a missing pin, a gasket and complete four types of data; or a fine defect data set is built, wherein the fine defect data set comprises six types of data, namely a missing gasket, a visually invisible missing pin, a missing gasket, a visually invisible missing pin and a complete piece of data;
s2, extracting characteristics of a bolt pair binding region; the method specifically comprises the following steps: clipping the bolt pair defect dataset into three regions, wherein one region is a bolt pair region, the other two regions respectively comprise bolt and nut independent regions, and extracting features from the three regions by using a Faster R-CNN detector; these features are then connected with position information of the geometric features of the encoding bolts and nuts and fed to the fully connected layer to produce the final feature d-dimensional feature vector f h ＝R d Initializing a defect feature as an input feature of the defect node;
s3, extracting the regional characteristics of the semantic objects;
s4, constructing a bolt pair knowledge graph based on a GGNN model, which specifically comprises the following steps: initializing the combined region features and the semantic object region features based on the thrombus parent pairs to obtain a defect node and a semantic object node of the thrombus parent pair knowledge graph, and obtaining an edge of the thrombus parent pair knowledge graph based on an adjacency matrix, wherein the adjacency matrix represents the association between the defect node and the semantic object node;
s5, calculating node level characteristics by adopting an output network realized by a complete connection layer, calculating characteristics of defective nodes and semantic object nodes, and taking the characteristics of connecting own nodes and the characteristics of related nodes as defect typesIts final eigenvector f i ；
S6, final feature vector f i Fed to the fully connected layer, a score vector s= { s for each type of defect is calculated according to the following formula 1 ,s 2 ,...,s M }，
s i ＝Wf i +b
The defect type with the largest score is the defect type of the bolt defect.
2. The bolt defect classification method based on the bolt-parent-pair knowledge graph according to claim 1, wherein in the step S3, extracting the semantic object region features specifically includes: and (3) taking the detected bolt pair region as a semantic object detection region, comparing all semantic object coordinates in the picture obtained after pre-training with the bolt pair joint coordinates, if the semantic object coordinates are in the bolt pair joint coordinates, meeting the semantic object detection region, using the semantic object o features, otherwise discarding, continuing to circulate the steps until the comparison is completed, and extracting the features of the semantic object region by using a FasterR-CNN detector.
3. The bolt defect classification method based on the bolt-parent-pair knowledge graph according to claim 1, wherein in the step S4, the defect node and the semantic object node of the bolt-parent-pair knowledge graph are obtained based on the initialization of the bolt-parent-pair combined region feature and the semantic object region feature, specifically comprising:
using a single hot vector to explicitly distinguish between two node types, namely defective nodes and semantic object nodes, use [1,0 ]]And [0,1 ]]Respectively representing a defect node and a semantic object node, and initializing a hidden state h under other conditions v 0 is [ [0,1 ]],0 d ]Indicating that the hidden state is initialized at time step t=0, indicated as
Figure FDA0004129493040000021
Wherein the d-dimensional feature vector f h ＝R d The feature is used as an input feature of a defect node to initialize the defect feature, and a d-dimensional feature vector f o ＝R d The feature is used as an input feature for the semantic object node to initialize the defect feature.
4. The bolt defect classification method based on bolt-parent-pair knowledge graph according to claim 3, wherein in the step S5, the node-level features are calculated using the output network implemented by the complete connection layer, the features of the defective node and the semantic object node are calculated, and for the defect class, the features of the connected node and the features of the related node are used as the final feature vector f thereof i The method specifically comprises the following steps:
s501, at each time step, each node first aggregates messages from its neighboring nodes, expressed as:
Figure FDA0004129493040000031
wherein A is v Is a sub-matrix of the adjacency matrix a, representing the connection of node v with its neighboring nodes, and then, the model updates the hidden state of each node by gating mechanism of the gating loop unit in combination with information from other nodes and previous time step, through T loops of formula (3),
Figure FDA0004129493040000032
where σ and tanh are logical sigmoid and hyperbolic tangent functions, respectively, z and r are update and reset gates,
Figure FDA0004129493040000033
control forgetting information, < >>
Figure FDA0004129493040000034
Control the newly generated information, W represents the weight of the input, U represents the input at the momentWeight of sample, ++>
Figure FDA0004129493040000035
Conceal state for other nodes,/->
Figure FDA0004129493040000038
Conceal state for node of previous time step, +.>
Figure FDA0004129493040000036
Hiding the state for the updated node;
s502, calculating node level characteristics by adopting an output network realized by a complete connection layer, wherein the node level characteristics are calculated by
Figure FDA0004129493040000037
Representing the output characteristics of the defective node and the semantic object node as { o } r1 ,o r2 ,...,o rM Sum { o } o1 ,o o2 ,..,o oN }；
For defect class r i The characteristics of the node connecting itself and the characteristics of the node concerned are taken as the final characteristics thereof, i.e
f i ＝[o ri ,α i1 o o1 ,α i2 o o2 ,...,α iN o oN ] (5)
Wherein N is i Is the neighbor node set of node i in the graph if the semantic object node j does not belong to N i Then alpha is ij Designated as 0, if the semantic object node j belongs to N i Then alpha is ij Designated as 1.
CN202010316196.0A 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph Active CN111523589B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010316196.0A CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010316196.0A CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Publications (2)

Publication Number Publication Date
CN111523589A CN111523589A (en) 2020-08-11
CN111523589B true CN111523589B (en) 2023-05-23

Family

ID=71910571

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010316196.0A Active CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Country Status (1)

Country Link
CN (1) CN111523589B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115391563B (en) * 2022-09-01 2024-02-06 广东工业大学 Knowledge graph link prediction method based on multi-source heterogeneous data fusion

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07100660A (en) * 1993-10-05 1995-04-18 Hirata:Kk Method and device for detecting nut defective welding
CN110084296A (en) * 2019-04-22 2019-08-02 中山大学 A kind of figure expression learning framework and its multi-tag classification method based on certain semantic
CN110263858A (en) * 2019-06-21 2019-09-20 华北电力大学（保定） A kind of bolt image composition method, device and relevant device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07100660A (en) * 1993-10-05 1995-04-18 Hirata:Kk Method and device for detecting nut defective welding
CN110084296A (en) * 2019-04-22 2019-08-02 中山大学 A kind of figure expression learning framework and its multi-tag classification method based on certain semantic
CN110263858A (en) * 2019-06-21 2019-09-20 华北电力大学（保定） A kind of bolt image composition method, device and relevant device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Zhenbing Zhao et al..Detection Method Based on Automatic Visual Shape Clustering for Pin-Missing Defect in Transmission Lines.《 IEEE Transactions on Instrumentation and Measurement 》.2020,第第69卷卷(第第09期期),全文. *
刘梓权 ; 王慧芳 ; .基于知识图谱技术的电力设备缺陷记录检索方法.电力系统自动化.2018,(第14期),全文. *

Also Published As

Publication number Publication date
CN111523589A (en) 2020-08-11

Similar Documents

Publication Publication Date Title
WO2021000903A1 (en) End-to-end sar image recognition method and apparatus, and storage medium
CN107527352B (en) Remote sensing ship target contour segmentation and detection method based on deep learning FCN network
CN108596053B (en) Vehicle detection method and system based on SSD and vehicle posture classification
WO2021232687A1 (en) Deep learning-based point cloud upsampling method
CN105930868B (en) A kind of low resolution airport target detection method based on stratification enhancing study
WO2020102988A1 (en) Feature fusion and dense connection based infrared plane target detection method
CN109886066B (en) Rapid target detection method based on multi-scale and multi-layer feature fusion
CN108257154B (en) Polarization SAR image change detection method based on regional information and CNN
CN112395987B (en) SAR image target detection method based on unsupervised domain adaptive CNN
CN105844637B (en) SAR image change detection method based on non-local CV model
CN109446894B (en) Multispectral image change detection method based on probability segmentation and Gaussian mixture clustering
CN111539422B (en) Flight target cooperative identification method based on fast RCNN
CN114937066A (en) Point cloud registration system and method based on cross offset features and space consistency
Han et al. Level set based shape prior and deep learning for image segmentation
Yang et al. Aircraft tracking based on fully conventional network and Kalman filter
CN111523589B (en) Bolt defect classification method based on bolt pair knowledge graph
Li et al. Topological data assimilation using Wasserstein distance
CN115853173A (en) Building curtain wall for construction and installation
CN109583456B (en) Infrared surface target detection method based on feature fusion and dense connection
Wang et al. Roof plane segmentation from lidar point cloud data using region expansion based l 0 gradient minimization and graph cut
CN112270285B (en) SAR image change detection method based on sparse representation and capsule network
CN111209813B (en) Remote sensing image semantic segmentation method based on transfer learning
Zhou et al. Infrared small target detection via learned infrared patch-image convolutional network
Zhang et al. An improved adaptive level set method for image segmentation
CN114814776B (en) PD radar target detection method based on graph attention network and transfer learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant