CN111523589A - Bolt defect classification method based on bolt pair knowledge graph - Google Patents

Bolt defect classification method based on bolt pair knowledge graph Download PDF

Info

Publication number
CN111523589A
CN111523589A CN202010316196.0A CN202010316196A CN111523589A CN 111523589 A CN111523589 A CN 111523589A CN 202010316196 A CN202010316196 A CN 202010316196A CN 111523589 A CN111523589 A CN 111523589A
Authority
CN
China
Prior art keywords
bolt
node
defect
pair
semantic object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010316196.0A
Other languages
Chinese (zh)
Other versions
CN111523589B (en
Inventor
孔英会
段记坤
赵振兵
翟永杰
赵文清
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
North China Electric Power University
Original Assignee
North China Electric Power University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by North China Electric Power University filed Critical North China Electric Power University
Priority to CN202010316196.0A priority Critical patent/CN111523589B/en
Publication of CN111523589A publication Critical patent/CN111523589A/en
Application granted granted Critical
Publication of CN111523589B publication Critical patent/CN111523589B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Animal Behavior & Ethology (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a bolt defect classification method based on a bolt-mother pair knowledge graph, which comprises the following steps of: constructing a bolt pair defect data set; extracting the feature of the bolt-nut pair combined area; extracting semantic object region features; constructing a key pair knowledge graph based on a GGNN model; calculating node-level characteristics by adopting an output network realized by a complete connection layer, calculating the characteristics of a defect node and a semantic object node, and connecting the characteristics of the node and the related characteristics of the defect type as a final characteristic vector of the defect type; and feeding the final feature vector to a complete connection layer, calculating score vectors of various defects, and taking the defect type with the maximum score as the defect type of the bolt defect. According to the bolt defect classification method based on the bolt pair knowledge graph, provided by the invention, the relation between a bolt and a nut is fully considered, the bolt pair characteristics are extracted by using a convolutional neural network, the bolt pair knowledge graph is constructed by combining priori knowledge, and the bolt defect classification is efficiently completed.

Description

Bolt defect classification method based on bolt pair knowledge graph
Technical Field
The invention relates to the technical field of bolt defect analysis, in particular to a bolt defect classification method based on a bolt-nut pair knowledge graph.
Background
With the continuous increase of the scale of a power grid, helicopters and unmanned aerial vehicles are widely applied to line patrol, the number of generated aerial images is increased dramatically, bolts are numerous and small in size and widely stored in various parts of a power transmission line, and the analysis of bolt defects is very important to strengthen due to frequent accidents caused by the bolt defects of the power transmission line. The existing research can only extract the surface characteristics of the bolt and neglects the correlation among targets, so the limitation of directly applying the convolutional neural network to classify the defects of the transmission line bolt is more and more obvious, and the following two problems are mainly solved:
(1) the number of bolts on the power transmission line is large, the size is small, and target features are difficult to extract.
(2) The distribution of the transmission lines is wide, and the interference of the complicated environment around the bolt image on the classification result is particularly serious.
At present, most of bolt defect classification researches are limited to the improvement of the existing algorithm, such as the adjustment of convolution kernel, the increase of network depth and the like, and the effect is not ideal. In addition, the existing power transmission line bolt defect classification method has the problems that the extraction of surface features is limited, the relation between targets is neglected, the influence of a complex environment is large, and the like.
Disclosure of Invention
The invention aims to provide a bolt defect classification method based on a bolt-pair knowledge graph, which sufficiently considers the relation between a bolt and a nut, provides a method for extracting bolt-pair characteristics by using a convolutional neural network and combines priori knowledge to construct the bolt-pair knowledge graph, and efficiently finishes bolt defect classification.
In order to achieve the purpose, the invention provides the following scheme:
a bolt defect classification method based on a bolt mother pair knowledge graph comprises the following steps:
s1, constructing a bolt pair defect data set;
s2, extracting the feature of the bolt-nut pair combined area;
s3, extracting semantic object region features;
s4, constructing a embolus pair knowledge graph based on the GGNN model, which specifically comprises the following steps: initializing to obtain a defect node and a semantic object node of a latch pair knowledge graph based on the latch pair joint region feature and the semantic object region feature, and obtaining an edge of the latch pair knowledge graph based on an adjacent matrix, wherein the adjacent matrix represents the association between the defect node and the semantic object node;
s5, calculating node-level characteristics by adopting an output network realized by a complete connection layer, calculating the characteristics of a defect node and a semantic object node, and connecting the characteristics of the self node and the related nodes as a final characteristic vector f of the defect typei；
S6, the final feature vector fiFeeding to a fully connected layer, calculating a score vector s ═ s for each type of defect according to the following formula1,s2,...,sM}，
si＝Wfi+b
Wherein the defect type with the largest score is the defect type of the bolt defect.
Optionally, in step S1, constructing a bolt pair defect data set specifically includes: constructing a coarse defect data set which comprises four types of data including a missing gasket, a missing pin and a gasket; or constructing a fine defect data set which comprises a missing gasket, a visually visible missing pin, a visually invisible missing pin, a missing gasket + a visually visible missing pin, a missing gasket + a visually invisible missing pin and complete data.
Optionally, in step S2, the extracting the feature of the bolt-nut pair combined area specifically includes: clipping the defect data set of the latch pair into three regions, wherein one region is the latch pairTwo additional separate areas containing bolts and nuts, respectively, and extracting features from these three areas using the Faster R-CNN detector; these features are then concatenated with position information encoding the geometric features of the bolt and nut and fed to a fully-connected layer to produce a final feature d-dimensional feature vector fh＝RdThe feature is used as an input feature for the defective node to initialize the defective feature.
Optionally, in step S3, extracting semantic object region features specifically includes: and taking the detected area of the bolt pair as a semantic object detection area, comparing all semantic object coordinates in the picture obtained after pre-training with the joint coordinates of the bolt pair, if the semantic object coordinates meet the semantic object detection area in the joint coordinates of the bolt pair, using the o characteristic of the semantic object, otherwise, discarding, and continuously circulating the step until the comparison is finished, and extracting the characteristic of the semantic object area by using a FasterR-CNN detector.
Optionally, in step S4, initializing based on the combined region feature and the semantic object region feature of the keybolt pair to obtain a defect node and a semantic object node of the keybolt pair knowledge graph, specifically including:
using one-hot vectors to unambiguously distinguish two node types, namely defect nodes and semantic object nodes, using [1,0]And [0,1 ]]Respectively representing defective nodes and semantic object nodes, and initializing hidden state hv 0By [ [0,1 ]],0d]In this case, the hidden state is initialized when the time step t is 0, and is expressed as
Figure BDA0002459529540000031
Wherein d-dimensional feature vector fh＝RdThe feature is used as an input feature of a defect node to initialize a defect feature, and a d-dimensional feature vector fo＝RdThe feature is used as an input feature for semantic object nodes to initialize defect features.
Optionally, in step S5, the output network implemented by the full connectivity layer is used to calculate the node level characteristics,calculating the characteristics of the defective nodes and the semantic object nodes, and connecting the characteristics of the nodes and the characteristics of the related nodes as final characteristic vectors f of the defective nodes and the semantic object nodesiThe method specifically comprises the following steps:
s501, at each time step, each node firstly aggregates messages from its neighboring nodes, which is represented as:
Figure BDA0002459529540000032
wherein A isvIs a sub-matrix of the adjacency matrix A and represents the connection of the node v and the neighbor nodes thereof, then, the model combines the information from other nodes and the previous time step, updates the hidden state of each node through the gating mechanism of the gating cycle unit, passes through T cycles of the formula (3),
Figure BDA0002459529540000033
where σ and tanh are the logical sigmoid and hyperbolic tangent functions, respectively, z and r are the update and reset gates,
Figure BDA0002459529540000034
the forgetting information is controlled to be left,
Figure BDA0002459529540000035
control newly generated information, W represents the weight of the input, U represents the weight of the sample input at that moment,
Figure BDA0002459529540000036
the state is hidden for the other nodes and,
Figure BDA0002459529540000037
is the node hidden state of the previous time step,
Figure BDA0002459529540000038
hiding the state for the updated node;
s502, calculating node-level characteristics by adopting an output network realized by a complete connection layer
Figure BDA0002459529540000041
Representing the output characteristics of the defective node and the semantic object node as { or1,or2,...,orMAnd { o }o1,oo2,..,ooN}；
For defect class riConnecting the characteristics of its own node with those of the nodes involved as its final characteristics, i.e.
fi＝[ori,αi1oo1,αi2oo2,...,αiNooN](5)
Wherein N isiIs a neighbor node set of node i in the graph, if semantic object node j does not belong to NiThen α will beijIs designated 0 if the semantic object node j belongs to NiThen α will beijIs designated as 1.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects: according to the bolt defect classification method based on the bolt-nut pair knowledge graph, provided by the invention, a bolt is not directly researched, but a bolt-nut pair is researched by combining the bolt and the nut which are connected together, firstly, the whole nut-bolt pin gasket is taken as a whole to be researched, and not only a single defect on the bolt is simply researched; secondly, defect nodes and semantic object nodes of the knowledge graph are easy to construct by combining bolt-nut pairs, and bolt-nut areas can be distinguished independently while the bolt-nut pairs are combined, so that the defect nodes can be initialized; and thirdly, the detection range is increased, the detection omission is reduced fundamentally, and the detection omission can be caused radically if the gasket or the pin cannot be detected by singly researching the bolt or the nut. In addition, since there may be multiple latch pairs in a picture, it may cause that detecting a latch pair will falsely detect the pads or pins of other latch pairs, which will fundamentally result in false detection. The invention utilizes the prior knowledge of the key pair represented by the graph structure to construct the key pair knowledge graph, and can most effectively and intuitively express the association between the defects of the key pair.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a block flow diagram of a bolt defect classification method based on a bolt-mother-to-knowledge-map according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating the accuracy and recall of a classification experiment for a coarse-scale defect image according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating the accuracy and recall of classification experiments for fine defect images according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention aims to provide a bolt defect classification method based on a bolt-pair knowledge graph, which sufficiently considers the relation between a bolt and a nut, provides a method for extracting bolt-pair characteristics by using a convolutional neural network and combines priori knowledge to construct the bolt-pair knowledge graph, and efficiently finishes bolt defect classification.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
As shown in FIG. 1, the bolt defect classification method based on the bolt mother-to-knowledge graph provided by the invention comprises the following steps:
s1, constructing a bolt pair defect data set;
s2, extracting the feature of the bolt-nut pair combined area;
s3, extracting semantic object region features;
s4, constructing a embolus pair knowledge graph based on the GGNN model, which specifically comprises the following steps: initializing to obtain a defect node and a semantic object node of a latch pair knowledge graph based on the latch pair joint region feature and the semantic object region feature, and obtaining an edge of the latch pair knowledge graph based on an adjacent matrix, wherein the adjacent matrix represents the association between the defect node and the semantic object node;
s5, calculating node-level characteristics by adopting an output network realized by a complete connection layer, calculating the characteristics of a defect node and a semantic object node, and connecting the characteristics of the self node and the related nodes as a final characteristic vector f of the defect typei；
S6, the final feature vector fiFeeding to a fully connected layer, calculating a score vector s ═ s for each type of defect according to the following formula1,s2,...,sM}，
si＝Wfi+b
Wherein the defect type with the largest score is the defect type of the bolt defect.
In step S1, constructing a bolt pair defect data set specifically includes: constructing a coarse defect data set which comprises four types of data including a missing gasket, a missing pin and a gasket; or constructing a fine defect data set which comprises a missing gasket, a visually visible missing pin, a visually invisible missing pin, a missing gasket + a visually visible missing pin, a missing gasket + a visually invisible missing pin and complete data. The results of knowledge-graph-guided classification of coarse and fine defect images based on the GGNN's emboli are reflected in fig. 2 and 3, respectively.
For the coarse defect data set, the bolt pair data set is divided into a training set of a plurality of images and relation examples and a testing set of a plurality of images and examples. For the fine defect data set, manually re-annotating the data set with the keycodes of the coarse defect data set into fine defect categories. And dividing the fine defect data set into a training set of a plurality of images and relation examples and a test set of the plurality of images and relation examples. For the fine defect data set, manually re-annotating the data set with the keycodes of the coarse defect data set into fine defect categories.
In step S2, the extracting of the feature of the bolt-nut pair combined area specifically includes: cutting the bolt pair defect data set into three regions, wherein one region is a bolt pair region, the other two regions respectively comprise bolt and nut single regions, and extracting features from the three regions by using a Faster R-CNN detector; these features are then concatenated with position information encoding the geometric features of the bolt and nut and fed to a fully-connected layer to produce a final feature d-dimensional feature vector fh＝RdThe feature is used as an input feature for the defective node to initialize the defective feature.
Wherein all components of the ResNet-101 model and VGG-16 model employed by the Faster R-CNN detector were trained using stochastic gradient descent, except for GGNN which was trained using moment-of-adaptation estimation. And respectively extracting the characteristics of the latch pair region and the semantic object region by utilizing a widely used ResNet-101 model and a VGG-16 model. For the GGNN propagation model, the hidden state dimension is set to 4098, the output feature dimension is set to 512, and the iteration time T is set to 5.
In step S3, extracting semantic object region features specifically includes: and taking the detected area of the bolt pair as a semantic object detection area, comparing all semantic object coordinates in the picture obtained after pre-training with the joint coordinates of the bolt pair, if the semantic object coordinates meet the semantic object detection area in the joint coordinates of the bolt pair, using the o characteristic of the semantic object, otherwise, discarding, and continuously circulating the step until the comparison is finished, and extracting the characteristic of the semantic object area by using a fast R-CNN detector.
The input features of the nodes of the semantic object o detected by the semantic object detection region are initialized by the features extracted from the corresponding semantic object region, otherwise, by the d-dimensional zero vector.
In step S4, initializing the combined region features and semantic object region features based on the keybolt pair to obtain a defect node and a semantic object node of the keybolt pair knowledge graph, which specifically include:
using one-hot vectors to unambiguously distinguish two node types, namely defect nodes and semantic object nodes, using [1,0]And [0,1 ]]Respectively representing defective nodes and semantic object nodes, and initializing hidden state hv0 [ [0,1 ]],0d]In this case, the hidden state is initialized when the time step t is 0, and is expressed as
Figure BDA0002459529540000071
Wherein d-dimensional feature vector fh＝RdThe feature is used as an input feature of a defect node to initialize a defect feature, and a d-dimensional feature vector fo＝RdThe feature is used as an input feature for semantic object nodes to initialize defect features.
In the step S5, the output network implemented by the complete connection layer is used to calculate the node-level features, calculate the features of the defective nodes and semantic object nodes, and for the defect category, connect the features of its own node and the related nodes as the final feature vector fiThe method specifically comprises the following steps:
s501, at each time step, each node firstly aggregates messages from its neighboring nodes, which is represented as:
Figure BDA0002459529540000072
wherein A isvIs a sub-matrix of the adjacency matrix A and represents the connection of the node v and the neighbor nodes thereof, then, the model combines the information from other nodes and the previous time step, updates the hidden state of each node through the gating mechanism of the gating cycle unit, passes through T cycles of the formula (3),
Figure BDA0002459529540000073
wherein σ and tanh are respectively a logical sigmoid and a hyperbolic tangent function; and represents an element multiplication operation. z and r are update and reset gates,
Figure BDA0002459529540000081
the forgetting information is controlled to be left,
Figure BDA0002459529540000082
control newly generated information, W represents the weight of the input, U represents the weight of the sample input at that moment,
Figure BDA0002459529540000083
the state is hidden for the other nodes and,
Figure BDA0002459529540000084
is the node hidden state of the previous time step,
Figure BDA0002459529540000085
hiding the state for the updated node; the model then updates the hidden state of each node through a gating mechanism similar to a gated round robin unit, with information from other nodes and the previous time step, through T cycles of equation (3). In this way, each node can aggregate information from its neighbor nodes while transmitting its own messages to other neighbor nodes, thereby achieving interaction between all nodes. After time T interaction, the node messages have propagated through the graph, and the final hidden state of each node, i.e., { h }, can be obtainedT 1,hT 2,....,HT |V|}。
S502, calculating node-level characteristics by adopting an output network realized by a complete connection layer
Figure BDA0002459529540000086
Representing the output characteristics of the defective node and the semantic object node as { or1,or2,...,orMAnd { o }o1,oo2,..,ooN}；
For defect class riConnecting the characteristics of its own node with those of the nodes involved as its final characteristics, i.e.
fi＝[ori,αi1oo1,αi2oo2,...,αiNooN](5)
Wherein N isiIs a neighbor node set of node i in the graph, if semantic object node j does not belong to NiThen α will beijIs designated 0 if the semantic object node j belongs to NiThen α will beijIs designated as 1.
The method adopts a FasterR-CNN detector to extract characteristics and uses a gated neural network (GGNN) model to carry out message propagation among nodes, then the final hidden state of a defective node and the final hidden state of a semantic object node are fused to obtain final characteristics, the node level characteristics are calculated through a complete connection layer, and finally, a classification prediction result is calculated through the complete connection layer. The invention provides a bolt-nut pair knowledge graph constructed by combining the expression of related prior knowledge of bolts and nuts on a power transmission line, and a ResNet-101 model and a VGG-16 model are respectively adopted as a feature extractor of a FasterR-CNN deep learning framework and a gated neural network (GGNN) model for message propagation between nodes. The current research on the power transmission line is limited to the changes of the existing algorithm, such as convolution kernel adjustment, network depth increase, application parameter adjustment or data set adjustment aiming at the algorithm. The method can extract the characteristics of the object by adopting the ResNet-101 model and the VGG-16 model as the characteristic extractor of the FasterR-CNN deep learning framework respectively, so as to initialize the defect nodes and semantic object nodes of the GGNN model, and can effectively represent the bolt-to-graph nodes. The invention utilizes the prior knowledge of the key pair represented by the graph structure to construct the key pair knowledge graph, and can most effectively and intuitively express the association between the defects of the key pair.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.

Claims (6)

1. A bolt defect classification method based on a bolt mother pair knowledge graph is characterized by comprising the following steps:
s1, constructing a bolt pair defect data set;
s2, extracting the feature of the bolt-nut pair combined area;
s3, extracting semantic object region features;
s4, constructing a embolus pair knowledge graph based on the GGNN model, which specifically comprises the following steps: initializing to obtain a defect node and a semantic object node of a latch pair knowledge graph based on the latch pair joint region feature and the semantic object region feature, and obtaining an edge of the latch pair knowledge graph based on an adjacent matrix, wherein the adjacent matrix represents the association between the defect node and the semantic object node;
s5, calculating node-level characteristics by adopting an output network realized by a complete connection layer, calculating the characteristics of a defect node and a semantic object node, and connecting the characteristics of the self node and the related nodes as a final characteristic vector f of the defect typei；
S6, the final feature vector fiFeeding to a fully connected layer, calculating a score vector s ═ s for each type of defect according to the following formula1,s2,...,sM}，
si＝Wfi+b
Wherein the defect type with the largest score is the defect type of the bolt defect.
2. The bolt defect classification method based on the bolt-pair knowledge-graph of claim 1, wherein in the step S1, constructing a bolt-pair defect data set specifically comprises: constructing a coarse defect data set which comprises four types of data including a missing gasket, a missing pin and a gasket; or constructing a fine defect data set which comprises six types of complete data including lack of a gasket, visually visible lack of a pin, visually invisible lack of a pin, lack of a gasket + visually visible lack of a pin, lack of a gasket + visually invisible lack of a pin.
3. The bolt defect classification method based on the bolt mother pair knowledge-graph according to claim 1, wherein in the step S2, the extracting the bolt mother pair combined region features specifically comprises: cutting the bolt pair defect data set into three regions, wherein one region is a bolt pair region, the other two regions respectively comprise bolt and nut single regions, and extracting features from the three regions by using a FasterR-CNN detector; these features are then concatenated with position information encoding the geometric features of the bolt and nut and fed to a fully-connected layer to produce a final feature d-dimensional feature vector fh＝RdThe feature is used as an input feature for the defective node to initialize the defective feature.
4. The bolt defect classification method based on the bolt-pair knowledge-graph of claim 1, wherein in the step S3, extracting semantic object region features specifically comprises: and taking the detected area of the bolt pair as a semantic object detection area, comparing all semantic object coordinates in the picture obtained after pre-training with the joint coordinates of the bolt pair, if the semantic object coordinates meet the semantic object detection area in the joint coordinates of the bolt pair, using the o characteristic of the semantic object, otherwise, discarding, and continuously circulating the step until the comparison is finished, and extracting the characteristic of the semantic object area by using a fast R-CNN detector.
5. The bolt defect classification method based on the bolt-pair knowledge-graph according to claim 1, wherein in the step S4, the defect node and the semantic object node of the bolt-pair knowledge-graph are obtained through initialization based on the bolt-pair combined region feature and the semantic object region feature, and specifically includes:
using one-hot vectors to unambiguously distinguish two node types, namely defect nodes and semantic object nodes, using [1,0]And [0,1 ]]Respectively indicate a defectTrap nodes and semantic object nodes, other cases, initialize hidden states
Figure FDA0002459529530000021
By [ [0,1 ]],0d]In this case, the hidden state is initialized when the time step t is 0, and is expressed as
Figure FDA0002459529530000022
Wherein d-dimensional feature vector fh＝RdThe feature is used as an input feature of a defect node to initialize a defect feature, and a d-dimensional feature vector fo＝RdThe feature is used as an input feature for semantic object nodes to initialize defect features.
6. The bolt defect classification method based on bolt-pair knowledge-graph of claim 5, characterized in that in step S5, the output network implemented by the complete connection layer is used to calculate node-level features, calculate the features of the defect node and semantic object node, and for defect category, connect the features of its own node and the involved node as its final feature vector fiThe method specifically comprises the following steps:
s501, at each time step, each node firstly aggregates messages from its neighboring nodes, which is represented as:
Figure FDA0002459529530000023
wherein A isvIs a sub-matrix of the adjacency matrix A and represents the connection of the node v and the neighbor nodes thereof, then, the model combines the information from other nodes and the previous time step, updates the hidden state of each node through the gating mechanism of the gating cycle unit, passes through T cycles of the formula (3),
Figure FDA0002459529530000031
where σ and tanh are the logical sigmoid and hyperbolic tangent functions, respectively, z and r are the update and reset gates,
Figure FDA0002459529530000032
the forgetting information is controlled to be left,
Figure FDA0002459529530000033
control newly generated information, W represents the weight of the input, U represents the weight of the sample input at that moment,
Figure FDA0002459529530000034
the state is hidden for the other nodes and,
Figure FDA0002459529530000035
is the node hidden state of the previous time step,
Figure FDA0002459529530000036
hiding the state for the updated node;
s502, calculating node-level characteristics by adopting an output network realized by a complete connection layer
Figure FDA0002459529530000037
Representing the output characteristics of the defective node and the semantic object node as { or1,or2,...,orMAnd { o }o1,oo2,..,ooN}；
For defect class riConnecting the characteristics of its own node with those of the nodes involved as its final characteristics, i.e.
fi＝[ori,αi1oo1,αi2oo2,...,αiNooN](5)
Wherein N isiIs a neighbor node set of node i in the graph, if semantic object node j does not belong to NiThen α will beijAssigned as 0, if semantic object node jIs of NiThen α will beijIs designated as 1.
CN202010316196.0A 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph Active CN111523589B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010316196.0A CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010316196.0A CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Publications (2)

Publication Number Publication Date
CN111523589A true CN111523589A (en) 2020-08-11
CN111523589B CN111523589B (en) 2023-05-23

Family

ID=71910571

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010316196.0A Active CN111523589B (en) 2020-04-21 2020-04-21 Bolt defect classification method based on bolt pair knowledge graph

Country Status (1)

Country Link
CN (1) CN111523589B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115391563A (en) * 2022-09-01 2022-11-25 广东工业大学 Knowledge graph link prediction method based on multi-source heterogeneous data fusion

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07100660A (en) * 1993-10-05 1995-04-18 Hirata:Kk Method and device for detecting nut defective welding
CN110084296A (en) * 2019-04-22 2019-08-02 中山大学 A kind of figure expression learning framework and its multi-tag classification method based on certain semantic
CN110263858A (en) * 2019-06-21 2019-09-20 华北电力大学（保定） A kind of bolt image composition method, device and relevant device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07100660A (en) * 1993-10-05 1995-04-18 Hirata:Kk Method and device for detecting nut defective welding
CN110084296A (en) * 2019-04-22 2019-08-02 中山大学 A kind of figure expression learning framework and its multi-tag classification method based on certain semantic
CN110263858A (en) * 2019-06-21 2019-09-20 华北电力大学（保定） A kind of bolt image composition method, device and relevant device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ZHENBING ZHAO ET AL.: "Detection Method Based on Automatic Visual Shape Clustering for Pin-Missing Defect in Transmission Lines" *
刘梓权;王慧芳;: "基于知识图谱技术的电力设备缺陷记录检索方法" *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115391563A (en) * 2022-09-01 2022-11-25 广东工业大学 Knowledge graph link prediction method based on multi-source heterogeneous data fusion
CN115391563B (en) * 2022-09-01 2024-02-06 广东工业大学 Knowledge graph link prediction method based on multi-source heterogeneous data fusion

Also Published As

Publication number Publication date
CN111523589B (en) 2023-05-23

Similar Documents

Publication Publication Date Title
CN105930868B (en) A kind of low resolution airport target detection method based on stratification enhancing study
CN112132818B (en) Pulmonary nodule detection and clinical analysis method constructed based on graph convolution neural network
CN109446894B (en) Multispectral image change detection method based on probability segmentation and Gaussian mixture clustering
CN109919108A (en) Remote sensing images fast target detection method based on depth Hash auxiliary network
CN110472483A (en) A kind of method and device of the small sample semantic feature enhancing towards SAR image
CN105930686B (en) A kind of secondary protein structure prediction method based on deep neural network
CN106682697A (en) End-to-end object detection method based on convolutional neural network
Xu et al. Pavement crack detection algorithm based on generative adversarial network and convolutional neural network under small samples
CN109743642B (en) Video abstract generation method based on hierarchical recurrent neural network
CN106023257A (en) Target tracking method based on rotor UAV platform
CN109859245A (en) Multi-object tracking method, device and the storage medium of video object
CN111539422B (en) Flight target cooperative identification method based on fast RCNN
Arnab et al. Higher order potentials in end-to-end trainable conditional random fields
CN114937066A (en) Point cloud registration system and method based on cross offset features and space consistency
Wang et al. Low-altitude infrared small target detection based on fully convolutional regression network and graph matching
CN112364747A (en) Target detection method under limited sample
CN111523589A (en) Bolt defect classification method based on bolt pair knowledge graph
CN115859781A (en) Flow field prediction method based on attention and convolutional neural network codec
Wang et al. Fusionnet: Coarse-to-fine extrinsic calibration network of lidar and camera with hierarchical point-pixel fusion
CN107871315B (en) Video image motion detection method and device
Fu et al. Monocular depth estimation based on multi-scale graph convolution networks
CN110826575A (en) Underwater target identification method based on machine learning
Zhao et al. A knowledge-guided model of fitting detection in aerial transmission line images
CN111008977B (en) Image segmentation method of high-order MRF model based on multi-node topological overlap measure
CN111161280B (en) Contour evolution segmentation method based on neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant