CN110210387B - Method, system and device for detecting insulator target based on knowledge graph - Google Patents
Method, system and device for detecting insulator target based on knowledge graph Download PDFInfo
- Publication number
- CN110210387B CN110210387B CN201910468534.XA CN201910468534A CN110210387B CN 110210387 B CN110210387 B CN 110210387B CN 201910468534 A CN201910468534 A CN 201910468534A CN 110210387 B CN110210387 B CN 110210387B
- Authority
- CN
- China
- Prior art keywords
- insulator
- image
- network
- target
- knowledge
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000012212 insulator Substances 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000001514 detection method Methods 0.000 claims abstract description 60
- 238000000605 extraction Methods 0.000 claims abstract description 38
- 238000012549 training Methods 0.000 claims description 63
- 238000002372 labelling Methods 0.000 claims description 15
- 238000010586 diagram Methods 0.000 claims description 9
- 230000000877 morphologic effect Effects 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 238000009413 insulation Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 238000011897 real-time detection Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 9
- 230000005540 biological transmission Effects 0.000 description 7
- 238000011161 development Methods 0.000 description 5
- 230000018109 developmental process Effects 0.000 description 5
- 238000005065 mining Methods 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000011176 pooling Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 125000006850 spacer group Chemical group 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of power management and image detection, and particularly relates to a method, a system and a device for detecting an insulator target based on a knowledge graph, aiming at solving the problems that the existing target detection algorithm is slow in speed and difficult to apply to real-time detection in a specific field. The method comprises the following steps: based on the obtained image containing the insulator, adopting an insulator target detection network to obtain and output an image of an insulator target candidate frame; the insulator target detection network comprises a feature extraction network, a target candidate frame generation network and a classification network. On one hand, the method introduces the knowledge map, provides richer semantic relations and enhances the learning ability of the machine; on the other hand, the space domain form consistency characteristic is utilized, so that the precision is ensured, and the detection speed is accelerated.
Description
Technical Field
The invention belongs to the field of power management and image detection, and particularly relates to a method, a system and a device for detecting an insulator target based on a knowledge graph.
Background
Under the development trend that realizes unmanned aerial vehicle aerial photography electric power inspection gradually, will collect the transmission line image data that the data bulk is more and more huge, but these image backgrounds are complicated, and processing and feedback to these data content become very tricky problem, and traditional manual work is low and easy the missed measure to the detection efficiency of image, and image processing untimely then can lead to overhauing untimely and breaking down. The development and application of computer vision technology provides a method and a way for solving the problem of image data processing. At present, in many countries in the world, research in the fields of computer vision and image processing, including target recognition, target detection, image classification and retrieval, and the like, lays a foundation for the development of the electric power professional field.
The task of object detection is to find all objects of interest (objects) in the image and determine their positions. Object detection is one of the core problems in the field of machine vision. Because various objects have different appearances, shapes and postures, and interference of factors such as illumination, shielding and the like during imaging is added, target detection is always one of the most challenging problems in the field of machine vision. In recent years, with the dramatic improvement of image detection and classification accuracy by deep learning, target detection algorithms based on deep learning have become mainstream.
The existing target detection algorithm can generate about 2000 candidate regions, however, in a real scene, one image does not contain so many targets, so that most of candidate frames generated by the algorithm are redundant, which causes high calculation cost, and the existing detection algorithm is low in detection speed and difficult to apply to real-time detection in a specific field.
Disclosure of Invention
In order to solve the above problems in the prior art, that is, the problems that the existing target detection algorithm is slow and is difficult to be applied to real-time detection in a specific field, the invention provides a method for detecting an insulator target based on a knowledge graph, which comprises the following steps:
step S10, acquiring an image containing the insulator as an image to be detected;
step S20, adopting an insulator target detection network, and acquiring an image with an insulator target candidate frame based on the image to be detected;
step S30, outputting the acquired image;
the insulation sub-target detection network comprises a feature extraction network, a target candidate box generation network and a classification network, and the training method comprises the following steps:
step B10, constructing a knowledge graph based on the acquired training image set;
step B20, randomly selecting a batch of images in the training image set, adjusting the images to a preset size, and extracting features by using a feature extraction network to obtain a feature map; adopting a target candidate frame generation network to obtain significance characteristics, and combining spatial priori knowledge in a knowledge graph to obtain a training image with the target candidate frame and an artificial labeling frame;
step B30, inputting the feature map and the training image with the target candidate frame into a classification network, calculating the total loss value of the network, and updating the network parameters;
and step B40, repeating the steps B20-B30 until reaching the preset training end condition, and obtaining the trained insulator sub-target detection network.
In some preferred embodiments, the training image set is obtained by:
manually marking the insulator of each image in the insulator image set by adopting a rectangular frame, and generating a corresponding file as a label of the image; the file comprises coordinates of the upper left corner point of the manual marking frame and the height and width of the marking frame.
In some preferred embodiments, the knowledge-graph is constructed by:
step B101, acquiring text description of image data in the training image set and a corpus of relevant knowledge of the electric power field where the insulator is located;
step B102, screening the text description of the image data and the entity vocabulary in the corpus of the relevant knowledge of the electric power field where the insulator is located by utilizing context information, verb phrases, relationship descriptions and the like, and defining names for the entity vocabulary;
step B103, based on the names of the entity vocabularies, describing the incidence relation among various entity vocabularies by using the relation;
and step B104, constructing entity links based on the entity vocabularies, the names of the entity vocabularies and the incidence relations among all kinds of entity vocabularies, and forming a knowledge graph.
In some preferred embodiments, in step B20, "obtaining a saliency feature by using a target candidate box generation network, and obtaining a training image with a target candidate box and an artificial labeling box by combining spatial prior knowledge in a knowledge graph", the method includes:
step B201, converting one image in the randomly selected training image set from an RGB space to an HIS space, extracting a significant feature based on color comparison and extracting a significant feature based on airspace morphological comparison;
and step B202, according to the spatial priori knowledge in the knowledge graph, fusing the color comparison-based saliency features and the airspace morphology comparison-based saliency features, and carrying out binarization by adopting a preset threshold value to remove a pseudo target with a smaller area so as to obtain a training image with a target candidate frame and an artificial labeling frame.
In some preferred embodiments, "extracting salient features based on color contrast" is performed by:
and calculating the color contrast through nonparametric color distribution based on the color histogram contrast of the HIS space image to obtain the significance characteristic of each pixel based on the color contrast.
In some preferred embodiments, "extracting salient features based on spatial domain morphological contrast" is performed by:
and extracting the gradient direction and gradient value of a pixel point of the HIS space image as structural features, and obtaining the saliency features of each pixel based on spatial domain morphological comparison.
In some preferred embodiments, step B20 ″ selects an image in the training image set at random, adjusts the image to a preset size, and extracts features using a feature extraction network to obtain a feature map; the method comprises the following steps of adopting a target candidate frame to generate a network to obtain significance characteristics, combining spatial prior knowledge in a knowledge graph, obtaining a training image with the target candidate frame and an artificial labeling frame, and then setting ROI Pooling, wherein the method comprises the following steps:
dividing target candidate frames with different sizes into grids with preset sizes, mapping the grids to corresponding image feature maps, and extracting the maximum value in each grid as the output value of the grid to obtain the feature map with the preset size.
In some preferred embodiments, the network total loss value is obtained by weighting the loss value corresponding to the category to which the target belongs and the target position loss value.
On the other hand, the invention provides an insulator target detection system based on a knowledge graph, which comprises an input module, an insulator detection module and an output module;
the input module is configured to acquire and input an image containing an insulator as an image to be detected;
the insulator detection module is configured to adopt an insulator target detection network and obtain an image of an insulator target candidate frame based on the image to be detected;
the output module is configured to output the acquired image;
the insulator detection module comprises a feature extraction module, a target candidate frame generation module, a classification module and a circulation control module;
the feature extraction module is configured to randomly select a batch of images in the training image set, adjust the batch of images to a preset size and extract features by using a feature extraction network to obtain a feature map;
the target candidate frame generation module is configured to adopt a target candidate frame generation network to obtain the significance characteristics of the training image corresponding to the characteristic diagram, and obtain the training image with the target candidate frame and the artificial labeling frame by combining the spatial priori knowledge in the knowledge graph;
the classification module is configured to input the feature map and the training image with the target candidate frame into a classification network, calculate a total loss value of the network, and update network parameters;
the circulation control module is configured to control the feature extraction module, the target candidate box generation module and the classification module to execute circularly until a preset training end condition is reached.
In a third aspect of the present invention, a storage device is provided, in which a plurality of programs are stored, the programs being adapted to be loaded and executed by a processor to implement the above-mentioned method for knowledge-graph-based insulator target detection.
In a fourth aspect of the present invention, a processing apparatus is provided, which includes a processor, a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing a plurality of programs; the program is adapted to be loaded and executed by a processor to implement the above-described knowledge-graph based insulator target detection method.
The invention has the beneficial effects that:
(1) the method for detecting the insulator target based on the knowledge graph adds the knowledge graph, provides richer semantic relations and enhances the learning capability of a machine.
(2) According to the method, the insulator sub-region candidate frames with fewer quantity and higher quality are obtained by utilizing the airspace form consistency characteristics, and the target detection speed is accelerated on the premise that the accuracy is not reduced or is improved by combining with a deep learning method.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is a system flow diagram of a method for knowledge-graph based insulator target detection according to the present invention;
FIG. 2 is a schematic diagram of a knowledge-graph construction process of an embodiment of the method for detecting an insulator target based on a knowledge-graph of the present invention;
FIG. 3 is an example knowledge graph of a method for knowledge graph-based insulator target detection according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a feature extraction network according to an embodiment of the method for detecting an insulator target based on a knowledge graph;
fig. 5 is a schematic view of a target candidate box generation flow of an embodiment of the method for detecting an insulator target based on a knowledge graph according to the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention discloses a knowledge graph-based insulator target detection method, which comprises the following steps:
step S10, acquiring an image containing the insulator as an image to be detected;
step S20, adopting an insulator target detection network, and acquiring an image with an insulator target candidate frame based on the image to be detected;
step S30, outputting the acquired image;
the insulation sub-target detection network comprises a feature extraction network, a target candidate box generation network and a classification network, and the training method comprises the following steps:
step B10, constructing a knowledge graph based on the acquired training image set;
step B20, randomly selecting a batch of images in the training image set, adjusting the images to a preset size, and extracting features by using a feature extraction network to obtain a feature map; adopting a target candidate frame generation network to obtain significance characteristics, and combining spatial priori knowledge in a knowledge graph to obtain a training image with the target candidate frame and an artificial labeling frame;
step B30, inputting the feature map and the training image with the target candidate frame into a classification network, calculating the total loss value of the network, and updating the network parameters;
and step B40, repeating the steps B20-B30 until reaching the preset training end condition, and obtaining the trained insulator sub-target detection network.
In order to more clearly explain the method for detecting the insulator target based on the knowledge graph, the following describes the steps in the embodiment of the method in detail with reference to fig. 1.
The insulator target detection method based on the knowledge graph comprises the steps of S10-S30, and the steps are described in detail as follows:
step S10, acquiring an image including the insulator as an image to be detected.
In recent years, with the increasing of the voltage grade of the power transmission line, the power consumption demand of various industries is continuously increased, the mileage of the power transmission line is also increased in a blowout manner, and the guarantee of safe and stable operation of the power transmission line becomes more important. In order to relieve the operation and maintenance pressure of the power transmission line, improve the quality and efficiency of line inspection and ensure the safety and stability of a power grid, more and more power supply companies start to apply unmanned aerial vehicles to perform line inspection. However, the background of the images is complex, and the traditional manual processing efficiency is low and the omission ratio is high. With the development of computer vision technology, insulator target detection by using a computer is becoming mainstream gradually.
And step S20, acquiring an image with an insulator target candidate frame based on the image to be detected by adopting an insulator target detection network.
The target detection, also called target extraction, is an image segmentation based on target geometry and statistical characteristics, which combines the segmentation and identification of targets into one, and the accuracy and real-time performance of the method are important capabilities of the whole system. Especially, in a complex scene, when a plurality of targets need to be processed in real time, automatic target extraction and identification are particularly important.
The insulator sub-target detection network comprises a feature extraction network, a target candidate box generation network and a classification network, and the training method comprises the following steps:
and step B10, constructing a knowledge graph based on the acquired training image set.
The acquisition method of the training image set comprises the following steps:
manually marking the insulator of each image in the insulator image set by adopting a rectangular frame, and generating a corresponding file as a label of the image; the file comprises coordinates of the upper left corner point of the manual marking frame and the height and width of the marking frame.
The appearance of the knowledge graph benefits from the development of technologies such as big data and machine learning, and is a representative technology of knowledge engineering, and the goal of the knowledge engineering is to integrate the domain and expert knowledge into a machine, so that the machine can solve the problem like an expert. The essence of the knowledge graph is a large-scale semantic network with hundreds of millions of or even more nodes, the semantic network expresses entities, concepts and various semantic relations among the entities and the concepts in the big data era, and the knowledge graph is applied to reasonably express the knowledge in a machine in a semantic network mode.
There are many nodes in the knowledge graph, and these nodes are very rich in composition, and basically contain entities, concepts, attributes and relationships. The entity is the basis on which the attribute occurs and can exist independently, for example, the umbrella skirt has the attribute of material, so the umbrella skirt is the entity, and the entity and the attribute are inseparable. The concept is a class of events with common attributes, such as insulators, power lines, etc. If there is entity concept in the map, it can help the machine to realize the capability of entity classification, which is called as category in the academia. The attributes refer to physical or conceptual attributes, such as the number of sheds of the insulator, the shape of the spacer, and the like. The relation refers to the relation between entities, concepts and concepts or between entities and concepts, for example, the insulator is connected with the power transmission line, and the shed and the insulator are in a forming relation.
As shown in fig. 2, which is a schematic diagram of a knowledge graph construction process of an embodiment of the method for detecting an insulator target based on a knowledge graph of the present invention, a relevant corpus is first obtained, then vocabulary and entity mining are performed, then relationship extraction is performed, and finally entity linking and storage are performed to obtain a constructed knowledge graph.
The construction method of the knowledge graph comprises the following steps:
and step B101, acquiring text description of image data in the training image set and a corpus of relevant knowledge of the electric power field where the insulator is located.
The essence of knowledge graph construction is to structure unstructured data, in our research, unstructured data is an insulator image data set, and text description of image data and a corpus of more power domain related knowledge need to be obtained to extract semantic networks included in the images. These corpora based on insulator image data sets include a large number of key words in the power field.
And step B102, screening the text description of the image data and the entity vocabulary in the corpus of the relevant knowledge of the electric power field where the insulator is located by utilizing context information, verb phrases, relationship descriptions and the like, and defining names for the entity vocabulary.
A topic word list of the electric power field is established by utilizing the corpus, which is the process of mining words. And then, entity mining is carried out, namely, context information, verb phrases, relationship description and the like are fully utilized to infer whether the vocabulary is an entity, if so, corresponding names are defined, in the process, the entity is taken as a main object, data of different sources in the corpus are mapped and merged, the description of the entity in different data sources is represented by attributes, and all-round description of the entity is formed. One of the bases for constructing the knowledge graph is obtained by performing vocabulary mining and entity mining on the corpus.
And step B103, describing the association relationship among various entity vocabularies by using the relationship based on the names of the entity vocabularies.
Relational extraction is to use relations to describe the incidence relation among data of various abstract modeling entities so as to support incidence analysis. Relationship extraction can be modeled as a relationship classification problem, i.e., classifying candidate entity pairs into known relationships that often hold between entity pairs of a particular type, so the category labels of named entities help with relationship classification. Relational extraction can help to achieve structuring of image data. In order to obtain more labeled data, the existing seed entity pairs can be utilized, and more seed entity pairs can be obtained by using an iterative updating extraction method. In the research, information extraction is performed by adopting a method of CloseIE, which is an information extraction method for extracting information based on field professional knowledge and oriented to a specific field, and the extraction relationship type needs to be predefined, so that the research has small scale and higher precision, and is very suitable for the requirements in the research.
And step B104, constructing entity links based on the entity vocabularies, the names of the entity vocabularies and the incidence relations among all kinds of entity vocabularies, and forming a knowledge graph.
With the entities and the relations, the next step is to construct entity links, namely, the related storage of various types of data surrounding the entities is realized. In one embodiment of the invention, Neo4j is used for storage, which is the graph database ranked first in the current graph data store, supporting raw graph storage and processing.
As shown in fig. 3, the example diagram of the knowledge graph based on the method for detecting the target of the insulator according to the embodiment of the present invention includes an entity, an attribute, a relationship between the attributes of the entity, and a relationship between the entity and the entity.
Step B20, randomly selecting a batch of images in the training image set, adjusting the images to a preset size, and extracting features by using a feature extraction network to obtain a feature map; and generating a network by adopting the target candidate frame to obtain the significance characteristics, and combining spatial prior knowledge in the knowledge graph to obtain a training image with the target candidate frame and the artificial labeling frame.
In one embodiment of the present invention, the image is resized to 224 x 224, and the resized image is input to the feature extraction portion of the feature extraction network, VGG 16.
As shown in fig. 4, which is a schematic diagram of a feature extraction network structure of an embodiment of the method for detecting an insulator target based on a knowledge graph of the present invention, the network has 15 layers, wherein: the 1 st layer and the 2 nd layer are convolution layers, the convolution kernel is 3 multiplied by 3, and the number of channels is 64; the 3 rd layer is a MaxPool layer; the 4 th layer and the 5 th layer are convolution layers, the convolution kernel is 3 multiplied by 3, and the number of channels is 128; the 6 th layer is a MaxPool layer; the 7 th layer to the 9 th layer are convolution layers, the convolution kernel is 3 multiplied by 3, and the number of channels is 256; the 10 th to 15 th layers are convolution layers, the convolution kernels of the 10 th, 11 th, 13 th and 14 th layers are 3 multiplied by 3, the convolution kernels of the 12 th and 15 th layers are 1 multiplied by 1, and the number of channels is 512.
In step B20, "adopt the target candidate box to generate the network to obtain the saliency characteristic, combine the spatial priori knowledge in the knowledge-graph, obtain the training image with target candidate box and artificial label box", its method is:
the saliency of a region in an image depends on the difference of its own features from the surroundings, which are usually reflected on color features, shape features, texture features or local features. The method adopts a salient region detection algorithm which fuses color contrast features and structure contrast features, so as to generate a target region candidate frame.
As shown in fig. 5, which is a schematic view of a target candidate frame generation process of an embodiment of the method for detecting an insulator target based on a knowledge graph of the present invention, an image is converted from an RGB space to an HIS space, saliency features based on color comparison and saliency features based on spatial domain morphology comparison are respectively extracted, feature fusion is performed according to prior knowledge of the knowledge graph, and then binarization, filtering, a threshold value is set, and a pseudo target is removed, so that an insulator target candidate region is obtained.
Step B201, converting the randomly selected training image set into an HIS space from an RGB space, extracting the significant features based on color comparison and extracting the significant features based on airspace morphological comparison.
The RGB format images in the training image set are converted into HSI format images, and three components of HSI (hue H, saturation S and intensity I) just correspond to the three aspects of the recognition of people on the classification, purity and brightness of colors. Since the hue H component does not change with brightness, contrast and the like and has higher robustness, the method adopts the H component of the image to extract the color contrast characteristic.
The method for extracting the salient features based on color contrast comprises the following steps:
and calculating the color contrast through nonparametric color distribution based on the color histogram contrast of the HIS space image to obtain the significance characteristic of each pixel based on the color contrast.
The method for extracting the salient features based on the spatial domain morphological contrast comprises the following steps:
and extracting the gradient direction and gradient value of a pixel point of the HIS space image as structural features, and obtaining the saliency features of each pixel based on spatial domain morphological comparison.
The insulators used in the power transmission line are various in types, but the insulator strings are formed in a consistent mode, each insulator string is formed by vertically arranging a central shaft and a plurality of insulator pieces which are completely the same in shape and color at equal intervals, the appearance structure of each insulator string can be similar to a long rectangle with a certain width, and therefore the insulator strings have certain regularity on the gradient characteristic, and the insulators are called to be consistent in the space domain form. Therefore, the gradient direction and the gradient value of the pixel point are extracted as structural features, and the significance feature of each pixel is obtained.
And step B202, according to the spatial priori knowledge in the knowledge graph, fusing the color comparison-based saliency features and the airspace morphology comparison-based saliency features, and carrying out binarization by adopting a preset threshold value to remove a pseudo target with a smaller area so as to obtain a training image with a target candidate frame and an artificial labeling frame.
Respectively obtaining significance characteristics based on color comparison and spatial domain form comparison, fusing the two significance characteristics according to spatial priori knowledge (derived from a knowledge graph) that pixels with the significance characteristics are generally distributed in the center of an image or are far away from the edge of the image to obtain a final significance detection result, then selecting a proper threshold value to carry out binarization on the significance detection result, removing noise points through filtering, removing a pseudo target with a smaller area by adopting a method of setting the threshold value, and using a reserved connected domain as an insulator region candidate frame.
Step B20' selecting one image in the training image set at random, adjusting the image to a preset size and extracting features by using a feature extraction network to obtain a feature map; the method comprises the following steps of adopting a target candidate frame to generate a network to obtain significance characteristics, combining spatial prior knowledge in a knowledge graph, obtaining a training image with the target candidate frame and an artificial labeling frame, and then setting ROI Pooling, wherein the method comprises the following steps:
dividing target candidate frames with different sizes into grids with preset sizes, mapping the grids to corresponding image feature maps, and extracting the maximum value in each grid as the output value of the grid to obtain the feature map with the preset size.
Since the target candidate regions (region candidates) are obtained based on the original HSI image, each target candidate region has a different size. While the subsequent full-link layer requires input of a fixed size, it is not possible to directly map target candidate regions (region templates) of different sizes to feature maps as output. The role of ROI Pooling is to extract fixed-size feature maps (feature maps) for different sizes of target candidate regions (region probes).
In one embodiment of the present invention, an h × w (height × width) target candidate region (region probability) is divided into 7 × 7 grids, then the target candidate region is mapped onto feature maps (feature maps), and finally the maximum value in each grid is extracted as the final output of the network. So regardless of the input size of ROI Pooling, its output size is always 7 × 7.
And step B30, inputting the feature map and the training image with the target candidate box into a classification network, calculating the total loss value of the network, and updating the network parameters.
And the network total loss value is obtained by weighting the loss value corresponding to the class to which the target belongs and the target position loss value.
And step B40, repeating the steps B20-B30 until reaching the preset training end condition, and obtaining the trained insulator sub-target detection network.
In step S30, the acquired image is output.
The insulator target detection system based on the knowledge graph comprises an input module, an insulator detection module and an output module;
the input module is configured to acquire and input an image containing an insulator as an image to be detected;
the insulator detection module is configured to adopt an insulator target detection network and obtain an image of an insulator target candidate frame based on the image to be detected;
the output module is configured to output the acquired image;
the insulator detection module comprises a feature extraction module, a target candidate frame generation module, a classification module and a circulation control module;
the feature extraction module is configured to randomly select a batch of images in the training image set, adjust the batch of images to a preset size and extract features by using a feature extraction network to obtain a feature map;
the target candidate frame generation module is configured to adopt a target candidate frame generation network to obtain the significance characteristics of the training image corresponding to the characteristic diagram, and obtain the training image with the target candidate frame and the artificial labeling frame by combining the spatial priori knowledge in the knowledge graph;
the classification module is configured to input the feature map and the training image with the target candidate frame into a classification network, calculate a total loss value of the network, and update network parameters;
the circulation control module is configured to control the feature extraction module, the target candidate box generation module and the classification module to execute circularly until a preset training end condition is reached.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process and related description of the system described above may refer to the corresponding process in the foregoing method embodiments, and will not be described herein again.
It should be noted that, the system for detecting an insulator target based on a knowledge graph provided in the foregoing embodiment is only illustrated by the division of the above functional modules, and in practical applications, the above functions may be allocated to different functional modules according to needs, that is, the modules or steps in the embodiment of the present invention are further decomposed or combined, for example, the modules in the foregoing embodiment may be combined into one module, or may be further split into multiple sub-modules, so as to complete all or part of the above described functions. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not to be construed as unduly limiting the present invention.
A storage device of a third embodiment of the present invention stores a plurality of programs, which are suitable for being loaded and executed by a processor to implement the above-mentioned method for detecting an insulator target based on a knowledge-map.
A processing apparatus according to a fourth embodiment of the present invention includes a processor, a storage device; a processor adapted to execute various programs; a storage device adapted to store a plurality of programs; the program is adapted to be loaded and executed by a processor to implement the above-described knowledge-graph based insulator target detection method.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes and related descriptions of the storage device and the processing device described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Those of skill in the art would appreciate that the various illustrative modules, method steps, and modules described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that programs corresponding to the software modules, method steps may be located in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. To clearly illustrate this interchangeability of electronic hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as electronic hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The terms "comprises," "comprising," or any other similar term are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.
Claims (10)
1. An insulator target detection method based on a knowledge graph is characterized by comprising the following steps:
step S10, acquiring an image containing the insulator as an image to be detected;
step S20, adopting an insulator target detection network, and acquiring an image with an insulator target candidate frame based on the image to be detected;
step S30, outputting the acquired image;
the insulation sub-target detection network comprises a feature extraction network, a target candidate box generation network and a classification network, and the training method comprises the following steps:
step B10, acquiring a training image set, text description of image data in the training image set and a corpus of relevant knowledge of the electric power field where the insulator is located;
screening the text description of the image data and the entity vocabulary in the corpus of the related knowledge of the electric power field where the insulator is located by utilizing context information, verb phrases and relationship descriptions, and defining a name for the entity vocabulary;
describing the incidence relation among various entity vocabularies by using the relation based on the names of the entity vocabularies;
constructing entity links based on the entity vocabularies, the names of the entity vocabularies and the incidence relations among all kinds of entity vocabularies to form a knowledge graph;
step B20, randomly selecting a batch of images in the training image set, adjusting the images to a preset size, and extracting features by using a feature extraction network to obtain a feature map; adopting a target candidate frame generation network to obtain significance characteristics, and combining spatial priori knowledge in a knowledge graph to obtain a training image with the target candidate frame and an artificial labeling frame;
step B30, inputting the feature map and the training image with the target candidate frame into a classification network, calculating the total loss value of the network, and updating the network parameters;
and step B40, repeating the steps B20-B30 until reaching the preset training end condition, and obtaining the trained insulator sub-target detection network.
2. The method for detecting the insulator target based on the knowledge-graph of claim 1, wherein the training image set is obtained by the method comprising the following steps:
manually marking the insulator of each image in the insulator image set by adopting a rectangular frame, and generating a corresponding file as a label of the image; the file comprises coordinates of the upper left corner point of the manual marking frame and the height and width of the marking frame.
3. The method for detecting the insulator target based on the knowledge-graph of claim 1, wherein in the step B20, "generating a network by using a target candidate frame to obtain a saliency feature, and obtaining a training image with the target candidate frame and an artificial labeling frame by combining spatial priori knowledge in the knowledge-graph" comprises the following steps:
step B201, converting one image in the randomly selected training image set from an RGB space to an HIS space, extracting a significant feature based on color comparison and extracting a significant feature based on airspace morphological comparison;
and step B202, according to the spatial priori knowledge in the knowledge graph, fusing the color comparison-based saliency features and the airspace morphology comparison-based saliency features, carrying out binarization by adopting a preset first threshold value, and removing a pseudo target with an area smaller than a preset second threshold value to obtain a training image with a target candidate frame and an artificial labeling frame.
4. The method for detecting the insulator target based on the knowledge-graph according to claim 3, wherein the method for extracting the significant features based on the color contrast comprises the following steps:
and calculating the color contrast through nonparametric color distribution based on the color histogram contrast of the HIS space image to obtain the significance characteristic of each pixel based on the color contrast.
5. The method for detecting the insulator target based on the knowledge graph according to claim 3, wherein the method for extracting the significant features based on the spatial domain morphological contrast comprises the following steps:
and extracting the gradient direction and gradient value of a pixel point of the HIS space image as structural features, and obtaining the saliency features of each pixel based on spatial domain morphological comparison.
6. The method for detecting the insulator target based on the knowledge-graph of claim 3, wherein in the step B20 ", one image in the training image set is randomly selected, adjusted to a preset size, and a feature extraction network is adopted to extract features to obtain a feature map; adopting a target candidate frame generation network to obtain the significance characteristics, combining with spatial prior knowledge in a knowledge graph, obtaining a training image with the target candidate frame and an artificial labeling frame, and then setting ROIPooling, wherein the method comprises the following steps:
dividing target candidate frames with different sizes into grids with preset sizes, mapping the grids to corresponding image feature maps, and extracting the maximum value in each grid as the output value of the grid to obtain the feature map with the preset size.
7. The method for detecting the insulator target based on the knowledge graph of claim 1, wherein the network total loss value is obtained by weighting a loss value corresponding to a category to which a target belongs and a target position loss value.
8. An insulator target detection system based on a knowledge graph is characterized by comprising an input module, an insulator detection module and an output module;
the input module is configured to acquire and input an image containing an insulator as an image to be detected;
the insulator detection module is configured to adopt an insulator target detection network and obtain an image of an insulator target candidate frame based on the image to be detected;
the output module is configured to output the acquired image;
the insulator detection module comprises a knowledge graph construction module, a feature extraction module, a target candidate frame generation module, a classification module and a circulation control module;
the knowledge graph building module is configured to obtain a training image set, text description of image data in the training image set and a corpus of relevant knowledge of the electric power field in which an insulator is located, screen entity vocabularies in the text description of the image data and the corpus of relevant knowledge of the electric power field in which the insulator is located by using context information, verb phrases and relationship description, define names for the entity vocabularies, describe association relations among various entity vocabularies by using relationships based on the names of the entity vocabularies, and build entity links to form a knowledge graph based on the entity vocabularies, the names of the entity vocabularies and the association relations among the various entity vocabularies;
the feature extraction module is configured to randomly select a batch of images in the training image set, adjust the batch of images to a preset size and extract features by using a feature extraction network to obtain a feature map;
the target candidate frame generation module is configured to adopt a target candidate frame generation network to obtain the significance characteristics of the training image corresponding to the characteristic diagram, and obtain the training image with the target candidate frame and the artificial labeling frame by combining the spatial priori knowledge in the knowledge graph;
the classification module is configured to input the feature map and the training image with the target candidate frame into a classification network, calculate a total loss value of the network, and update network parameters;
and the circulation control module is configured to control the characteristic extraction module, the target candidate box generation module and the classification module to execute circularly until a preset training end condition is reached, so as to obtain the trained insulator sub-target detection network.
9. A storage device having stored therein a plurality of programs, wherein the programs are adapted to be loaded and executed by a processor to implement the method of intellectual property graph based insulator target detection as claimed in any one of claims 1 to 7.
10. A treatment apparatus comprises
A processor adapted to execute various programs; and
a storage device adapted to store a plurality of programs;
wherein the program is adapted to be loaded and executed by a processor to perform:
the method for detecting an insulator target based on a knowledge-graph according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910468534.XA CN110210387B (en) | 2019-05-31 | 2019-05-31 | Method, system and device for detecting insulator target based on knowledge graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910468534.XA CN110210387B (en) | 2019-05-31 | 2019-05-31 | Method, system and device for detecting insulator target based on knowledge graph |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110210387A CN110210387A (en) | 2019-09-06 |
| CN110210387B true CN110210387B (en) | 2021-08-31 |
Family
ID=67789964
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910468534.XA Active CN110210387B (en) | 2019-05-31 | 2019-05-31 | Method, system and device for detecting insulator target based on knowledge graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110210387B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110750600A (en) * | 2019-10-15 | 2020-02-04 | 北京明略软件系统有限公司 | Information processing method and device |
| CN111881934A (en) * | 2020-06-18 | 2020-11-03 | 江苏大学 | Method for discovering spatial relationship between attributes and categories of electronic components |
| CN112149713B (en) * | 2020-08-21 | 2022-12-16 | 中移雄安信息通信科技有限公司 | Method and device for detecting insulator image based on insulator image detection model |
| CN112052907A (en) * | 2020-09-15 | 2020-12-08 | 浙江智慧视频安防创新中心有限公司 | Target detection method and device based on image edge information and storage medium |
| CN112184666A (en) * | 2020-09-28 | 2021-01-05 | 国网江西省电力有限公司电力科学研究院 | Target equipment detection method, device and system |
| CN113158743B (en) * | 2021-01-29 | 2022-07-12 | 中国科学院自动化研究所 | Small target real-time detection and positioning method, system and equipment based on priori knowledge |
| CN114821374B (en) * | 2022-06-27 | 2022-09-13 | 中国电子科技集团公司第二十八研究所 | Knowledge and data collaborative driving unmanned aerial vehicle aerial photography target detection method |
| CN116665215A (en) * | 2023-05-25 | 2023-08-29 | 北京航星永志软件技术有限公司 | Image salient region extraction method, device, computer equipment and storage medium |
| CN117591901B (en) * | 2024-01-17 | 2024-05-03 | 合肥中科类脑智能技术有限公司 | Insulator breakage detection method and device, storage medium and electronic equipment |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7809822B2 (en) * | 2005-12-28 | 2010-10-05 | Telecom Italia S.P.A. | Method for the automatic generation of workflow models, in particular for interventions in a telecommunication network |
| CN105976368B (en) * | 2016-04-28 | 2018-12-11 | 华北电力大学（保定） | A kind of insulator localization method |
| US10140709B2 (en) * | 2017-02-27 | 2018-11-27 | International Business Machines Corporation | Automatic detection and semantic description of lesions using a convolutional neural network |
| US20180268292A1 (en) * | 2017-03-17 | 2018-09-20 | Nec Laboratories America, Inc. | Learning efficient object detection models with knowledge distillation |
| US10949660B2 (en) * | 2017-04-10 | 2021-03-16 | Pearson Education, Inc. | Electronic handwriting processor with convolutional neural networks |
| CN107145846B (en) * | 2017-04-26 | 2018-10-19 | 贵州电网有限责任公司输电运行检修分公司 | A kind of insulator recognition methods based on deep learning |
| CN107862562B (en) * | 2017-09-15 | 2022-01-07 | 唯品会(海南)电子商务有限公司 | Method and device for recommending commodities according to picture preference selection of user |
| CN108171213A (en) * | 2018-01-22 | 2018-06-15 | 北京邮电大学 | A kind of Relation extraction method for being applicable in picture and text knowledge mapping |
| CN108229578B (en) * | 2018-01-25 | 2019-10-11 | 海南大学 | Image data target identification method based on three layers of data, information and knowledge map framework |
| CN109272440B (en) * | 2018-08-14 | 2023-11-03 | 阿基米德（上海）传媒有限公司 | Thumbnail generation method and system combining text and image content |
| CN109241912B (en) * | 2018-09-08 | 2020-08-07 | 河南大学 | Target identification method based on brain-like cross-media intelligence and oriented to unmanned autonomous system |
| CN109344285B (en) * | 2018-09-11 | 2020-08-07 | 武汉魅瞳科技有限公司 | Monitoring-oriented video map construction and mining method and equipment |
| CN109378053B (en) * | 2018-11-30 | 2021-07-06 | 安徽影联云享医疗科技有限公司 | Knowledge graph construction method for medical image |
| CN109753969A (en) * | 2019-01-23 | 2019-05-14 | 河海大学 | A kind of insulator based on shape feature and image segmentation identifies and positions method |
-
2019
- 2019-05-31 CN CN201910468534.XA patent/CN110210387B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110210387A (en) | 2019-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110210387B (en) | Method, system and device for detecting insulator target based on knowledge graph | |
| Pape et al. | 3-D histogram-based segmentation and leaf detection for rosette plants | |
| CN108121991B (en) | Deep learning ship target detection method based on edge candidate region extraction | |
| Xue et al. | Design of digital camouflage by recursive overlapping of pattern templates | |
| CN108898145A (en) | A kind of image well-marked target detection method of combination deep learning | |
| CN105184763B (en) | Image processing method and device | |
| Yu et al. | Automatic interesting object extraction from images using complementary saliency maps | |
| KR20150107579A (en) | Image processing apparatus and image processing method | |
| Xu et al. | Weakly supervised deep semantic segmentation using CNN and ELM with semantic candidate regions | |
| Li et al. | DeepCotton: in-field cotton segmentation using deep fully convolutional network | |
| Havel et al. | Efficient tree construction for multiscale image representation and processing | |
| CN111027538A (en) | Container detection method based on instance segmentation model | |
| CN110910401A (en) | Semi-automatic image segmentation data annotation method, electronic device and storage medium | |
| Shete et al. | TasselGAN: An application of the generative adversarial model for creating field-based maize tassel data | |
| CN109213886B (en) | Image retrieval method and system based on image segmentation and fuzzy pattern recognition | |
| Han | Improved SLIC imagine segmentation algorithm based on K-means | |
| Wang et al. | Unsupervised segmentation of greenhouse plant images based on modified Latent Dirichlet Allocation | |
| CN111415364A (en) | Method, system and storage medium for converting image segmentation samples in computer vision | |
| Obeso et al. | Saliency-based selection of visual content for deep convolutional neural networks: application to architectural style classification | |
| CN108664968B (en) | Unsupervised text positioning method based on text selection model | |
| CN111368865B (en) | Remote sensing image oil storage tank detection method and device, readable storage medium and equipment | |
| CN111178196A (en) | Method, device and equipment for cell classification | |
| JP2009110526A (en) | Method and apparatus for analysing image | |
| CN113221773A (en) | Method for quickly constructing airplane classification data set based on remote sensing image | |
| Bell et al. | Reflections on connoisseurship and computer vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |