CN116468666A - Inspection image defect detection method special for operation and maintenance of power transmission line - Google Patents
Inspection image defect detection method special for operation and maintenance of power transmission line Download PDFInfo
- Publication number
- CN116468666A CN116468666A CN202310243599.0A CN202310243599A CN116468666A CN 116468666 A CN116468666 A CN 116468666A CN 202310243599 A CN202310243599 A CN 202310243599A CN 116468666 A CN116468666 A CN 116468666A
- Authority
- CN
- China
- Prior art keywords
- model
- convolution
- defect detection
- transmission line
- image defect
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 132
- 238000001514 detection method Methods 0.000 title claims abstract description 84
- 238000007689 inspection Methods 0.000 title claims abstract description 82
- 230000005540 biological transmission Effects 0.000 title claims abstract description 78
- 238000012423 maintenance Methods 0.000 title claims abstract description 44
- 238000012549 training Methods 0.000 claims abstract description 16
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 15
- 230000001537 neural effect Effects 0.000 claims abstract description 7
- 238000004364 calculation method Methods 0.000 claims abstract description 6
- 230000004927 fusion Effects 0.000 claims abstract description 6
- 238000000034 method Methods 0.000 claims description 37
- 238000000605 extraction Methods 0.000 claims description 18
- 230000000694 effects Effects 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 239000012212 insulator Substances 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 230000009471 action Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012804 iterative process Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 238000000354 decomposition reaction Methods 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0008—Industrial image inspection checking presence/absence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V10/225—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition based on a marking or identifier characterising the area
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/17—Terrestrial scenes taken from planes or by drones
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Remote Sensing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a inspection image defect detection method special for transmission line operation and maintenance, which comprises the steps of providing an inspection image defect detection model special for transmission line operation and maintenance based on a ResNet convolutional neural network architecture; the detection precision of the small target is improved by adopting a multi-scale prediction scheme and a pyramid association model; the depth decomposable model is adopted to replace a convolutional neural module, so that the inspection picture defect detection speed is improved; constructing a power transmission line power target defect data set, setting model parameters, and training a patrol image defect detection model; and identifying the defects of the power equipment in the power transmission line inspection picture by using the trained model. At the precision level, the context information fusion architecture is used for fusing the shallow feature map and the depth feature map, so that information loss is reduced, at the speed level, standard convolution calculation is decomposed into depth convolution and point-by-point convolution by convolution decomposition, the calculated amount of a model is reduced, and the problems of low precision and low speed in the inspection image recognition of the power transmission line are solved.
Description
Technical Field
The invention relates to the technical field of digital images, in particular to a patrol image defect detection method special for operation and maintenance of a power transmission line.
Background
The safe, stable and reliable operation of the power grid has an important effect on the stable development of national economy, and is a main means for ensuring the normal operation of the power grid for the inspection and maintenance of the power transmission line. Traditional electric power inspection is mainly completed manually, and an inspection staff periodically inspects and checks the power transmission line and the surrounding environment along the line, but the inspection mode has the following problems: (1) inspection safety is difficult to ensure. (2) inspection efficiency is low. And (3) the inspection fault finding rate is low. In 2013, the national grid company and the southern grid company sequentially develop a 'power transmission line machine patrol operation' plan, a helicopter, an unmanned aerial vehicle carrying a camera and an on-line monitoring means are adopted to carry out patrol inspection or state monitoring on the overhead power transmission line, and in the year 2020, the 'machine patrol mainly and man patrol mainly' collaborative patrol targets are basically realized.
The transmission line inspection image has typical large data characteristics of large volume, rapid growth and low value density, and the fault identification is mainly carried out by manual interpretation. Although this approach may also find a fault hazard, there are significant drawbacks: on one hand, the running condition of the power transmission channel is complex, the running condition and the channel condition of the equipment can be judged on site by operation and maintenance personnel, the obtained information is limited, and the given result has subjectivity, ambiguity and incompleteness, and is easy to have the problems of missed detection and false detection; on the other hand, the communication conditions around the long-distance high-voltage transmission line are poor, the transmission speed is slow, and the amount of transmitted data is limited, so that offline detection is mainly used.
In recent years, with the remarkable improvement of computing hardware technology and unmanned aerial vehicle technology and the rapid development of the fields of computer vision, artificial intelligence and the like, remarkable targets and characteristics in images can be extracted through deep mining of massive images, and practical application shows that the current technology has the advantages of high speed, high precision, high expansibility and the like, and the judging indexes in all aspects exceed that of artificial judgment. Based on the method, the related computer vision and artificial intelligence technology is to be applied, the actual application requirement of the inspection image of the power transmission line machine is correspondingly improved, a high-reliability target detection model for the operation and maintenance of the power transmission line is provided, the high-reliability target detection model is used for detecting main defects of the power transmission line such as appearance, operation environment and element abnormality of the power transmission line, and an alarm is provided in time, so that references are provided for equipment management, operation and maintenance, and the inspection efficiency and reliability of the power transmission line are improved.
Disclosure of Invention
This section is intended to outline some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the description summary and in the title of the application, to avoid obscuring the purpose of this section, the description summary and the title of the invention, which should not be used to limit the scope of the invention.
The present invention has been made in view of the above-described problems occurring in the prior art.
Therefore, the invention provides a inspection image defect detection method special for the operation and maintenance of the power transmission line, which can solve the problems of low precision and low speed in the inspection image recognition of the power transmission line in the prior art and reduce the calculated amount of a model under the condition of ensuring the precision.
In order to solve the technical problems, the invention provides a method for detecting the defects of the inspection image special for the operation and maintenance of the power transmission line, which comprises the following steps:
based on a ResNet convolutional neural network architecture, a patrol image defect detection model special for operation and maintenance of a power transmission line is provided;
adopting a multi-scale prediction scheme and a pyramid association model;
replacing a convolutional neural module by adopting a depth decomposable model;
constructing a power transmission line power target defect data set, setting model parameters, and training a patrol image defect detection model;
and identifying the defects of the power equipment in the power transmission line inspection picture by using the trained inspection image defect detection model.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the inspection image defect detection model special for the operation and maintenance of the power transmission line comprises the steps of adopting a residual network model as a characteristic extraction network of the inspection image defect detection model, obtaining target candidate area information with different sizes through an area suggestion strategy network, and obtaining accurate target position information by using a regression branch.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the method comprises the steps of extracting multi-scale features by adopting a feature pyramid structure, adding a memory channel through a residual error network to form a pyramid context correlation model, and fusing low-level detail information with high-level semantic information.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the method comprises the steps of improving the inspection picture defect detection speed by adopting a depth decomposable model to replace a convolutional neural module, separating a standard convolution kernel to form a depth convolution kernel and a point-by-point convolution, and respectively adding batch regularization and ReLu activation functions after the depth convolution and the point-by-point convolution.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the method comprises the steps of acquiring visible light pictures of a transmission line tower, an insulator, a ground wire and hardware fittings through an unmanned aerial vehicle carried image acquisition device, selecting pictures with defects of the power equipment, manually marking the position of the top left of the defect in a patrol picture and the width and height of the defect, classifying the types of the defect, sequentially writing marking information into text files according to the picture names, the defect types, the horizontal coordinates of the top left of the defect, the vertical coordinates of the top left of the defect, the width of the defect and the height of the defect, keeping the names of the text files consistent with the names of the pictures, randomly selecting 80% of pictures to form a training sample library for model learning and excavating the typical defect characteristics of the transmission line, and verifying the accuracy of the model in the training process by 20% of the pictures after selection.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the inspection image defect model comprises a multi-scale feature extraction module, a context correlation module and a relay module;
the multi-scale feature extraction module extracts image features by utilizing a depth resolvable network, reduces the calculated amount of a model under the condition of ensuring the feature extraction effect, and simultaneously primarily identifies the defect type and position based on a softmax activation function;
the context correlation module fuses the shallow features and the deep features, and improves the precision of the model for small-size defects;
the relay module is responsible for deconvoluting the depth features into feature graphs with larger dimensions, and the feature graphs keep the same dimensions as the shallow features, so that matrix addition operation can be performed.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the method for extracting multi-scale features by adopting the feature pyramid structure comprises the steps that images with different sizes in the images are subjected to feature processing mechanism of staggered layers of the pyramid model structure in order to accurately distinguish objects from backgrounds, a module at the bottom layer detects small-size targets of the images, pixel information of the small-size targets is mainly distributed at the bottom layer, and once high-level information is easy to lose or merge, the module at the high level extracts large-size targets;
the pyramid context correlation model comprises a model combining single feature map detection and a feature pyramid hierarchical structure, a model which fuses low-level information and high-level semantic information is provided, compared with the pyramid feature hierarchical structure, a feature pyramid context correlation model uses a deeper convolutional neural network to construct a feature pyramid, the model accumulates processed low-level features and processed high-level features to provide optimized position information, multiple downsampling and upsampling operations lead positioning information of a deep network to have errors, the downsampling information and the upsampling information are combined through the model, a deep feature pyramid is constructed, multi-layer feature information is fused, and different features are output;
when the high-dimensional features are sampled, combining the corresponding features of the previous layer, and before context feature fusion is carried out, changing the number of output units of the lower layer by 1X 1 convolution so as to keep the same size with the feature map obtained by up-sampling, and repeating the iterative process to generate a fine feature map;
when the fused feature map is processed by 3×3 convolution kernel to eliminate the aliasing effect of upsampling, the final feature map { C2, C3, C4, C5} layer is generated with the corresponding fused feature layers { P2, P3, P4, P5}, the corresponding layer space sizes are in communication, and all the layers share the classification layer and the regression layer in the pyramid feature extraction model.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the two operations of separating the standard convolution kernel into a depth convolution kernel and a point-by-point convolution include that the standard convolution kernel is used on all input channels, and the depth convolution adopts different convolution kernels for each input channel, wherein one convolution kernel corresponds to one input channel;
when the point-by-point convolution is a normal convolution, a convolution kernel of 1×1 is employed;
when the depth decomposable convolutional neural network is adopted, different input channels are subjected to convolution respectively by adopting a depth convolution check, then the above outputs are combined again by adopting point-by-point convolution, so that the calculated amount and model parameters can be reduced;
the feature map F has a size (D F ,D F M) using a labelThe quasi-convolution K is (D K ,D K M, N) output is (D) G ,D G ,N)；
The convolution calculation formula of the standard convolution is:
wherein M is the number of channels input, and N is the number of channels output;
the corresponding calculated amount is that,
D K ·D K ·M·N·D F ·D F
the standard convolution (D K ,D K M, N) is split into depth convolution and point-by-point convolution,
the depth convolution is responsible for the filtering action, and is of size (D K ,D K 1, M) with an output characteristic of (D G ,D G ,M)；
The point-wise convolution is responsible for converting the channel, with dimensions (1, m, n), and the final output is (D G ,D G ,N)；
The convolution formula of the depth convolution is that,
in the method, in the process of the invention,is a deep convolution with a convolution kernel (D K ,D K 1, M), wherein m th Application of the convolution kernel to the mth of F th On the individual channels, produce->Upper mth th The output of each channel corresponds to the calculated amount:
D K ·D K ·M·D F ·D F +M·N·D F ·D F
the calculated amount of the depth resolvable convolutional neural network is as follows:
as a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the setting of the model parameters comprises the steps of taking ResNet101 as a backbone network of a patrol image defect detection model, and initializing the backbone network by adopting parameters pre-trained on a VOC data set.
As a preferable scheme of the inspection image defect detection method special for the operation and maintenance of the power transmission line, the invention comprises the following steps: the method for identifying the defects of the power equipment in the power transmission line inspection picture by using the trained inspection image defect detection model comprises the steps of testing the inspection image defect detection model on a data set which is acquired by an unmanned aerial vehicle and contains a pole tower, an insulator, a ground wire and hardware fittings, and evaluating the model by adopting average precision AP.
The invention has the beneficial effects that: the invention provides a inspection image defect detection model special for detecting a defect target of a power transmission line, and the model is optimized from two aspects of detection precision and detection speed. In the precision level, the model uses a context information fusion architecture to fuse the shallow feature map and the depth feature map, so that information loss is reduced. In the speed level, the model uses a convolution decomposition technology to decompose standard convolution calculation into depth convolution and point-by-point convolution, and the calculated amount of the model is reduced under the condition of ensuring the accuracy. The model can effectively solve the problems of low precision and low speed in the inspection image recognition of the transmission line.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
fig. 1 is a schematic flow chart of a patrol image dedicated to operation and maintenance of a power transmission line according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a inspection image defect detection model dedicated to operation and maintenance of a power transmission line according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a residual network learning module of a method for detecting a defect of an inspection image dedicated to operation and maintenance of a power transmission line according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of feature extraction of a feature pyramid context correlation model of a method for detecting a defect of a patrol image dedicated to operation and maintenance of a power transmission line according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a standard convolution kernel depth resolvable convolution network of a method for detecting a fault of a patrol image, which is specially used for operation and maintenance of a transmission line according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a depth decomposition convolution module of a inspection image defect detection method dedicated to operation and maintenance of a power transmission line according to an embodiment of the present invention.
Detailed Description
So that the manner in which the above recited objects, features and advantages of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present invention is not limited to the specific embodiments disclosed below.
Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
While the embodiments of the present invention have been illustrated and described in detail in the drawings, the cross-sectional view of the device structure is not to scale in the general sense for ease of illustration, and the drawings are merely exemplary and should not be construed as limiting the scope of the invention. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.
Also in the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper, lower, inner and outer", etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
The terms "mounted, connected, and coupled" should be construed broadly in this disclosure unless otherwise specifically indicated and defined, such as: can be fixed connection, detachable connection or integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or may be indirectly connected through an intermediate medium, or may be a communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Example 1
Referring to fig. 1-6, a first embodiment of the present invention provides a method for detecting a defect of a patrol image dedicated to operation and maintenance of a power transmission line, including:
s1: based on a ResNet convolutional neural network architecture, a patrol image defect detection model special for operation and maintenance of a power transmission line is provided;
furthermore, the inspection image defect detection model special for the operation and maintenance of the power transmission line comprises the steps of adopting a residual network model as a characteristic extraction network of the inspection image defect detection model, constructing an improved target detection model, wherein the characteristic extraction network is based on the residual network model, the remarkable characteristic is that a memory channel is added, target candidate region information with different sizes is obtained through a region suggestion strategy network, and accurate target position information is obtained through a regression branch.
S2: the detection precision of the small target is improved by adopting a multi-scale prediction scheme and a pyramid association model;
furthermore, the method for improving the small target detection precision by adopting the multi-scale prediction scheme and the associated architecture comprises the steps of extracting multi-scale features by adopting a feature pyramid structure;
and adding a memory channel to form a pyramid context correlation model, and fusing the detail information of the lower layer with the semantic information of the upper layer.
S3: the depth decomposable model is adopted to replace a convolutional neural module, so that the inspection picture defect detection speed is improved;
furthermore, the method for improving the small target detection precision by adopting the multi-scale prediction scheme and the associated architecture comprises the steps of extracting multi-scale features by adopting a feature pyramid structure;
and adding a memory channel to form a pyramid context correlation model, and fusing the detail information of the lower layer with the semantic information of the upper layer.
It should be noted that, the method adopts a depth decomposable model to replace a convolutional neural module, improves the inspection picture defect detection speed, separates a standard convolution kernel to form two operations of a depth convolution kernel and a point-by-point convolution;
batch regularization and ReLu activation functions are added after the depth convolution and point-by-point convolution, respectively.
S4: constructing a power transmission line power target defect data set, setting model parameters, and training a patrol image defect detection model;
further, the method comprises the steps of constructing a power transmission line power target defect data set, setting model parameters, training and inspecting an image defect detection model, wherein an unmanned aerial vehicle carries image acquisition equipment to acquire visible light pictures of a power transmission line pole tower, an insulator, a ground lead and hardware fittings, selecting pictures with defects of the four types of power equipment, manually marking the position and the width and height of the top left of the defect in the inspection picture, classifying the defect types of the defect, sequentially writing marking information into a text file according to the picture name, the defect type, the abscissa of the top left of the defect, the ordinate of the top left of the defect, the width of the defect and the height of the defect, keeping consistency between the text file name and the picture name, randomly selecting 80% of pictures to form a training sample library for model learning and excavating characteristics of typical defects of the power transmission line, leaving 20% of pictures to form a verification set for accuracy in a model verification process, and if the pictures with less than 80% of the pattern are selected to form the training sample library, missing typical defect characteristics provided for the power transmission line, causing training sample library to have model training missing, and if the pictures with the picture composition of more than 80% are selected to form a training sample library, the verification sample is lost, and the verification condition is lost is reduced when the verification condition of the picture is lost.
Further, the inspection image defect model comprises a model consisting of three parts, a multi-scale feature extraction module, a context correlation module and a relay module; the multi-scale feature extraction module extracts image features by utilizing a depth resolvable network, reduces the calculated amount of a model under the condition of ensuring the feature extraction effect, and simultaneously primarily identifies the defect type and position based on a softmax activation function; the correlation module fuses the shallow features and the deep features, so that the precision of the model for small-size defects is improved; the relay module is responsible for deconvoluting the depth features into feature maps with larger dimensions, so that the depth features and the shallow features keep the same dimensions, and matrix addition operation can be performed.
It should be noted that, the extracting multi-scale features by using the feature pyramid structure includes that, for different sizes of images in the image, in order to accurately distinguish objects from backgrounds, the pyramid model structure is a staggered feature processing mechanism, the module at the bottom layer can detect the small-size target of the image, because the pixel information of the small-size object is mainly distributed at the bottom layer, and once the high-level information is easily lost or combined, the module at the high layer can extract the large-size target, because the edge information of the large size is more easily detected after being multi-layered; the pyramid context correlation model comprises a model combining single feature map detection and a feature pyramid hierarchical structure, a model combining low-level information and high-level semantic information is provided, compared with the pyramid feature hierarchical structure, a feature pyramid context correlation model uses a deeper convolutional neural network to construct a feature pyramid, the extracted features are more robust, the model provides more accurate position information by accumulating the processed low-level features and the processed high-level features, the positioning information of a deep network has errors through multiple downsampling and upsampling operations, a deeper feature pyramid is constructed by combining the downsampling information and the upsampling information through the model, and multiple layers of feature information are fused and output at different features; the high-dimensional features are up-sampled by 2 times, the high-dimensional features are combined with the corresponding features of the previous layer, the number of output units of the lower layer is required to be changed by 1X 1 convolution before context feature fusion is carried out, so that the same size as the feature map obtained by up-sampling is kept, and the iterative process is repeated until the finest feature map is generated; the fused feature map is processed by using a 3×3 convolution kernel to eliminate the aliasing effect of the upsampling, so as to generate a final feature map { C2, C3, C4, C5} layer, the corresponding fused feature layers of { P2, P3, P4, P5} layer, the corresponding layer space sizes are communicated, all layers share the classification layer and the regression layer in the pyramid feature extraction model, the number of channels in the fixed feature map is 256, and all the additional convolution layers have 256 channel outputs.
Further, the separating the standard convolution kernel to form a depth convolution kernel and a point-by-point convolution operation includes that the standard convolution kernel is used on all input channels, and the depth convolution adopts different convolution kernels for each input channel, wherein one convolution kernel corresponds to one input channel;
the point-by-point convolution is a common convolution, and adopts a convolution kernel of 1 multiplied by 1;
the depth decomposable convolutional neural network adopts the depth convolutional to check different input channels to respectively convolve, and then adopts the point-by-point convolution to combine the above outputs, so that the calculated amount and model parameters can be reduced;
the feature map F has a size (D F ,D F M) using a standard convolution K as (D K ,D K M, N) output is (D) G ,D G ,N)；
The convolution calculation formula of the standard convolution is:
wherein M is the number of channels input, and N is the number of channels output;
the corresponding calculated amount is that,
D K ·D K ·M·N·D F ·D F
the standard convolution (D K ,D K M, N) is split into depth convolution and point-by-point convolution,
the depth convolution is responsible for the filtering action, and is of size (D K ,D K 1, M) with an output characteristic of (D G ,D G ,M)；
The point-wise convolution is responsible for converting the channel, with dimensions (1, m, n), and the final output is (D G ,D G ,N)；
The convolution formula of the depth convolution is that,
in the method, in the process of the invention,is a deep convolution with a convolution kernel (D K ,D K 1, M), wherein m th Application of the convolution kernel to the mth of F th On the individual channels, produce->Upper mth th The output of each channel corresponds to the calculated amount:
D K ·D K ·M·D F ·D F +M·N·D F ·D F
the calculated amount of the depth resolvable convolutional neural network is as follows:
it should be noted that, the setting of the model parameters includes that ResNet101 is used as a backbone network of a inspection image defect detection model, and the backbone network is initialized by using parameters pre-trained on the VOC data set;
training is carried out by adopting a batch gradient descent method, the batch size of batch training is initially set to be 32, the initial learning rate is set to be 0.001, the initial momentum is 0.9, the initial weight delay is 0.0005, and the model is trained until the loss function reaches convergence.
S5: and identifying the defects of the power equipment in the power transmission line inspection picture by using the trained model.
It should be noted that, the identification of the defects of the power equipment in the inspection picture of the power transmission line by using the trained inspection image defect detection model comprises the step of testing the model on a data set which is acquired by an unmanned aerial vehicle and contains a pole tower, an insulator, a ground wire and a hardware fitting, wherein the model is evaluated by adopting an average precision AP.
Example 2
Referring to fig. 1-6, for one embodiment of the present invention, a method for detecting a defect of an inspection image dedicated to operation and maintenance of a power transmission line is provided, and in order to verify the beneficial effects of the present invention, scientific demonstration is performed through experiments.
The embodiment provides an end-to-end power transmission line defect identification method, which comprises the steps of firstly constructing a patrol image defect detection model on a server platform, uploading the patrol image obtained by patrol shooting of a power transmission line to the server platform, and directly reading the patrol image by using the model to process and output analysis results in batches. For the picture with the defects, the algorithm outputs information such as the picture name, the type of each defect in the picture, the position of the defect in the picture and the like, and the information can be used as overhaul reference opinion or data for subsequent further processing.
The backbone network used in this example is ResNet101, whose network configuration parameters are shown in Table 1.
TABLE 1ResNet101 network architecture
To analyze the effectiveness of this example, this example was tested in batch on a test sample set and compared to typical target detection models of fast RCNN, YOLO v3, SSD, etc., and the results are shown in Table 2.
Table 2 comparison of model test results
The mAP values reflect the overall performance of the model for the detection accuracy of various defects, and as can be seen from Table 2, mAP in the Faster RCNN model based on the regional recommendation strategy is higher than that in the YOLO model and SSD model in the model of the integrated convolutional neural network. Meanwhile, table 2 also shows that the mAP of the model is superior to the single-order method and the double-order method which are proposed at present, and the core is that the context information fusion architecture of the model ensures that the model can extract defect characteristic information with different sizes.
Meanwhile, the model adopts the thought similar to a single-order method, a target candidate domain is not generated in the characteristic extraction process, the image defect characteristic is directly extracted by using a convolutional neural network, and the calculation amount of the model is reduced by adopting a convolutional decomposition technology, so that the detection speed of the model is also due to the existing target detection model.
It should be understood that parts of the specification not specifically set forth herein are all prior art.
It should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the technical solution of the present invention, which is intended to be covered by the scope of the claims of the present invention.
Claims (10)
1. A inspection image defect detection method special for operation and maintenance of a power transmission line is characterized by comprising the following steps of: comprising the steps of (a) a step of,
based on a ResNet convolutional neural network architecture, a patrol image defect detection model special for operation and maintenance of a power transmission line is provided;
adopting a multi-scale prediction scheme and a pyramid context correlation model;
replacing a convolutional neural module by adopting a depth decomposable model;
constructing a power transmission line power target defect data set, setting model parameters, and training a patrol image defect detection model;
and identifying the defects of the power equipment in the power transmission line inspection picture by using the trained inspection image defect detection model.
2. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 1, wherein the inspection image defect detection method is characterized by comprising the following steps of: the inspection image defect detection model special for the operation and maintenance of the power transmission line comprises the steps of adopting a residual network model as a characteristic extraction network of the inspection image defect detection model, obtaining target candidate area information with different sizes through an area suggestion strategy network, and obtaining accurate target position information by using a regression branch.
3. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 2, wherein the inspection image defect detection method is characterized by comprising the following steps of: the method comprises the steps of extracting multi-scale features by adopting a feature pyramid structure, adding a memory channel through a residual error network to form a pyramid context correlation model, and fusing low-level detail information with high-level semantic information.
4. A method for detecting defects of inspection images special for operation and maintenance of a power transmission line as claimed in claim 3, wherein the method comprises the following steps: the method comprises the steps of improving the inspection picture defect detection speed by adopting a depth decomposable model to replace a convolutional neural module, separating a standard convolution kernel to form a depth convolution kernel and a point-by-point convolution, and respectively adding batch regularization and ReLu activation functions after the depth convolution and the point-by-point convolution.
5. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 4, wherein the inspection image defect detection method is characterized by comprising the following steps of: the method comprises the steps of acquiring visible light pictures of a transmission line tower, an insulator, a ground wire and hardware fittings through an unmanned aerial vehicle carried image acquisition device, selecting pictures with defects of the power equipment, manually marking the position of the top left of the defect in a patrol picture and the width and height of the defect, classifying the types of the defect, sequentially writing marking information into text files according to the picture names, the defect types, the horizontal coordinates of the top left of the defect, the vertical coordinates of the top left of the defect, the width of the defect and the height of the defect, keeping the names of the text files consistent with the names of the pictures, randomly selecting 80% of pictures to form a training sample library for model learning and excavating the typical defect characteristics of the transmission line, and verifying the accuracy of the model in the training process by 20% of the pictures after selection.
6. The inspection image defect detection method special for operation and maintenance of the power transmission line according to claim 5, wherein the inspection image defect detection method is characterized by comprising the following steps of: the inspection image defect model comprises a multi-scale feature extraction module, a context correlation module and a relay module;
the multi-scale feature extraction module extracts image features by utilizing a depth resolvable network, reduces the calculated amount of a model under the condition of ensuring the feature extraction effect, and simultaneously primarily identifies the defect type and position based on a softmax activation function;
the context correlation module fuses the shallow features and the deep features, and improves the precision of the model for small-size defects;
the relay module is responsible for deconvoluting the depth features into feature graphs with larger dimensions, and the feature graphs keep the same dimensions as the shallow features, so that matrix addition operation can be performed.
7. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 6, wherein the inspection image defect detection method is characterized by comprising the following steps: the method for extracting multi-scale features by adopting the feature pyramid structure comprises the steps that images with different sizes in the images are subjected to feature processing mechanism of staggered layers of the pyramid model structure in order to accurately distinguish objects from backgrounds, a module at the bottom layer detects small-size targets of the images, pixel information of the small-size targets is mainly distributed at the bottom layer, and once high-level information is easy to lose or merge, the module at the high level extracts large-size targets;
the pyramid context correlation model comprises a model combining single feature map detection and a feature pyramid hierarchical structure, a model which fuses low-level information and high-level semantic information is provided, compared with the pyramid feature hierarchical structure, a feature pyramid context correlation model uses a deeper convolutional neural network to construct a feature pyramid, the model accumulates processed low-level features and processed high-level features to provide optimized position information, multiple downsampling and upsampling operations lead positioning information of a deep network to have errors, the downsampling information and the upsampling information are combined through the model, a deep feature pyramid is constructed, multi-layer feature information is fused, and different features are output;
when the high-dimensional features are sampled, combining the corresponding features of the previous layer, and before context feature fusion is carried out, changing the number of output units of the lower layer by 1X 1 convolution so as to keep the same size with the feature map obtained by up-sampling, and repeating the iterative process to generate a fine feature map;
when the fused feature map is processed by 3×3 convolution kernel to eliminate the aliasing effect of upsampling, the final feature map { C2, C3, C4, C5} layer is generated with the corresponding fused feature layers { P2, P3, P4, P5}, the corresponding layer space sizes are in communication, and all the layers share the classification layer and the regression layer in the pyramid feature extraction model.
8. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 7, wherein the inspection image defect detection method is characterized by comprising the following steps of: the two operations of separating the standard convolution kernel into a depth convolution kernel and a point-by-point convolution include that the standard convolution kernel is used on all input channels, and the depth convolution adopts different convolution kernels for each input channel, wherein one convolution kernel corresponds to one input channel;
when the point-by-point convolution is a normal convolution, a convolution kernel of 1×1 is employed;
when the depth decomposable convolutional neural network is adopted, different input channels are subjected to convolution respectively by adopting a depth convolution check, then the above outputs are combined again by adopting point-by-point convolution, so that the calculated amount and model parameters can be reduced;
the feature map F has a size (D F ,D F M) using a standard convolution K as (D K ,D K M, N) output is (D) G ,D G ,N)；
The convolution calculation formula of the standard convolution is:
wherein M is the number of channels input, and N is the number of channels output;
the corresponding calculated amount is that,
D K ·D K ·M·N·D F ·D F
the standard convolution (D K ,D K M, N) is split into depth convolution and point-by-point convolution,
the depth convolution is responsible for the filtering action, and is of size (D K ,D K 1, M) with an output characteristic of (D G ,D G ,M)；
The point-wise convolution is responsible for converting the channel, with dimensions (1, m, n), and the final output is (D G ,D G ,N)；
The convolution formula of the depth convolution is that,
in the method, in the process of the invention,is a deep convolution with a convolution kernel (D K ,D K 1, M), wherein m th Application of the convolution kernel to the mth of F th On the individual channels, produce->Upper mth th The output of each channel corresponds to the calculated amount:
D K ·D K ·M·D F ·D F +M·N·D F ·D F
the calculated amount of the depth resolvable convolutional neural network is as follows:
9. the inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 8, wherein the inspection image defect detection method is characterized by comprising the following steps: the setting of the model parameters comprises the steps of taking ResNet101 as a backbone network of a patrol image defect detection model, and initializing the backbone network by adopting parameters pre-trained on a VOC data set.
10. The inspection image defect detection method special for operation and maintenance of the power transmission line as claimed in claim 1, wherein the inspection image defect detection method is characterized by comprising the following steps of: the method for identifying the defects of the power equipment in the power transmission line inspection picture by using the trained inspection image defect detection model comprises the steps of testing the inspection image defect detection model on a data set which is acquired by an unmanned aerial vehicle and contains a pole tower, an insulator, a ground wire and hardware fittings, and evaluating the model by adopting average precision AP.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310243599.0A CN116468666A (en) | 2023-03-14 | 2023-03-14 | Inspection image defect detection method special for operation and maintenance of power transmission line |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310243599.0A CN116468666A (en) | 2023-03-14 | 2023-03-14 | Inspection image defect detection method special for operation and maintenance of power transmission line |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116468666A true CN116468666A (en) | 2023-07-21 |
Family
ID=87174243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310243599.0A Pending CN116468666A (en) | 2023-03-14 | 2023-03-14 | Inspection image defect detection method special for operation and maintenance of power transmission line |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116468666A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274748A (en) * | 2023-11-16 | 2023-12-22 | 国网四川省电力公司电力科学研究院 | Lifelong learning power model training and detecting method based on outlier rejection |
| CN117333491A (en) * | 2023-12-01 | 2024-01-02 | 北京航空航天大学杭州创新研究院 | Steel surface defect detection method and system |
| CN117690063A (en) * | 2024-02-04 | 2024-03-12 | 广东电网有限责任公司广州供电局 | Cable line detection method, device, electronic equipment and computer readable medium |
-
2023
- 2023-03-14 CN CN202310243599.0A patent/CN116468666A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117274748A (en) * | 2023-11-16 | 2023-12-22 | 国网四川省电力公司电力科学研究院 | Lifelong learning power model training and detecting method based on outlier rejection |
| CN117274748B (en) * | 2023-11-16 | 2024-02-06 | 国网四川省电力公司电力科学研究院 | Lifelong learning power model training and detecting method based on outlier rejection |
| CN117333491A (en) * | 2023-12-01 | 2024-01-02 | 北京航空航天大学杭州创新研究院 | Steel surface defect detection method and system |
| CN117333491B (en) * | 2023-12-01 | 2024-03-15 | 北京航空航天大学杭州创新研究院 | Steel surface defect detection method and system |
| CN117690063A (en) * | 2024-02-04 | 2024-03-12 | 广东电网有限责任公司广州供电局 | Cable line detection method, device, electronic equipment and computer readable medium |
| CN117690063B (en) * | 2024-02-04 | 2024-04-12 | 广东电网有限责任公司广州供电局 | Cable line detection method, device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116468666A (en) | Inspection image defect detection method special for operation and maintenance of power transmission line | |
| CN111563557B (en) | Method for detecting target in power cable tunnel | |
| CN111415329B (en) | Workpiece surface defect detection method based on deep learning | |
| CN109117876A (en) | A kind of dense small target deteection model building method, model and detection method | |
| CN108734143A (en) | A kind of transmission line of electricity online test method based on binocular vision of crusing robot | |
| CN111797890A (en) | Method and system for detecting defects of power transmission line equipment | |
| CN110533631A (en) | SAR image change detection based on the twin network of pyramid pondization | |
| CN108648169A (en) | The method and device of high voltage power transmission tower defects of insulator automatic identification | |
| CN112132205B (en) | Remote sensing image classification method based on convolutional neural network | |
| CN109801265A (en) | A kind of real-time transmission facility foreign matter detection system based on convolutional neural networks | |
| US20230360390A1 (en) | Transmission line defect identification method based on saliency map and semantic-embedded feature pyramid | |
| CN113205039B (en) | Power equipment fault image recognition disaster investigation system and method based on multiple DCNN networks | |
| CN111639530B (en) | Method and system for detecting and identifying power transmission tower and insulator of power transmission line | |
| CN108711148A (en) | A kind of wheel tyre defect intelligent detecting method based on deep learning | |
| CN112801230A (en) | Intelligent acceptance method for unmanned aerial vehicle of power distribution line | |
| CN112070135A (en) | Power equipment image detection method and device, power equipment and storage medium | |
| CN114359756A (en) | Rapid and intelligent detection method for house damaged by remote sensing image of post-earthquake unmanned aerial vehicle | |
| CN116503318A (en) | Aerial insulator multi-defect detection method, system and equipment integrating CAT-BiFPN and attention mechanism | |
| CN116258980A (en) | Unmanned aerial vehicle distributed photovoltaic power station inspection method based on vision | |
| CN112766409A (en) | Feature fusion method for remote sensing image target detection | |
| CN116883801A (en) | YOLOv8 target detection method based on attention mechanism and multi-scale feature fusion | |
| CN116012653A (en) | Method and system for classifying hyperspectral images of attention residual unit neural network | |
| CN115546664A (en) | Cascaded network-based insulator self-explosion detection method and system | |
| Liu et al. | An improved faster R-CNN for UAV-based catenary support device inspection | |
| CN115761300A (en) | Method, system and detection device for dividing safety exit abnormity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |