CN112183453A - Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system - Google Patents

Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system Download PDF

Info

Publication number
CN112183453A
CN112183453A CN202011105125.2A CN202011105125A CN112183453A CN 112183453 A CN112183453 A CN 112183453A CN 202011105125 A CN202011105125 A CN 202011105125A CN 112183453 A CN112183453 A CN 112183453A
Authority
CN
China
Prior art keywords
attention
self
cover plate
layer
water injection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011105125.2A
Other languages
Chinese (zh)
Other versions
CN112183453B (en
Inventor
战岭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Kejia General Mechanical and Electrical Co Ltd
Original Assignee
Harbin Kejia General Mechanical and Electrical Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Kejia General Mechanical and Electrical Co Ltd filed Critical Harbin Kejia General Mechanical and Electrical Co Ltd
Priority to CN202011105125.2A priority Critical patent/CN112183453B/en
Publication of CN112183453A publication Critical patent/CN112183453A/en
Application granted granted Critical
Publication of CN112183453B publication Critical patent/CN112183453B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

A method and a system for detecting faults that a water injection opening cover plate is not locked in place based on deep learning belong to the technical field of image detection. The problems of low detection accuracy and low detection efficiency in the conventional manual image checking mode are solved. The method comprises the steps of obtaining a text position of a water filling port and a position of the water filling port by using a trained target detection model, further determining the accurate position of a cover plate of the water filling port, and carrying out binarization on an image to obtain a binary image of the cover plate of the water filling port; then, according to the opening and closing directions of cover plates of water injection ports of different vehicle types, a gap shadow image between the cover plate of the water injection port and a vehicle body is captured on the binary image, and the gap shadow area between the cover plate of the water injection port and the vehicle body is obtained; and finally, judging whether the gap shadow area exceeds a threshold value according to gap shadow area threshold values corresponding to different vehicle types, so as to determine whether the accurate position of the water filling port cover plate is a fault area. The method is mainly used for fault detection when the cover plate of the water injection port is not locked in place.

Description

Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system
Technical Field
The invention belongs to the technical field of image detection, and particularly relates to a method and a system for detecting a fault that a water injection port cover plate is not locked in place.
Background
The water filling port cover plate can ensure the air tightness of the train body of the motor train unit in high-speed running, and can protect the water filling port and nearby components from water leakage, component loss and other faults caused by the influence of the difference between the internal air pressure and the external air pressure, so that the water filling port cover plate is very important for detecting the unlocking.
In the existing detection method, the fault detection is usually carried out by manually checking images. The situations of missed detection, wrong detection and the like are caused by the fact that vehicle detection personnel are easy to fatigue in the working process. Therefore, the automatic alarm device has important significance in timely and automatically alarming the fault that the water filling port cover plate of the motor train unit is not locked in place.
The detection method using the existing detection operator cannot well detect the fault that the water injection port cover plate is not locked in place, and the detection accuracy often cannot meet the actual requirement. With the development of deep learning technology, although the existing neural network can be applied to fault detection when a water filling port cover plate is not locked in place, the following problems still exist:
when the existing neural network model with a relatively simple structure is used for locking and detecting the water injection port cover plate, the problems of low detection accuracy and high false detection rate and missing detection rate exist; when the neural network model with a complex structure is used for locking and detecting the water injection port cover plate, long training time is needed, the number of model parameters is very large, the operation efficiency of the model is seriously reduced, and the detection time is prolonged.
Disclosure of Invention
The invention aims to solve the problems of low detection accuracy and low detection efficiency of the conventional manual image checking mode.
The method for detecting the fault that the water injection port cover plate is not locked in place based on deep learning comprises the following steps:
s1, acquiring an interested area image including a water injection port part;
s2, acquiring the text position of a water filling port and the position of the water filling port by using the trained target detection model; the target detection model is a target detection neural network model based on a multi-head region self-attention neural network;
s3, determining the accurate position of the cover plate of the water filling port according to the text position of the water filling port and the position of the water filling port;
s4, determining the self-adaptive binarization threshold value of the image corresponding to the accurate position of the water filling port cover plate obtained in the S3 by adopting an OTSU algorithm, carrying out binarization, setting the pixel value smaller than the self-adaptive binarization threshold value to be 0, and setting the pixel value larger than or equal to the self-adaptive binarization threshold value to be 255, and obtaining a water filling port cover plate binary image;
s5, according to the opening and closing directions of cover plates of water injection ports of different vehicle types, a gap shadow image between the cover plates of the water injection ports and a vehicle body is captured on the binary image of the cover plates of the water injection ports;
s6, calculating the number of pixels with the pixel value of 0 in the gap shadow image, and acquiring the gap shadow area between the water filling port cover plate and the vehicle body;
s7, judging whether the gap shadow area obtained in S6 exceeds a threshold value or not according to the gap shadow area threshold values corresponding to different vehicle types; and if so, recording the accurate position of the water filling port cover plate obtained in the step S3 as fault information.
Further, the processing procedure of the multi-head region self-attention neural network is as follows:
(1) selecting a sub-region pixel matrix with a window size of m x m pixels from the input image;
(2) copying 3 parts of the matrix obtained in the step (1), and respectively naming the matrix as a query matrix Q, a key matrix K and a value matrix V;
(3) and (3) respectively carrying out linear transformation on the Q, K, V matrixes obtained in the step (2) to obtain matrixes subjected to linear transformation
Figure BDA0002726702760000021
(4) And (3) calculating
Figure BDA0002726702760000022
Correlation matrix between matrices
Figure BDA0002726702760000023
Figure BDA0002726702760000024
The self-attention matrix a is calculated by the softmax function:
Figure BDA0002726702760000025
wherein:
Figure BDA0002726702760000026
and
Figure BDA0002726702760000027
self-attention maps in the vertical direction and the horizontal direction of the image respectively; concat is matrix horizontal splicing; wCThe weight parameters are related to linear transformation, and the size is 2m x m; b isCThe bias parameters are related to linear transformation, and the size is m x m; the size of the self-attention matrix A is m x m;
(5) and using the sub-region matrix obtained from the attention matrix A obtained in the step (4) to the sub-region matrix obtained in the step (3)
Figure BDA0002726702760000028
Weighting to obtain a sub-region characteristic matrix F;
(6) repeating the processes of the steps (4) and (5) for H times, wherein each time of repeating is called a self-attention head;
(7) splicing H sub-area characteristic matrixes F corresponding to H self-attention heads to obtain H sub-area characteristic matrixes F
Figure BDA0002726702760000029
The size is m x mH;
(8) obtained by (7) above
Figure BDA00027267027600000210
Linear transformation is carried out to obtain the final characteristics of the subareas
Figure BDA00027267027600000211
(9) And (3) moving the subareas in the step (1) to the right line by step length S, and repeating the processes (1) to (8) until all the features of the input image are extracted, so as to obtain a feature map with the same size as the input image.
Further, the process of step (3) for performing linear transformation on the Q, K, V three matrices obtained in step (2) is as follows:
Figure BDA00027267027600000212
Figure BDA00027267027600000213
Figure BDA00027267027600000214
wherein: wQ、WK、WVLinear transformation weight parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation weight parameters are m x m; b isQ、BK、BVLinear transformation bias parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation bias parameters are m x m;
Figure BDA0002726702760000031
Figure BDA0002726702760000032
q, K, V matrixes after linear transformation are respectively, and the sizes of the matrixes are m x m.
Further, the model structure of the target detection neural network model based on the multi-head region self-attention neural network comprises a region 1 self-attention layer to a region 7 candidate layer and an output layer;
zone 1 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 2 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 3 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 4 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 5 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 6 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
region 7 candidate layer: 2, connecting a plurality of multi-head area self-attention neural networks in parallel, wherein the multi-head area self-attention neural networks are respectively a position prediction layer and a probability prediction layer;
an output layer: and outputting the predicted position information, the corresponding prediction type and the prediction confidence of the target in the image.
Further, the number H of the multi-headed regions in the series structure of the multi-headed regions from the attention neural network in the region 1 from the attention layer to the region 6 candidate layer from the attention head is 2, 4, 8, 16, 32, 64, respectively;
in the multi-head region self-attention neural network parallel structure in the 7 th region candidate layer, the number H of the multi-head region self-attention heads in the position prediction layer is 4; the number of multi-headed regions in the probability prediction layer from the attention head H is 2.
Further, the size of the sub-region in the multi-head region self-attention neural network from the attention layer 1 to the candidate layer 6 is 3x3 pixels, and the step length S is 3 pixels;
in the multi-head area self-attention neural network in the 7 th area candidate layer, the size of the position prediction layer sub-area is 1x1 pixels, and the step length S is 1 pixel; the size of the probability prediction layer sub-region is 1x1 pixels, and the step S is 1 pixel.
Further, the window size of the region 1 from the attention layer to the largest pool layer among the region 6 candidate layers is 2x2 with a step size of 2.
Further, the labeling process for the data set of the trained target detection model comprises the steps of:
according to different motor train units of which the water injection ports belong to, labeling the images by using a labelImg data labeling tool:
the label name of the text of the 'water filling port' is text, and the label category is 1;
the label names of the water injection ports of different vehicle types are cover + vehicle types, the corresponding label types are 2, 3 and 4, and the like.
Further, before the labeling of the data set, a data amplification operation is performed on the acquired image data set, so as to obtain a data set for labeling.
The water injection port cover plate non-locking in-place fault detection system based on deep learning is used for executing a water injection port cover plate non-locking in-place fault detection method based on deep learning.
Has the advantages that:
1. the deep learning method is used for replacing manual work to carry out automatic fault detection on the motor train unit, the influence of subjective factors of detection personnel and the limitation of working time are avoided, and the detection quality (high detection accuracy, low false detection rate and low omission factor) and the detection efficiency of faults of the water filling port of the motor train unit can be effectively improved.
2. The invention provides a Multi-head Region Self-attention (Multi-head Region Self-attention) neural network for replacing a traditional convolution neural network to construct a target detection model. The proposed multi-headed region self-attention neural network has the following advantages: (1) by dividing the image into a plurality of sub-regions for independent operation, the number of self-attention parameters can be reduced, and the model training and reasoning speed is improved; (2) the self-attention matrix is calculated by combining the self-attention diagrams in the vertical direction and the horizontal direction, so that the feature extraction capability is stronger; (3) by replacing convolution operation with self-attention operation, key features in image sub-regions can be extracted more effectively, and background and noise are better suppressed; (4) through the parallel connection of a plurality of self-attention heads, parameters of all the heads are independent, and the features of different semantic subspaces in the region can be extracted.
The method has the advantages of high detection accuracy and rapid detection, and can effectively solve the problem of low detection accuracy or long detection time when the existing neural network model is used for detecting the locking of the water injection port cover plate.
Drawings
FIG. 1 is a schematic view of a fault detection process for a water injection port cover plate not locked in place;
FIG. 2 is a schematic diagram of a multi-head region self-attention neural network structure;
FIG. 3 is a schematic diagram of a self-attention matrix calculation;
FIG. 4 is a schematic diagram of a target detection neural network model structure.
Detailed Description
The first embodiment is as follows:
the method for detecting the fault that the water injection port cover plate is not locked in place based on deep learning comprises the following steps:
s1, acquiring an interested area image including a water injection port part;
s2, acquiring the text position of a water filling port and the position of the water filling port by using the trained target detection model; the target detection model is a target detection neural network model based on a multi-head region self-attention neural network;
s3, determining the accurate position of the cover plate of the water filling port according to the text position of the water filling port and the position of the water filling port;
s4, determining the self-adaptive binarization threshold value of the image corresponding to the accurate position of the water filling port cover plate obtained in the S3 by adopting an OTSU algorithm, carrying out binarization, setting the pixel value smaller than the self-adaptive binarization threshold value to be 0, and setting the pixel value larger than or equal to the self-adaptive binarization threshold value to be 255, and obtaining a water filling port cover plate binary image;
s5, according to the opening and closing directions of cover plates of water injection ports of different vehicle types, a gap shadow image between the cover plates of the water injection ports and a vehicle body is captured on the binary image of the cover plates of the water injection ports;
s6, calculating the number of pixels with the pixel value of 0 in the gap shadow image, and acquiring the gap shadow area between the water filling port cover plate and the vehicle body;
s7, judging whether the gap shadow area obtained in S6 exceeds a threshold value or not according to the gap shadow area threshold values corresponding to different vehicle types; and if so, recording the accurate position of the water filling port cover plate obtained in the step S3 as fault information.
The second embodiment is as follows:
in the method for detecting a fault that a water injection port cover plate is not locked in place based on deep learning in the embodiment, the processing process of the multi-head area self-attention neural network is as follows:
(1) selecting a sub-region pixel matrix with a window size of m x m pixels from the input image;
(2) copying 3 parts of the matrix obtained in the step (1), and respectively naming the matrix as a query matrix Q, a key matrix K and a value matrix V;
(3) and (3) respectively carrying out linear transformation on the Q, K, V matrixes obtained in the step (2) to obtain matrixes subjected to linear transformation
Figure BDA0002726702760000051
(4) And (3) calculating
Figure BDA0002726702760000052
Correlation matrix between matrices
Figure BDA0002726702760000053
Figure BDA0002726702760000054
The self-attention matrix a is calculated by the softmax function:
Figure BDA0002726702760000055
wherein:
Figure BDA0002726702760000056
and
Figure BDA0002726702760000057
self-attention maps in the vertical direction and the horizontal direction of the image respectively; concat is matrix horizontal splicing; wCThe weight parameters are related to linear transformation, and the size is 2m x m; b isCThe bias parameters are related to linear transformation, and the size is m x m; the size of the self-attention matrix A is m x m;
(5) and using the sub-region matrix obtained from the attention matrix A obtained in the step (4) to the sub-region matrix obtained in the step (3)
Figure BDA0002726702760000058
Weighting to obtain a sub-region characteristic matrix F;
(6) repeating the processes of the steps (4) and (5) for H times, wherein each time of repeating is called a self-attention head;
(7) splicing H sub-area characteristic matrixes F corresponding to H self-attention heads to obtain H sub-area characteristic matrixes F
Figure BDA0002726702760000059
The size is m x mH;
(8) obtained by (7) above
Figure BDA0002726702760000061
Linear transformation is carried out to obtain the final characteristics of the subareas
Figure BDA0002726702760000062
(9) And (3) moving the subareas in the step (1) to the right line by step length S, and repeating the processes (1) to (8) until all the features of the input image are extracted, so as to obtain a feature map with the same size as the input image.
Other steps and parameters are the same as in the first embodiment.
The third concrete implementation mode:
in the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning in the present embodiment, the process of performing linear transformation on the Q, K, V three matrices obtained in step (2) in step (3) is as follows:
Figure BDA0002726702760000063
Figure BDA0002726702760000064
Figure BDA0002726702760000065
wherein: wQ、WK、WVLinear transformation weight parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation weight parameters are m x m; b isQ、BK、BVLinear transformation bias parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation bias parameters are m x m;
Figure BDA0002726702760000066
Figure BDA0002726702760000067
q, K, V matrixes after linear transformation are respectively, and the sizes of the matrixes are m x m.
Other steps and parameters are the same as in the second embodiment.
The fourth concrete implementation mode:
in the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning in the present embodiment, the model structure of the target detection neural network model based on the multi-head region self-attention neural network includes a 1 st region self-attention layer to a 7 th region candidate layer and an output layer;
zone 1 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 2 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 3 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 4 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 5 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 6 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
region 7 candidate layer: 2, connecting a plurality of multi-head area self-attention neural networks in parallel, wherein the multi-head area self-attention neural networks are respectively a position prediction layer and a probability prediction layer;
an output layer: and outputting the predicted position information, the corresponding prediction type and the prediction confidence of the target in the image.
Other steps and parameters are the same as in the second or third embodiment.
The fifth concrete implementation mode:
in the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning in the present embodiment, the self-attention head number H of a multi-head region in a multi-head region self-attention neural network series structure in a 1 st region self-attention layer to a 6 th region candidate layer is 2, 4, 8, 16, 32, and 64, respectively;
in the multi-head region self-attention neural network parallel structure in the 7 th region candidate layer, the number H of the multi-head region self-attention heads in the position prediction layer is 4; the number of multi-headed regions in the probability prediction layer from the attention head H is 2.
Other steps and parameters are the same as in embodiment four.
The sixth specific implementation mode:
in the method for detecting a fault that a water injection port cover plate is not locked in place based on deep learning in the present embodiment, the size of a sub-region in a multi-head region self-attention neural network from an attention layer in a region 1 to a candidate layer in a region 6 is 3x3 pixels, and the step length S is 3 pixels;
in the multi-head area self-attention neural network in the 7 th area candidate layer, the size of the position prediction layer sub-area is 1x1 pixels, and the step length S is 1 pixel; the size of the probability prediction layer sub-region is 1x1 pixels, and the step S is 1 pixel.
Other steps and parameters are the same as those in the fifth embodiment.
The seventh embodiment:
in the method for detecting an unlocked-in-place water injection port cover plate based on deep learning in the present embodiment, the window size of the largest pool layer from the attention layer to the candidate layer in the 1 st area to the 6 th area is 2 × 2, and the step length is 2.
The other steps and parameters are the same as in embodiment six.
The specific implementation mode is eight:
in the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning in the embodiment, a marking process of a data set of a target detection model for training includes the following steps:
according to different motor train units of which the water injection ports belong to, labeling the images by using a labelImg data labeling tool:
the label name of the text of the 'water filling port' is text, and the label category is 1;
the label names of the water injection ports of different vehicle types are cover + vehicle types, the corresponding label types are 2, 3 and 4, and the like.
Other steps and parameters are the same as in one of the first to seventh embodiments.
The specific implementation method nine:
in the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning according to this embodiment, before a data set is marked, a data amplification operation is performed on an acquired image data set to obtain a data set for marking.
The other steps and parameters are the same as in embodiment eight.
The detailed implementation mode is ten:
the embodiment is a water injection port cover plate non-locking in-place fault detection system based on deep learning, and the system is used for a water injection port cover plate non-locking in-place fault detection method based on deep learning.
Examples
Specifically describing the present embodiment with reference to fig. 1, the method for detecting an unlocked-in-place fault of a water injection port cover plate based on deep learning in the present embodiment includes the following steps:
step one, obtaining a side integral image of a motor train unit
And carrying the linear array camera by using the fixing equipment arranged beside the track. And calculating the shooting frequency of the linear array camera according to the moving speed of the motor train unit, and continuously shooting the motor train unit. And seamlessly splicing the plurality of shot linear images to obtain a complete integral image of the lateral part of the motor train unit.
Step two, coarse positioning of water injection port
And acquiring wheelbase information between the axles of the motor train unit by using a sensor arranged beside the track. And (4) according to the wheel base information and the prior information of the position of the water filling port relative to the axle, roughly positioning the water filling port on the whole image of the side part of the motor train unit obtained in the step one, and obtaining an image of the region of interest possibly containing the water filling port.
Step three, preprocessing the image data set
(3.1) construction of raw image dataset
And (5) filtering out images without water injection ports from all the images of the region of interest obtained in the step two, and constructing an original image data set.
(3.2) data amplification
And (3) carrying out image processing operations such as brightness transformation, histogram equalization, Gaussian noise addition and the like on the image samples of the original image data set obtained in the step (3.1) to carry out data amplification, increasing the number of samples and improving the robustness of the trained model.
(3.3) data annotation
And (3) marking the image data set obtained in the step (3.2) by utilizing a labelImg data marking tool according to different motor train units to which the water injection ports belong.
The tag name of the "fill port" text is text and the tag class is 1.
The label names of the water injection ports of different vehicle types are cover + vehicle types, such as cover XXX, cover YY and cover ZZZ, the corresponding label types are 2, 3 and 4, and the like.
(3.4) data set partitioning
And (4) dividing the image data sets obtained in the step (3.2) and the step (3.3) and the corresponding labels into a training data set and a testing data set according to the proportion.
Step four, designing a multi-head area self-attention neural network
The proposed multi-headed region self-attention neural network structure is shown in fig. 2:
(1) a sub-area pixel matrix with a window size of m x m pixels is selected in the input image. Because the pixels far away from each other in the image generally have no correlation, the image is divided into a plurality of sub-regions to be subjected to attention calculation independently, so that the number of parameters of the attention calculation can be greatly reduced, and the calculation speed is improved on the premise of not influencing the extraction effect of the local features of the image.
(2) Copying 3 copies of the matrix obtained in the step (1), and respectively naming the matrix Q of 'Query', 'Key' and 'Value' as matrix V.
(3) And (3) respectively carrying out linear transformation on the Q, K, V matrixes obtained in the step (2), wherein the formula is as follows:
Figure BDA0002726702760000091
Figure BDA0002726702760000092
Figure BDA0002726702760000093
wherein: wQ、WK、WVLinear transformation weight parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation weight parameters are m x m; b isQ、BK、BVLinear transformation bias parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation bias parameters are m x m;
Figure BDA0002726702760000094
Figure BDA0002726702760000095
q, K, V matrixes after linear transformation are respectively, and the sizes of the matrixes are m x m.
(4) Calculated (3) to obtain
Figure BDA0002726702760000096
Correlation matrix between matrices
Figure BDA0002726702760000097
And calculating a self-attention matrix A through a softmax function, wherein the formula is as follows:
Figure BDA0002726702760000098
Figure BDA0002726702760000099
wherein:
Figure BDA00027267027600000910
and
Figure BDA00027267027600000911
self-attention maps in the vertical direction and the horizontal direction of the image respectively; concat is matrix horizontal splicing; wCThe weight parameters are related to linear transformation, and the size is 2m x m; b isCThe bias parameters are related to linear transformation, and the size is m x m; the size of the self-attention matrix a is m x m. The softmax function is formulated as:
Figure BDA0002726702760000101
wherein, aijThe matrix A is the element of the ith row and the jth column; c. CijIs composed of
Figure BDA0002726702760000102
The ith row and the jth column; c. CmjIs composed of
Figure BDA0002726702760000103
Row m and column j.
This process is used to extract the key information in the current sub-region, which is saved in the form of the attention matrix a. The more critical the information contained in a part of a sub-area is, the higher the corresponding attention weight is.
The self-attention matrix calculation schematic is shown in fig. 3:
self-attention force diagram for only adopting single direction relative to traditional self-attention force matrix calculation to obtain correlation matrix
Figure BDA0002726702760000104
The invention provides a self-attention map for splicing vertical and horizontal directions
Figure BDA0002726702760000105
And
Figure BDA0002726702760000106
then, linear transformation is carried out to obtain correlation matrix
Figure BDA0002726702760000107
The method obtains the self-attention map of the image from the vertical and horizontal visual angles, and is more suitable for extracting the image characteristics. Through adding linear transformation, the spliced self-attention map is weighted and subjected to size transformation, automatic selection of vertical and horizontal angle features is realized, and the automatic selection is combined with a 'value' matrix
Figure BDA0002726702760000108
Is adapted to the size of the sensor.
Compared with the method of directly using the whole input image to carry out the self-attention operation, the method of dividing the image into a plurality of sub-areas can greatly reduce the operation amount of the self-attention mechanism. Take an input image of 9 × 9 pixels as an example: if the self-attention calculation is performed directly on the whole image, the number of parameters included in the self-attention matrix a is 81 × 81, which is 6561: if the input image is divided into 9 sub-regions of 3 × 3, the number of parameters included in the attention matrix a is 729, x 9 × 9.
(5) Using the sub-region matrix obtained from the attention matrix A obtained in (4) to the sub-region matrix obtained in (3)
Figure BDA0002726702760000109
And weighting to obtain a sub-region characteristic matrix F. The formula is as follows:
Figure BDA00027267027600001010
wherein: the size of F is m x m.
(6) The processes (4) and (5) are repeated, each process is called a self-attention Head (Head), and the process is repeated for H times.
Wherein: h is a neural network hyper-parameter which is manually set.
Because the parameters of each self-attention head are independent, the information of different semantic subspaces in the current sub-region can be respectively extracted, and the feature extraction capability is stronger.
(7) Splicing H sub-region feature matrixes F corresponding to H self-attention heads to obtain
Figure BDA00027267027600001011
The size is m x mH.
(8) Obtained in the pair (7)
Figure BDA00027267027600001012
Linear transformation is carried out to obtain the final characteristics of the subareas
Figure BDA00027267027600001013
The calculation formula is as follows:
Figure BDA00027267027600001014
wherein:
Figure BDA0002726702760000111
is composed of
Figure BDA0002726702760000112
The corresponding linear transformation weight parameter has the size mH x m;
Figure BDA0002726702760000113
is composed of
Figure BDA0002726702760000114
Corresponding linear transformation bias parameters with the size of m x m;
Figure BDA0002726702760000115
is m x m.
(9) And (4) moving the sub-region pixel matrix (equivalent to a sliding window) in the step (1) to the right line by using a manually set step length (Stride) S, and repeating the steps (1) to (8) until all the features of the input image are extracted. And obtaining a characteristic diagram with the same size as the input image.
Step five, building a target detection model
The structure of the target detection neural network model provided by the invention is shown in FIG. 4:
zone 1 self-attention layer: the 2 same multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 2. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Zone 2 self-attention layer: the 2 same multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 4. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Zone 3 self-attention layer: the 3 same multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 8. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Zone 4 self-attention layer: the 3 identical multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 16. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Zone 5 self-attention layer: 3 identical multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 32. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Zone 6 self-attention layer: the 3 identical multi-head regions are connected in series from the attention neural network, the size of the sub-region is 3x3 pixels, the step length S is 3 pixels, and the number H of the multi-head regions from the attention head is 64. The field of reception is enlarged with a maximum pooling with a window size of 2x2 and a step size of 2.
Region 7 candidate layer: the 2 multi-head areas are connected in parallel with the attention neural network and are respectively a position prediction layer and a probability prediction layer. The size of the position prediction layer sub-region is 1x1 pixels, the step length S is 1 pixel, and the number H of the heads of the multi-head region from the attention is 4 (corresponding to the prediction target coordinate x, the coordinate y, the width w and the height H). The size of the probability prediction layer sub-region is 1x1 pixels, the step length S is 1 pixel, and the number of the multi-head region from the attention head H is 2 (corresponding to the prediction target type c and the prediction confidence conf).
An output layer: and outputting the predicted position information, the corresponding prediction type and the prediction confidence of the target in the image.
The target detection neural network model structure is constructed in a targeted manner aiming at the processed image after being researched, and can effectively extract key features in the image sub-region, so that the prediction accuracy is improved.
Sixthly, training and testing a target detection neural network model
And (6.1) training the target detection neural network model provided in the fifth step by adopting the data set preprocessed in the third step. Wherein the target detection model loss function is mean square error loss;
(6.2) manually adjusting and optimizing neural network hyper-parameters such as the training times of the model, the learning rate, the data batch size and the like according to the training result;
and repeating the processes (6.1) and (6.2) until the neural network model achieves the optimal performance.
Seventhly, detecting the fault of the passing image of the motor train unit
(1) Acquiring a train passing image and corresponding train type information of the motor train unit;
(2) according to different vehicle types, roughly positioning a water injection port to obtain an image of an interested area;
(3) loading the weight of the target detection model obtained in the step six;
(4) acquiring a text position of a water filling port and a position of the water filling port through a target detection model;
(5) acquiring the accurate position of a cover plate of the water filling port according to the text position of the water filling port and the position of the water filling port;
(6) and (3) solving a self-adaptive binary threshold corresponding to the water filling port cover plate accurate position image obtained in the step (5) by adopting an OTSU algorithm, and binarizing the water filling port cover plate accurate position image through the threshold so as to obtain a water filling port cover plate accurate position binary image. Setting the pixel value smaller than the self-adaptive binarization threshold value in the image to be 0, and setting the pixel value larger than or equal to the self-adaptive binarization threshold value to be 255;
(7) according to the opening and closing directions of cover plates of water injection ports of different vehicle types, a gap shadow image between the cover plate of the water injection port and the vehicle body is captured from the binary image of the cover plate of the water injection port obtained in the step (6);
(8) calculating the number of pixels with the pixel value of 0 in the gap shadow image obtained in the step (7) to obtain the gap shadow area between the water filling port cover plate and the vehicle body;
(9) and (4) judging whether the gap shadow area obtained in the step (8) exceeds a threshold value or not according to the gap shadow area threshold values set by different vehicle types. If yes, recording the accurate position of the cover plate of the water filling port obtained in the step (5) as fault information;
(10) and uploading all fault information to an alarm platform.
The present invention is capable of other embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and scope of the present invention.

Claims (10)

1. The method for detecting the fault that the water injection port cover plate is not locked in place based on deep learning is characterized by comprising the following steps of:
s1, acquiring an interested area image including a water injection port part;
s2, acquiring the text position of a water filling port and the position of the water filling port by using the trained target detection model; the target detection model is a target detection neural network model based on a multi-head region self-attention neural network;
s3, determining the accurate position of the cover plate of the water filling port according to the text position of the water filling port and the position of the water filling port;
s4, determining the self-adaptive binarization threshold value of the image corresponding to the accurate position of the water filling port cover plate obtained in the S3 by adopting an OTSU algorithm, carrying out binarization, setting the pixel value smaller than the self-adaptive binarization threshold value to be 0, and setting the pixel value larger than or equal to the self-adaptive binarization threshold value to be 255, and obtaining a water filling port cover plate binary image;
s5, according to the opening and closing directions of cover plates of water injection ports of different vehicle types, a gap shadow image between the cover plates of the water injection ports and a vehicle body is captured on the binary image of the cover plates of the water injection ports;
s6, calculating the number of pixels with the pixel value of 0 in the gap shadow image, and acquiring the gap shadow area between the water filling port cover plate and the vehicle body;
s7, judging whether the gap shadow area obtained in S6 exceeds a threshold value or not according to the gap shadow area threshold values corresponding to different vehicle types; and if so, recording the accurate position of the water filling port cover plate obtained in the step S3 as fault information.
2. The deep learning based water injection port cover plate unlocking-in-place fault detection method according to claim 1, wherein the multi-head area self-attention neural network is processed as follows:
(1) selecting a sub-region pixel matrix with a window size of m x m pixels from the input image;
(2) copying 3 parts of the matrix obtained in the step (1), and respectively naming the matrix as a query matrix Q, a key matrix K and a value matrix V;
(3) and (3) respectively carrying out linear transformation on the Q, K, V matrixes obtained in the step (2) to obtain the processed linear transformationTransformed matrix
Figure FDA0002726702750000011
(4) And (3) calculating
Figure FDA0002726702750000012
Correlation matrix between matrices
Figure FDA0002726702750000013
Figure FDA0002726702750000014
The self-attention matrix a is calculated by the softmax function:
Figure FDA0002726702750000015
wherein:
Figure FDA0002726702750000016
and
Figure FDA0002726702750000017
self-attention maps in the vertical direction and the horizontal direction of the image respectively; concat is matrix horizontal splicing; wCThe weight parameters are related to linear transformation, and the size is 2m x m; b isCThe bias parameters are related to linear transformation, and the size is m x m; the size of the self-attention matrix A is m x m;
(5) weighting the subregion matrix V-obtained in the step (3) by using the attention matrix A obtained in the step (4) to obtain a subregion characteristic matrix F;
(6) repeating the processes of the steps (4) and (5) for H times, wherein each time of repeating is called a self-attention head;
(7) splicing H sub-area characteristic matrixes F corresponding to H self-attention heads to obtain H sub-area characteristic matrixes F
Figure FDA0002726702750000021
The size is m x mH;
(8) obtained by (7) above
Figure FDA0002726702750000022
Linear transformation is carried out to obtain the final characteristics of the subareas
Figure FDA0002726702750000023
(9) And (3) moving the subareas in the step (1) to the right line by step length S, and repeating the processes (1) to (8) until all the features of the input image are extracted, so as to obtain a feature map with the same size as the input image.
3. The deep learning based water injection port cover plate unlocking fault detection method according to claim 2, wherein the step (3) is that the three matrixes Q, K, V obtained in the step (2) are respectively subjected to linear transformation as follows:
Figure FDA0002726702750000024
Figure FDA0002726702750000025
Figure FDA0002726702750000026
wherein: wQ、WK、WVLinear transformation weight parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation weight parameters are m x m; b isQ、BK、BVLinear transformation bias parameters corresponding to the matrix Q, K, V are respectively, and the sizes of the linear transformation bias parameters are m x m;
Figure FDA0002726702750000027
Figure FDA0002726702750000028
q, K, V matrixes after linear transformation are respectively, and the sizes of the matrixes are m x m.
4. The deep learning based water injection port cover plate unlocking-in-place fault detection method according to claim 2, wherein the model structure of the target detection neural network model based on the multi-head region self-attention neural network comprises a 1 st region self-attention layer to a 7 th region candidate layer and an output layer;
zone 1 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 2 self-attention layer: 2 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 3 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 4 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 5 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
zone 6 self-attention layer: 3 same multi-head regions are connected in series with the attention neural network and then connected with a maximum pooling layer;
region 7 candidate layer: 2, connecting a plurality of multi-head area self-attention neural networks in parallel, wherein the multi-head area self-attention neural networks are respectively a position prediction layer and a probability prediction layer;
an output layer: and outputting the predicted position information, the corresponding prediction type and the prediction confidence of the target in the image.
5. The deep learning based water injection port cover plate unlocking-in-place fault detection method according to claim 4, wherein the number of multi-headed regions in the multi-headed region self-attention neural network series structure from the attention layer 1 to the candidate layer 6 is 2, 4, 8, 16, 32, 64;
in the multi-head region self-attention neural network parallel structure in the 7 th region candidate layer, the number H of the multi-head region self-attention heads in the position prediction layer is 4; the number of multi-headed regions in the probability prediction layer from the attention head H is 2.
6. The deep learning based water injection port cover plate unlocking-in-place fault detection method according to claim 5, wherein the size of the sub-region in the multi-head region self-attention neural network from the attention layer 1 to the 6 th region candidate layer is 3x3 pixels, and the step length S is 3 pixels;
in the multi-head area self-attention neural network in the 7 th area candidate layer, the size of the position prediction layer sub-area is 1x1 pixels, and the step length S is 1 pixel; the size of the probability prediction layer sub-region is 1x1 pixels, and the step S is 1 pixel.
7. The deep learning based water injection port cover plate non-locking-in-place fault detection method of claim 6, wherein the window size of the 1 st zone from the attention floor to the largest pool floor among the 6 th zone candidate floors is 2x2 with a step size of 2.
8. The deep learning based water injection port cover plate unlocking fault detection method according to one of claims 1 to 7, wherein the marking process of the data set for the trained target detection model comprises the following steps:
according to different motor train units of which the water injection ports belong to, labeling the images by using a labelImg data labeling tool:
the label name of the text of the 'water filling port' is text, and the label category is 1;
the label names of the water injection ports of different vehicle types are cover + vehicle types, the corresponding label types are 2, 3 and 4, and the like.
9. The method for detecting the unlocked fault of the water injection port cover plate based on the deep learning of claim 8, wherein the data set used for marking is obtained by performing a data amplification operation on the acquired image data set before the data set is marked.
10. Deep learning based water injection port cover plate non-locking in-place fault detection system, characterized in that the system is used to perform the deep learning based water injection port cover plate non-locking in-place fault detection method of one of claims 1 to 9.
CN202011105125.2A 2020-10-15 2020-10-15 Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system Active CN112183453B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011105125.2A CN112183453B (en) 2020-10-15 2020-10-15 Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011105125.2A CN112183453B (en) 2020-10-15 2020-10-15 Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system

Publications (2)

Publication Number Publication Date
CN112183453A true CN112183453A (en) 2021-01-05
CN112183453B CN112183453B (en) 2021-05-11

Family

ID=73950428

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011105125.2A Active CN112183453B (en) 2020-10-15 2020-10-15 Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system

Country Status (1)

Country Link
CN (1) CN112183453B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN207882170U (en) * 2018-02-28 2018-09-18 北京康拓红外技术股份有限公司 A kind of EMU critical component installation condition image detection device
CN109858456A (en) * 2019-02-18 2019-06-07 中国铁路沈阳局集团有限公司科学技术研究所 A kind of rolling stock status fault analysis system
CN109918681A (en) * 2019-03-29 2019-06-21 哈尔滨理工大学 It is a kind of based on Chinese character-phonetic fusion problem semantic matching method
CN110321815A (en) * 2019-06-18 2019-10-11 中国计量大学 A kind of crack on road recognition methods based on deep learning
CN110491146A (en) * 2019-08-21 2019-11-22 浙江工业大学 A kind of traffic signal control scheme real-time recommendation method based on deep learning
CN111079819A (en) * 2019-12-12 2020-04-28 哈尔滨市科佳通用机电股份有限公司 Method for judging state of coupler knuckle pin of railway wagon based on image recognition and deep learning
CN111199288A (en) * 2019-12-20 2020-05-26 山东众阳健康科技集团有限公司 Novel multi-head attention mechanism
CN111445493A (en) * 2020-03-27 2020-07-24 北京市商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN111461190A (en) * 2020-03-24 2020-07-28 华南理工大学 Deep convolutional neural network-based non-equilibrium ship classification method
CN111721535A (en) * 2020-06-23 2020-09-29 中国人民解放军战略支援部队航天工程大学 Bearing fault detection method based on convolution multi-head self-attention mechanism

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN207882170U (en) * 2018-02-28 2018-09-18 北京康拓红外技术股份有限公司 A kind of EMU critical component installation condition image detection device
CN109858456A (en) * 2019-02-18 2019-06-07 中国铁路沈阳局集团有限公司科学技术研究所 A kind of rolling stock status fault analysis system
CN109918681A (en) * 2019-03-29 2019-06-21 哈尔滨理工大学 It is a kind of based on Chinese character-phonetic fusion problem semantic matching method
CN110321815A (en) * 2019-06-18 2019-10-11 中国计量大学 A kind of crack on road recognition methods based on deep learning
CN110491146A (en) * 2019-08-21 2019-11-22 浙江工业大学 A kind of traffic signal control scheme real-time recommendation method based on deep learning
CN111079819A (en) * 2019-12-12 2020-04-28 哈尔滨市科佳通用机电股份有限公司 Method for judging state of coupler knuckle pin of railway wagon based on image recognition and deep learning
CN111199288A (en) * 2019-12-20 2020-05-26 山东众阳健康科技集团有限公司 Novel multi-head attention mechanism
CN111461190A (en) * 2020-03-24 2020-07-28 华南理工大学 Deep convolutional neural network-based non-equilibrium ship classification method
CN111445493A (en) * 2020-03-27 2020-07-24 北京市商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN111721535A (en) * 2020-06-23 2020-09-29 中国人民解放军战略支援部队航天工程大学 Bearing fault detection method based on convolution multi-head self-attention mechanism

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
ASHISH VASWANI ET AL.: "Attention Is All You Need", 《31ST CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS》 *
JINBAO XIE ET AL.: "Chinese Alt Text Writing Based on Deep Learning", 《TRAITEMENT DU SIGNAL》 *
SEOK-JUN BU ET AL.: "Time Series Forecasting with Multi-Headed Attention-Based Deep Learning for Residential Energy Consumption", 《ENERGIES》 *
ZILONG HUANG ET AL.: "CCNet: Criss-Cross Attention for Semantic Segmentation", 《ARXIV》 *
熊漩 等: "融合多头自注意力机制的中文分类方法", 《电子测量技术》 *
管全梅 ET AL.: "高速列车注水口盖板结构优化设计", 《中国铁路》 *

Also Published As

Publication number Publication date
CN112183453B (en) 2021-05-11

Similar Documents

Publication Publication Date Title
US11144814B2 (en) Structure defect detection using machine learning algorithms
CN111784633B (en) Insulator defect automatic detection algorithm for electric power inspection video
Kim et al. Investigation of steel frame damage based on computer vision and deep learning
CN111080609B (en) Brake shoe bolt loss detection method based on deep learning
CN115690561B (en) Pavement anomaly monitoring method
CN112926563B (en) Fault diagnosis system for steel coil spray printing mark
CN113830136B (en) Method for identifying malposition fault of angle cock handle of railway wagon
CN111353396A (en) Concrete crack segmentation method based on SCSEOCUnet
CN111105389A (en) Detection method for pavement crack by fusing Gabor filter and convolutional neural network
CN111563896A (en) Image processing method for catenary anomaly detection
CN115546223A (en) Method and system for detecting loss of fastening bolt of equipment under train
CN115995056A (en) Automatic bridge disease identification method based on deep learning
CN112183453B (en) Deep learning-based water injection port cover plate unlocking-in-place fault detection method and system
CN111144203B (en) Pedestrian shielding detection method based on deep learning
CN113920140B (en) Wagon pipe cover falling fault identification method based on deep learning
CN102073868A (en) Digital image closed contour chain-based image area identification method
Shin et al. Visualization for explanation of deep learning-based defect detection model using class activation map
Cho et al. Application of deep learning-based crack assessment technique to civil structures
CN117593300B (en) PE pipe crack defect detection method and system
Yao et al. Detection of Bughole on Concrete Surface with Convolutional Neural Network
Ramachandraiah et al. Evaluation of Pavement Surface Distress Using Image Processing and Artificial Neural Network
CN116823737B (en) Tunnel wall abnormity detection method and system in low-texture environment
CN116385414B (en) Component multi-scale defect feature extraction method, computing device and storage medium
CN111583176B (en) Image-based lightning protection simulation disc element fault detection method and system
Sahane et al. A Crevice-Centric Approach to Surface Crack Detection

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant