CN114821296A - Underground disease ground penetrating radar image identification method and system, storage medium and terminal - Google Patents
Underground disease ground penetrating radar image identification method and system, storage medium and terminal Download PDFInfo
- Publication number
- CN114821296A CN114821296A CN202210245842.8A CN202210245842A CN114821296A CN 114821296 A CN114821296 A CN 114821296A CN 202210245842 A CN202210245842 A CN 202210245842A CN 114821296 A CN114821296 A CN 114821296A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- disease
- underground
- loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 201000010099 disease Diseases 0.000 title claims abstract description 168
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 title claims abstract description 168
- 238000000034 method Methods 0.000 title claims abstract description 63
- 230000000149 penetrating effect Effects 0.000 title claims abstract description 62
- 238000003860 storage Methods 0.000 title claims abstract description 7
- 238000004088 simulation Methods 0.000 claims abstract description 39
- 238000009826 distribution Methods 0.000 claims abstract description 23
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 6
- 230000004913 activation Effects 0.000 claims description 27
- 238000012549 training Methods 0.000 claims description 24
- 238000011176 pooling Methods 0.000 claims description 21
- 239000010426 asphalt Substances 0.000 claims description 18
- 239000002689 soil Substances 0.000 claims description 18
- 239000002699 waste material Substances 0.000 claims description 13
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000007547 defect Effects 0.000 claims description 10
- 235000014653 Carica parviflora Nutrition 0.000 claims description 9
- 241000243321 Cnidaria Species 0.000 claims description 9
- 238000010521 absorption reaction Methods 0.000 claims description 9
- 239000000463 material Substances 0.000 claims description 9
- 238000005520 cutting process Methods 0.000 claims description 8
- 210000002569 neuron Anatomy 0.000 claims description 7
- 239000011800 void material Substances 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000008030 elimination Effects 0.000 claims description 4
- 238000003379 elimination reaction Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 4
- 230000006978 adaptation Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 239000003245 coal Substances 0.000 abstract 2
- 230000006870 function Effects 0.000 description 34
- 230000000694 effects Effects 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 5
- 238000013145 classification model Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000013508 migration Methods 0.000 description 4
- 230000005012 migration Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000008092 positive effect Effects 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004080 punching Methods 0.000 description 1
- 238000012502 risk assessment Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2431—Multiple classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention belongs to the technical field of image processing, and discloses an underground disease ground penetrating radar image identification method, a system, a storage medium and a terminal, wherein the underground disease ground penetrating radar image identification method comprises the following steps: (1) collecting urban road ground penetrating radar data; (2) preprocessing radar data and marking underground diseases; (3) establishing a forward simulation database of underground diseases; (4) an AlexNet network (5) which simultaneously inputs forward simulation data and real disease data is built, the coal loss is adopted, the distribution difference (6) of the forward simulation data and the real disease data is calculated, the AlexNet network is trained, and meanwhile, the classification cross entropy loss and the coal loss are minimized. The method mainly solves the problems that a classification network is easy to over-fit and the classification accuracy is low under the condition that real underground disease samples are few, and is an effective identification method.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to an underground disease ground penetrating radar image identification method, an underground disease ground penetrating radar image identification system, a storage medium and a terminal.
Background
The ground penetrating radar technology is a nondestructive exploration technology for detecting the distribution rule of an underground medium by utilizing a high-frequency ultra-wideband signal, is an effective means for detecting an underground target, and is also a main means for underground detection of the urban road at present. In the current urban road detection based on a ground penetrating radar method, underground diseases are mainly identified by interpreting a B-scan image of the ground penetrating radar. The underground target in the B-scan image of the ground penetrating radar presents a pair of hyperbolic curve shape characteristics, algorithms such as a Hough algorithm, an LS least square method and the like are used for extracting and fitting a hyperbolic curve to identify the underground target object, however, the characteristics presented by the underground disease in the B-scan image are more complex and do not have obvious hyperbolic curve characteristics; researchers put forward an algorithm to identify linear and hyperbolic characteristics of underground objects in binary images, an SVM classifier classifies the linear and hyperbolic characteristics into linear or hyperbolic shapes, the algorithm artificially designs characteristics of underground target bodies and then identifies the underground target bodies by using the classifier, and the algorithm is suitable for simple underground target bodies and has no good applicability to identification of underground diseases; the artificial neural network DCNN has strong classification performance, but the algorithm has high data requirement, and a large amount of labeled data sets are needed. At present, for the identification and interpretation of underground disease information in images, the following problems mainly exist: firstly, underground pipelines, commercial facilities and underground traffic networks are numerous, and the underground targets can be imaged when the ground penetrating radar is used, so that the interpretation of diseases in the images is greatly interfered; secondly, the disease lacks prior knowledge given by a physical model, the disease explanation lacks standard, and the disease is judged by the experience of professional personnel; thirdly, the manual interpretation consumes longer time, and is easy to miss and misjudge; and fourthly, labeled data are scarce, and the classification model can generate an overfitting phenomenon, so that the classification accuracy is not high.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) underground pipelines, commercial facilities and underground traffic networks are numerous, and the underground targets can be imaged when the ground penetrating radar is used, so that the interpretation of diseases in the images is greatly interfered.
(2) The disease lacks prior knowledge given by a physical model, the disease explanation lacks standard, and judgment is carried out by depending on the experience of professional personnel.
(3) The manual interpretation takes longer time, and the misjudgment is easy to miss.
(4) The labeled data is scarce, and the classification model can generate an overfitting phenomenon, so that the classification accuracy is low.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an underground disease ground penetrating radar image identification method, a system, a storage medium and a terminal.
The invention is realized in such a way, and the method for identifying the underground disease ground penetrating radar image is characterized by comprising the following steps: collecting urban road ground penetrating radar data; preprocessing radar data and marking underground diseases; establishing a forward simulation database of underground diseases; constructing an AlexNet network for inputting forward simulation data and real disease data simultaneously; calculating the distribution difference of forward modeling data and real disease data by adopting the coral loss; the AlexNet network is trained while minimizing class cross-entropy loss and corral loss.
Further, the underground disease ground penetrating radar image identification method comprises the following steps:
the method comprises the steps of firstly, collecting urban road ground penetrating radar data, preprocessing the radar data, marking underground diseases and constructing a real disease data set. The real disease data set is a target task to be classified, labeled data is very precious, and the data set is a data basis for training a classifier and testing classification.
And secondly, performing forward simulation on the underground disease radar image according to the engineering characteristics of the underground disease and the underground disease characteristics to construct a simulation data set. The method of the step adopts a computer numerical simulation method to establish a simulation database, and can effectively provide data knowledge for the training of a neural network and balance the fitting ability and generalization performance of the network.
Introducing a corral loss, calculating the distribution difference of the forward modeling data and the real disease data, effectively measuring the distance between the forward modeling data and the real disease data in a characteristic space, and providing the corral loss of an optimization function;
and fourthly, training the AlexNet network, minimizing the classified cross entropy loss and the corral loss, and performing distribution adaptation on the forward modeling data and the real disease data in a feature space, wherein the forward modeling data is effectively utilized in the training process, the recognition of important features of the real disease data of the neural network is enhanced, and the classification accuracy is improved. And stopping training when the trained network converges to obtain a classification network model.
Further, data merging is carried out on the urban road ground penetrating radar data, and data which are not continuously collected are merged into the same survey line data; on the basis of merging the data, replacing the data of the waste channel with the data around the waste channel to finish the waste channel elimination; time zero calibration is carried out on the data subjected to the rejection of the waste channels, the position of the ground reflected wave is adjusted to a time zero point, and the data acquisition result is ensured to be consistent with the field result; gain is adjusted, deep geology and target body signals are enhanced, and a higher-quality image is obtained; performing intra-channel equalization to enhance deep signals; and (3) cutting the image of the disease area in the radar data, wherein the cutting principle is that the upper part, the lower part, the left part and the right part of the obtained cut image are 20 pixels more than the whole disease area, and the image is stored according to the category.
Further, the forward modeling method is based on the gprMax platform, data are randomly generated in batches, and three disease parameters are set as follows:
cavity diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting a material layer as an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box order of 04.3508.04.50.01, the mixed layer is set to be in a 03.808.04.350.01 order, the base soil layer is set to be in a 0008.03.80.01 order, a defect shape is set to be a rectangle with random size, the position is the random size in the middle of the area, the dielectric constant is 1, and a rough surface is added;
and (3) void diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting the material layer as an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box order of 04.3508.04.50.01, the mixed layer is set to be in a 03.808.04.350.01 order, the base soil layer is set to be in a 0008.03.80.01 order, the defect body is set to be a long and narrow random rectangle with the length-width ratio of more than 3, the position is located at the middle upper part of the area, the dielectric constant is 1, and adding a rough surface;
loosening diseases: 8.05.00.01, adding a PML absorption boundary, exciting at the frequency of 400MHz, setting the material layers to be an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box instruction of 04.3508.04.50.01, the mixed layer is set to be in an instruction of 03.808.04.350.01, the base soil layer is set to be in an instruction of 0008.03.80.01, the defect body is a random rectangular combination with the length-width ratio of more than 4, the position is the random size in the middle of the area, the dielectric constant is a random number, and the range is 10-30; adding a rough surface setting.
Further, an AlexNet network is built, the first five convolutional layers are used as a main network for feature extraction, and the structure sequentially comprises: a first convolution layer, an activation function layer, a maximum pooling layer, a second convolution layer, an activation function layer, a maximum pooling layer, a third convolution layer, an activation function layer, a fourth convolution layer, an activation function layer, a fifth convolution layer, an activation function layer, and a maximum pooling layer; setting the number of convolution kernels of the first to fifth convolution layers to be 48, 128, 128, 192 and 128 in sequence, setting the sizes of the convolution kernels to be 11, 5, 3, 3 and 3 in sequence, setting the step size of the first convolution layer to be 4, setting the step sizes of the second, third, fourth and fifth convolution layers to be 1, filling the first convolution layer and the second convolution layer to be 2, and filling the third, fourth and fifth convolution layers to be 1; the size of the core of the maximum pooling layer pooling area is set to be 3 multiplied by 3, and the step length is set to be 2; the activation function layer adopts a linear rectification function, and the maximum pooling layer adopts a regional maximum pooling function. The last three-layer full-connection layer is as the classifier, and the structure once is: a sixth full connection layer, an activation function layer, a seventh full connection layer, an activation function layer, and an eighth full connection layer; the number of neurons in the sixth and seventh full connection layers is 2048, the number of neurons in the eighth full connection layer is 3, and the activation function layer adopts a linear rectification function. The linear rectification function is expressed as follows:
where a is a fixed parameter greater than 1 and x is an input.
Further, a method for calculating the distribution difference of forward modeling data and real disease data by adopting the coral loss is adopted to calculate the distance between the forward modeling data and the second-order statistic of the real disease data, and the loss function is defined as follows:
wherein l CORAL Representing the corral loss, d representing the number of columns of the matrix, C S And C T Respectively representing forward modeling data and a characteristic covariance matrix of real disease data,represents the Frobenius norm; the feature covariance matrix of the modeling data and the real disease data is defined as follows:
wherein n is S Representing the number of forward simulated data samples, n T Representing true disease data samplesThe number; 1 is a column vector with all elements equal to 1, X T Represents the transpose of matrix X; the gradient of the input feature can be calculated using the following chain rule:
Further, a method of training an AlexNet network while minimizing class cross entropy loss and corral loss, comprising: the overall loss function is defined as follows:
where l is the total loss, l is CLASS Is the categorical cross entropy loss, t is the number of layers of the corral loss, λ is the hyper-parameter that adjusts the corral loss, defined as follows:
and b is the ratio of the current iteration turn to the total turn during network training.
Another object of the present invention is to provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the underground disease ground penetrating radar image identification method.
The invention further aims to provide an information data processing terminal, and the information data processing terminal is used for realizing the underground disease ground penetrating radar image identification method.
Another object of the present invention is to provide an underground disease ground penetrating radar image recognition system implementing the underground disease ground penetrating radar image recognition method, the underground disease ground penetrating radar image recognition system comprising:
the data acquisition module is used for acquiring urban road ground penetrating radar data;
the data preprocessing module is used for preprocessing the radar data and marking underground diseases;
the database establishing module is used for establishing a forward simulation database of the underground diseases;
the network building module is used for building an AlexNet network for inputting forward simulation data and real disease data;
the distribution difference calculation module is used for calculating the distribution difference of forward modeling data and real disease data through the corral loss;
and the network training module is used for AlexNet network training to minimize the classification cross entropy loss and the corral loss.
In combination with the technical solutions and the technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected in the present invention from the following aspects:
first, aiming at the technical problems existing in the prior art and the difficulty in solving the problems, the technical problems to be solved by the technical scheme of the present invention are closely combined with results, data and the like in the research and development process, and some creative technical effects are brought after the problems are solved. The specific description is as follows:
according to the method, a large amount of forward modeling simulation data is constructed by combining engineering characteristics and underground physical characteristics of underground diseases, prior guidance of numerical simulation is given, and meanwhile, the overfitting phenomenon in the network training process is reduced; the method comprises the steps of extracting features of forward simulation data and real disease data by utilizing a deep neural network AlexNet, measuring distribution difference of the forward simulation data and the real disease data by adopting a coral loss, and calculating a distance between the forward simulation data and second-order statistics of the real disease data; the invention trains the neural network by combining forward modeling data and real disease data to obtain a model with good classification effect, thereby effectively completing the underground disease identification task.
Secondly, considering the technical scheme as a whole or from the perspective of products, the technical effect and advantages of the technical scheme to be protected by the invention are specifically described as follows:
the invention has the advantages of end-to-end identification, and simplifies the steps of manual design and feature extraction; the method for identifying the underground diseases in the ground penetrating radar image provided by the invention provides the migration from the forward modeling data set to the real data to the real disease data, and the field personnel can change a network structure such as ResNet and a migration algorithm such as MMD and the like according to the method and the actual requirements, thereby providing a new solution for the task of carrying out classification and identification on the underground target body based on the ground penetrating radar image under the condition of few samples; the method has the characteristics of automation and intellectualization, has higher accuracy rate compared with the manual identification of the underground diseases, simultaneously reduces the requirements on professional interpretation workers, saves the time for identifying the underground diseases and improves the identification efficiency; the method can be used for sample labeling of the underground diseases of the ground penetrating radar image, so that the data scale is expanded, more data can be adopted for training the classification model, and the accuracy of the classification model is further improved.
Third, as an inventive supplementary proof of the claims of the present invention, there are also presented several important aspects:
the ground penetrating radar has the advantages of high detection precision, high speed, low cost, good effect and the like, and a nondestructive detection mode is adopted, so that normal traffic operation is not influenced. Whether the underground diseases of the road exist or not is detected and judged through the ground penetrating radar, and data support and technical basis are provided for monitoring and controlling the urban road diseases. As a key ring of municipal road maintenance, the research on the automatic identification technology of underground diseases in ground penetrating radar images is particularly important, and due to the shortage of labeled data and the complex waveform of the disease images, the research on the current identification method is restricted. The method researches engineering characteristics and geophysical characteristics of urban road underground diseases, further constructs a forward modeling data set, expands samples and provides a good paradigm for solving similar problems; meanwhile, at present, the identification of underground diseases at home and abroad is mainly focused on the identification of underground cavities, and the method diagnoses the voids, cavities and loose diseases at the same time.
Drawings
FIG. 1 is a flow chart of an image identification method for an underground disease ground penetrating radar provided by an embodiment of the invention;
FIG. 2 is a schematic structural diagram of an underground disease ground penetrating radar image identification system provided by an embodiment of the invention;
FIG. 3 is a schematic diagram of the overall network structure provided by the embodiment of the present invention;
FIG. 4 is a schematic diagram of the network architecture layers provided by the embodiment of the present invention;
FIG. 5 is a schematic diagram of forward simulation data of an underground disease provided by an embodiment of the present invention;
FIG. 6 is a schematic diagram of underground disease real data of a ground penetrating radar provided by an embodiment of the invention;
FIG. 7 is a diagram illustrating the recognition effect of the recognition method proposed in this patent according to an embodiment of the present invention;
in the figure: 1. a data acquisition module; 2. a data preprocessing module; 3. a database establishing module; 4. building a module; 5. a distribution difference calculation module; 6. and a network training module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
First, an embodiment is explained. This section is an illustrative example developed to explain the claims in order to enable those skilled in the art to fully understand how to implement the present invention.
As shown in fig. 1, the method for identifying the underground disease ground penetrating radar image provided by the invention comprises the following steps:
s101: collecting urban road ground penetrating radar data;
s102: preprocessing radar data and marking underground diseases;
s103: establishing a forward simulation database of underground diseases;
s104: constructing an AlexNet network for inputting forward simulation data and real disease data;
s105: calculating the distribution difference of forward modeling data and real disease data by the corral loss;
s106: AlexNet network training minimizes class cross entropy loss as well as corral loss.
As shown in fig. 2, the underground disease ground penetrating radar image recognition system provided by the invention comprises:
the data acquisition module 1 is used for acquiring urban road ground penetrating radar data;
the data preprocessing module 2 is used for preprocessing the radar data and marking underground diseases;
the database building module 3 is used for building a forward simulation database of the underground diseases;
the network building module 4 is used for building an AlexNet network for inputting forward simulation data and real disease data;
the distribution difference calculation module 5 is used for calculating the distribution difference of forward modeling data and real disease data according to the corral loss;
and the network training module 6 is used for AlexNet network training to minimize the classification cross entropy loss and the corral loss.
The radar image acquisition provided by the invention is mainly used for urban main roads, and data preprocessing is carried out after the images are acquired, wherein the method comprises the following steps: data merging, waste channel elimination, time zero correction, gain adjustment, in-channel balance and image clipping. The cutting principle is that the upper part, the lower part, the left part and the right part of the obtained cutting picture are 20 pixels more than the whole disease area, the image is stored according to the category, and a data set of void, cavity and disease is obtained.
A forward simulation database of underground diseases is established, and the forward simulation database is mainly used for urban road underground disease data of empty roads, cavities and diseases. Each data is subjected to parameter range setting according to engineering characteristics and geophysical characteristics of urban road underground diseases, python scripts are compiled, simulation instructions are generated in batches, and a large amount of data are generated.
As shown in fig. 3, the method for identifying an underground disease in a ground penetrating radar image provided by the present invention provides a migration from a forward simulation data set to real data to real disease data, and an industry practitioner can replace a network structure such as ResNet and a migration algorithm such as MMD according to the method.
The method for identifying the underground diseases in the ground penetrating radar image, provided by the invention, is characterized in that the data of the urban road ground penetrating radar is collected, the data of the radar is preprocessed, the underground diseases are marked, and a real disease data set is constructed; forward modeling of underground diseases is carried out through a gprMax platform, and a large-batch forward modeling data set is constructed; building an AlexNet network for inputting forward simulation data and real disease data simultaneously, wherein in the AlexNet network, the distribution difference of the forward simulation data and the real disease data is calculated by adopting the corral loss, the AlexNet network is trained, the classified cross entropy loss and the corral loss are minimized at the same time, and the forward simulation data and the real disease data are subjected to distribution adaptation in a feature space; and stopping training when the trained network converges to obtain a classification network model.
Specifically, data merging is carried out on urban road ground penetrating radar data, and data which are not continuously collected are merged into the same survey line data; on the basis of merging the data, replacing the data of the waste channel with the data around the waste channel to finish the waste channel elimination; time zero calibration is carried out on the data subjected to the rejection of the waste channels, the position of the ground reflected wave is adjusted to a time zero point, and the data acquisition result is ensured to be consistent with the field result; gain is adjusted, deep geology and target body signals are enhanced, and a higher-quality image is obtained; performing intra-channel equalization to enhance deep signals; and (3) cutting the image of the disease area in the radar data, wherein the cutting principle is that the upper part, the lower part, the left part and the right part of the obtained cut image are 20 pixels more than the whole disease area, and storing the image according to categories to obtain a data set of void, cavity and disease.
Further, a forward modeling data set of the underground disease radar image is constructed, the forward modeling method provided by the invention is based on a gprMax platform, data are randomly generated in batches, and the parameters of three disease parts are set as follows:
cavity diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting the material layer as an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box-type 04.3508.04.50.01, the mixed layer is set to be in a 03.808.04.350.01, the base soil layer is set to be in a 0008.03.80.01, the defect shape is set to be a rectangle with random size, the position is the random size in the middle of the area, the dielectric constant is 1, and the rough surface is added.
And (3) void diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting the material layers to be an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be a box:04.3508.04.50.01, the mixed layer is set to be 03.808.04.350.01, the base soil layer is set to be 0008.03.80.01, the defect body is a long and narrow random rectangle with the length-width ratio of more than 3, the position is located at the middle upper part of the area, the dielectric constant is 1, and adding a rough surface setting.
Loosening diseases: 8.05.00.01, adding a PML absorption boundary, exciting at the frequency of 400MHz, setting the material layers to be an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box instruction of 04.3508.04.50.01, the mixed layer is set to be in an instruction of 03.808.04.350.01, the base soil layer is set to be in an instruction of 0008.03.80.01, the defect body is a random rectangular combination with the length-width ratio of more than 4, the position is the random size in the middle of the area, the dielectric constant is a random number, and the range is 10-30; adding a rough surface setting. And compiling a script by adopting python, and generating forward modeling data in a large batch.
As shown in a network structure model of fig. 4, an AlexNet network is built, the first five convolutional layers are used as a backbone network for feature extraction, and the structure is as follows: a first convolution layer, an activation function layer, a maximum pooling layer, a second convolution layer, an activation function layer, a maximum pooling layer, a third convolution layer, an activation function layer, a fourth convolution layer, an activation function layer, a fifth convolution layer, an activation function layer, and a maximum pooling layer; setting the number of convolution kernels of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer to be 48, 128, 128, 192 and 128 in sequence, setting the sizes of the convolution kernels to be 11, 5, 3, 3 and 3 in sequence, setting the step size of the first convolution layer to be 4, setting the step sizes of the second convolution layer, the third convolution layer, the fourth convolution layer and the fifth convolution layer to be 1, filling the first convolution layer and the second convolution layer to be 2, and filling the third convolution layer, the fourth convolution layer and the fifth convolution layer to be 1; the size of the core of the maximum pooling layer pooling area is set to be 3 multiplied by 3, and the step length is set to be 2; the activation function layer adopts a linear rectification function, and the maximum pooling layer adopts a regional maximum pooling function. The last three-layer full-connection layer is as the classifier, and the structure once is: a sixth full connection layer, an activation function layer, a seventh full connection layer, an activation function layer, and an eighth full connection layer; the full connection layer dropput is set to be 0.5, the number of the sixth full connection layer neuron and the number of the seventh full connection layer neuron are both 2048, the number of the eighth full connection layer neuron is 3, a softmax function is adopted to output classification probability, and the softmax function is defined as follows:
x represents the vector output by the eighth fully-connected layer, and the activation function layer adopts a linear rectification function. The linear rectification function is expressed as follows:
wherein a is a fixed parameter greater than 1 and x is an input;
further, the method measures the distribution difference of the forward modeling data and the real disease data by adopting the coral loss, calculates the distance between the forward modeling data and the second-order statistic of the real disease data, and defines the loss function as follows:
wherein l CORAL Representing the corral loss, d representing the number of columns of the matrix, C S And C T Respectively representing forward modeling data and a characteristic covariance matrix of real disease data,representing the Frobenius norm. The feature covariance matrix of the modeling data and the real disease data is defined as follows:
wherein n is S Representing the number of forward simulated data samples, n T Representing the number of real disease data samples. 1 is a column vector with all elements equal to 1, X T Representing the transpose of matrix X. The gradient of the input feature can be calculated using the following chain rule:
Further training the AlexNet network, inputting forward simulation data and real disease data, and simultaneously minimizing classification cross entropy loss and corral loss, wherein a loss function of the whole network is defined as follows:
where l is the total loss, l is CLASS Is the categorical cross entropy loss, t is the number of layers of the corral loss, λ is the hyper-parameter that adjusts the corral loss, defined as follows:
and b is the ratio of the current iteration turn to the total turn during network training.
Second, the application embodiment. In order to prove the creativity and the technical value of the technical scheme of the invention, the part is the application example of the technical scheme of the claims on specific products or related technologies.
The method is implemented in underground disease data acquired by a real ground penetrating radar. The data of the ground penetrating radar is collected by a GSSI two-dimensional ground penetrating radar produced by the American Lao Rei industry, the data are marked with holes, voids and loose diseases according to a classification standard in 'JGJ/T437-plus 2018 urban underground disease body comprehensive detection and risk assessment technical standard' issued by the Ministry of housing and urban and rural construction in 2018, part of marking information is from on-site punching verification, and part of marking information is from research and judgment of professional detection personnel. In the implementation process, an underground disease forward modeling data set in the ground penetrating radar image is constructed, and the data set has prior guiding significance based on engineering characteristics and geophysical characteristics of the underground disease. In the implementation process of the invention, a deep neural network AlexNet is constructed as a feature extractor, and a full-connection layer softmax function is adopted as a classifier. When the network is trained in the implementation process, the distribution difference of forward simulation data and real disease data is calculated by adopting the corral loss, and meanwhile, the classification cross entropy loss and the corral loss are minimized to obtain a classification network, so that a good classification effect is obtained in the test stage.
And thirdly, evidence of relevant effects of the embodiment. The embodiment of the invention achieves some positive effects in the process of research and development or use, and has great advantages compared with the prior art, and the following contents are described by combining data, diagrams and the like in the test process.
1. Experimental Environment
The hardware platform of the experiment of the invention is as follows: the processor is an Intel (R) core (TM) i7-10700K CPU, the main frequency is 2.9GHz, the memory is 64GB, and the display card is NVIDIA GeForce GTX 1080 Ti. The software platform of the simulation experiment of the invention is as follows: windows 10 operating system, Pycharm2021 software, python3.6 and Pythroch deep learning framework.
2. Content of experiment and analysis of results
Two data sets are used in an experiment, wherein the first data set is a forward simulation data set, the data set is generated by using a gprMax platform in a simulation mode, three types of disease images including void, cavity and loose are generated, and each image is 1000 images; the second data set is a real disease data set, and is respectively provided with 160, 146 and 120 graphs of three types of diseases including void, cavity and loose.
In simulation experiments, the gprMax platform refers to Warren, C., Giannopoulos, A., & Giannakis I. (2016.) the gprMax: Open source software to complex electronic wave mapping for group networking radio, Computer Physics Communications
In order to qualitatively describe the classification effect, the classification accuracy is adopted in the experiment, and the classification accuracy refers to the ratio of the number of correctly classified samples to the total number of samples in the test. The classification accuracy in this experiment was 80.4%. The classification effect graph is shown in fig. 7.
It should be noted that the embodiments of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the apparatus and methods described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided on a carrier medium such as a disk, CD-or DVD-ROM, programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier, for example. The apparatus and its modules of the present invention may be implemented by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of hardware circuits and software, e.g., firmware.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. The method for identifying the underground disease ground penetrating radar image is characterized by comprising the following steps of: collecting urban road ground penetrating radar data; preprocessing radar data and marking underground diseases; establishing a forward simulation database of underground diseases; constructing an AlexNet network for simultaneously inputting forward simulation data and real disease data; calculating the distribution difference of forward modeling data and real disease data by adopting the corral loss; the AlexNet network is trained while minimizing class cross-entropy loss and corral loss.
2. The underground disease ground penetrating radar image identification method of claim 1, wherein the underground disease ground penetrating radar image identification method comprises:
firstly, collecting urban road ground penetrating radar data, preprocessing the radar data, marking underground diseases and constructing a real disease data set;
secondly, forward modeling of an underground disease radar image is carried out according to the engineering characteristics of the underground disease and the characteristics of the underground disease, and a simulation data set is constructed;
thirdly, introducing a coral loss, and calculating the distribution difference of forward modeling data and real disease data;
fourthly, training the AlexNet network, minimizing the classified cross entropy loss and the corral loss, and performing distribution adaptation on forward modeling data and real disease data in a feature space; and stopping training when the trained network converges to obtain a classification network model.
3. The method for identifying an underground disease ground penetrating radar image as claimed in claim 2, wherein the data of the urban road ground penetrating radar are merged, and the data which are not collected continuously are merged into the same survey line data; on the basis of merging the data, replacing the data of the waste channel with the data around the waste channel to finish the waste channel elimination; time zero calibration is carried out on the data subjected to the rejection of the waste channels, the position of the ground reflected wave is adjusted to a time zero point, and the data acquisition result is ensured to be consistent with the field result; gain is adjusted, deep geology and target body signals are enhanced, and a higher-quality image is obtained; performing intra-channel equalization to enhance deep signals; and (3) cutting the image of the disease area in the radar data, wherein the cutting principle is that the upper part, the lower part, the left part and the right part of the obtained cut image are 20 pixels more than the whole disease area, and the image is stored according to the category.
4. The method for identifying the underground disease ground penetrating radar image as claimed in claim 2, wherein the forward modeling method is based on a gprMax platform, data are randomly generated in batches, and three disease parameters are set as follows:
cavity diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting a material layer as an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box order of 04.3508.04.50.01, the mixed layer is set to be in a 03.808.04.350.01 order, the base soil layer is set to be in a 0008.03.80.01 order, a defect shape is set to be a rectangle with random size, the position is the random size in the middle of the area, the dielectric constant is 1, and a rough surface is added;
and (3) void diseases: 8.05.00.01, adding a PML absorption boundary, exciting the frequency of 400MHz, setting the material layer as an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box order of 04.3508.04.50.01, the mixed layer is set to be in a 03.808.04.350.01 order, the base soil layer is set to be in a 0008.03.80.01 order, the defect body is set to be a long and narrow random rectangle with the length-width ratio of more than 3, the position is located at the middle upper part of the area, the dielectric constant is 1, and adding a rough surface;
loosening diseases: 8.05.00.01, adding a PML absorption boundary, exciting at the frequency of 400MHz, setting the material layers to be an asphalt layer, a mixed layer and a base soil layer, wherein the asphalt layer is set to be in a box instruction of 04.3508.04.50.01, the mixed layer is set to be in an instruction of 03.808.04.350.01, the base soil layer is set to be in an instruction of 0008.03.80.01, the defect body is a random rectangular combination with the length-width ratio of more than 4, the position is the random size in the middle of the area, the dielectric constant is a random number, and the range is 10-30; adding a rough surface setting.
5. The method for identifying the underground disease ground penetrating radar image as claimed in claim 2, wherein an AlexNet network is built, the first five convolutional layers are used as a main network for feature extraction, and the structure sequentially comprises: a first convolution layer, an activation function layer, a maximum pooling layer, a second convolution layer, an activation function layer, a maximum pooling layer, a third convolution layer, an activation function layer, a fourth convolution layer, an activation function layer, a fifth convolution layer, an activation function layer, and a maximum pooling layer; setting the number of convolution kernels of the first to fifth convolution layers to be 48, 128, 128, 192 and 128 in sequence, setting the sizes of the convolution kernels to be 11, 5, 3, 3 and 3 in sequence, setting the step size of the first convolution layer to be 4, setting the step sizes of the second, third, fourth and fifth convolution layers to be 1, filling the first convolution layer and the second convolution layer to be 2, and filling the third, fourth and fifth convolution layers to be 1; the size of the core of the maximum pooling layer pooling area is set to be 3 multiplied by 3, and the step length is set to be 2; the activation function layer adopts a linear rectification function, and the maximum pooling layer adopts a regional maximum pooling function; the last three-layer full-connection layer is as the classifier, and the structure once is: a sixth full connection layer, an activation function layer, a seventh full connection layer, an activation function layer, and an eighth full connection layer; the number of neurons in the sixth and seventh full connection layers is 2048, the number of neurons in the eighth full connection layer is 3, and the activation function layer adopts a linear rectification function; the linear rectification function is expressed as follows:
where a is a fixed parameter greater than 1 and x is an input.
6. The method for identifying an underground disease ground penetrating radar image as claimed in claim 2, wherein the method for calculating the distribution difference of the forward modeling data and the real disease data by adopting the coral loss is adopted to calculate the distance between the forward modeling data and the second-order statistic of the real disease data, and the loss function is defined as follows:
wherein l CORAL Representing the corral loss, d representing the number of columns of the matrix, C S And C T Respectively representing forward modeling data and a characteristic covariance matrix of real disease data,represents the Frobenius norm; the feature covariance matrix of the modeling data and the real disease data is defined as follows:
wherein n is S Representing the number of forward simulated data samples, n T Representing the number of real disease data samples; 1 is a column vector with all elements equal to 1, X T Represents the transpose of matrix X; the gradient of the input feature can be calculated using the following chain rule:
7. The method for identifying an underground disease ground penetrating radar image as claimed in claim 2, wherein the method for training the AlexNet network while minimizing the classification cross entropy loss and the corral loss comprises the following steps: the overall loss function is defined as follows:
where l is the total loss, l is CLASS Is the categorical cross entropy loss, t is the number of layers of the corral loss, λ is the hyper-parameter that adjusts the corral loss, defined as follows:
and b is the ratio of the current iteration turn to the total turn during network training.
8. A computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to execute the steps of the method for identifying an image of an underground disease georadar according to any one of claims 1 to 7.
9. An information data processing terminal, characterized in that the information data processing terminal is used for realizing the underground disease ground penetrating radar image identification method as claimed in any one of claims 1 to 7.
10. An underground disease ground penetrating radar image recognition system for implementing the underground disease ground penetrating radar image recognition method according to any one of claims 1 to 7, wherein the underground disease ground penetrating radar image recognition system comprises:
the data acquisition module is used for acquiring urban road ground penetrating radar data;
the data preprocessing module is used for preprocessing the radar data and marking underground diseases;
the database establishing module is used for establishing a forward simulation database of the underground diseases;
the network building module is used for building an AlexNet network for inputting forward simulation data and real disease data;
the distribution difference calculation module is used for calculating the distribution difference of forward modeling data and real disease data according to the corral loss;
and the network training module is used for AlexNet network training to minimize the classification cross entropy loss and the corral loss.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210245842.8A CN114821296B (en) | 2022-03-14 | 2022-03-14 | Underground disease ground penetrating radar image identification method, system, storage medium and terminal |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210245842.8A CN114821296B (en) | 2022-03-14 | 2022-03-14 | Underground disease ground penetrating radar image identification method, system, storage medium and terminal |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114821296A true CN114821296A (en) | 2022-07-29 |
CN114821296B CN114821296B (en) | 2024-07-19 |
Family
ID=82529904
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210245842.8A Active CN114821296B (en) | 2022-03-14 | 2022-03-14 | Underground disease ground penetrating radar image identification method, system, storage medium and terminal |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114821296B (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115343685A (en) * | 2022-08-29 | 2022-11-15 | 北京国电经纬工程技术有限公司 | Multi-dimensional ground penetrating radar detection method, device and equipment applied to disease identification |
CN117077450A (en) * | 2023-10-17 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area volume evolution prediction method, electronic equipment and storage medium |
CN117077449A (en) * | 2023-10-17 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area height evolution prediction method, electronic equipment and storage medium |
WO2024119624A1 (en) * | 2022-12-05 | 2024-06-13 | 中公高科养护科技股份有限公司 | Method and apparatus for determining actual size of disease, and device |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112232392A (en) * | 2020-09-29 | 2021-01-15 | 深圳安德空间技术有限公司 | Data interpretation and identification method for three-dimensional ground penetrating radar |
CN112462346A (en) * | 2020-11-26 | 2021-03-09 | 西安交通大学 | Ground penetrating radar roadbed defect target detection method based on convolutional neural network |
CN113009447A (en) * | 2021-03-05 | 2021-06-22 | 长安大学 | Road underground cavity detection early warning method based on deep learning and ground penetrating radar |
-
2022
- 2022-03-14 CN CN202210245842.8A patent/CN114821296B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112232392A (en) * | 2020-09-29 | 2021-01-15 | 深圳安德空间技术有限公司 | Data interpretation and identification method for three-dimensional ground penetrating radar |
CN112462346A (en) * | 2020-11-26 | 2021-03-09 | 西安交通大学 | Ground penetrating radar roadbed defect target detection method based on convolutional neural network |
CN113009447A (en) * | 2021-03-05 | 2021-06-22 | 长安大学 | Road underground cavity detection early warning method based on deep learning and ground penetrating radar |
Non-Patent Citations (2)
Title |
---|
KANG M S等: "Deep learning-based automated underground cavity detection using three-dimensional ground penetrating radar", 《STRUCTURAL HEALTH MORNITORING》, vol. 19, no. 1, 29 March 2019 (2019-03-29), pages 173 - 185 * |
高旭: "基于深度学习的围岩钻孔裂隙技术研究及应用", 《中国优秀硕士学位论文全文数据库 工程科技I辑》, 15 January 2021 (2021-01-15), pages 021 - 260 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115343685A (en) * | 2022-08-29 | 2022-11-15 | 北京国电经纬工程技术有限公司 | Multi-dimensional ground penetrating radar detection method, device and equipment applied to disease identification |
WO2024119624A1 (en) * | 2022-12-05 | 2024-06-13 | 中公高科养护科技股份有限公司 | Method and apparatus for determining actual size of disease, and device |
CN117077450A (en) * | 2023-10-17 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area volume evolution prediction method, electronic equipment and storage medium |
CN117077449A (en) * | 2023-10-17 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area height evolution prediction method, electronic equipment and storage medium |
CN117077450B (en) * | 2023-10-17 | 2024-03-26 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area volume evolution prediction method, electronic equipment and storage medium |
CN117077449B (en) * | 2023-10-17 | 2024-03-26 | 深圳市城市交通规划设计研究中心股份有限公司 | Road void area height evolution prediction method, electronic equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN114821296B (en) | 2024-07-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN114821296A (en) | Underground disease ground penetrating radar image identification method and system, storage medium and terminal | |
Zheng et al. | Application of wavelet-packet transform driven deep learning method in PM2. 5 concentration prediction: A case study of Qingdao, China | |
Rahaman et al. | An efficient multilevel thresholding based satellite image segmentation approach using a new adaptive cuckoo search algorithm | |
CN107169556A (en) | stem cell automatic counting method based on deep learning | |
CN112785450B (en) | Soil environment quality partitioning method and system | |
CN102184423B (en) | Full-automatic method for precisely extracting regional impervious surface remote sensing information | |
Beucher et al. | Interpretation of convolutional neural networks for acid sulfate soil classification | |
CN116229380A (en) | Method for identifying bird species related to bird-related faults of transformer substation | |
CN115602337A (en) | Cryptocaryon irritans disease early warning method and system based on machine learning | |
CN111949535A (en) | Software defect prediction device and method based on open source community knowledge | |
CN104680185A (en) | Hyperspectral image classification method based on boundary point reclassification | |
Al_Janabi et al. | Pragmatic method based on intelligent big data analytics to prediction air pollution | |
CN109669017B (en) | Refinery distillation tower top cut water ion concentration prediction method based on deep learning | |
Hariri et al. | Tipburn disorder detection in strawberry leaves using convolutional neural networks and particle swarm optimization | |
Wang et al. | Research on air pollution gases recognition method based on lstm recurrent neural network and gas sensors array | |
CN114973019A (en) | Deep learning-based geospatial information change detection classification method and system | |
CN114782740A (en) | Remote sensing water quality monitoring method combining genetic optimization and extreme gradient promotion | |
CN111242028A (en) | Remote sensing image ground object segmentation method based on U-Net | |
Huang et al. | Prediction of chlorophyll a and risk assessment of water blooms in Poyang Lake based on a machine learning method | |
Chen et al. | A novel predictor for exploring PM2. 5 spatiotemporal propagation by using convolutional recursive neural networks | |
Chen et al. | A novel prediction approach for exploring PM2. 5 spatiotemporal propagation based on convolutional recursive neural networks | |
Arif et al. | Image pattern recognition in spatial data using artificial neural network | |
CN115909086A (en) | SAR target detection and identification method based on multistage enhanced network | |
Liu et al. | ANN-based prediction of ammonia nitrogen for wastewater discharge indicators under carbon neutral trend | |
CN115392128A (en) | Method for simulating basin runoff by utilizing space-time convolution LSTM network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |